読者です 読者をやめる 読者になる 読者になる

Deisをproduction運用するための設定の仕方調査

背景

Deisはhelmコマンドでインストールすれば簡単に入るけど、そのままだとデータ保存先がephemeralなものになってしまうのでダメ。他にもproduction運用のためにした方が設定があるのでまとめる。

調べ方

変えなきゃいけなさそうなところ

  • kubernetesのバージョン
  • クラスタインスタンスの推奨スペック
  • object storage はminioからGCS
  • postgres は Cloud SQL for postgresql or postgresql on GCE
  • docker registryはGCR
  • firewallの設定。そのままだと0.0.0.0/0からつながってしまう。
  • deis registerできるのをadminだけにする。controllerの設定
  • Grafanaのsign upをdisableにする。
  • SSLの設定。そのままだとオレオレ認証のまま?
  • monitorのinfluxDBの設定。そのままだとmetricsが永続化されない。
  • DNS
  • routerをscaleさせた方がいいとある -> 今回はいらないかも?

変え方

kubernetsの色んなリソースをまとめたものをChartという。まぁざっくりいうとyumとかhomebrew的なもの。 実はChartには海図という意味があり、kubernetesがギリシャ語で船長みたいな意味で、helmが舵という意味なので、kubernetesがchartをもとにhelmをとると考えればなんとなく意味が分かる。

Chartはある決まったstructureをしてるんだけど、それのうちパラメータを設定できるのがvalues.ymlというもの。CLI経由で--setで一つ一つパラメータを渡すこともできるけどめんどいから基本やらない。ただ、--setでoverrideするとか追加するとかはやりそうな気がする。 で、DeisもChartで管理されてるので、当然values.yml経由でパラメータを変更する。

values.ymlの持ってきかたは、helm inspectサブコマンドで取得できる。valuesをつけなければ、values以外の基本的な情報を取得できたりする。ちなみに、当然だけど、deis/workflow以外のものの場合も同様で、deis/workflowの部分をstable/mysqlとかにすれば、情報がとれる。

helm inspect values deis/workflow | sed -n '1!p' > values.yml

values.ymlの内容はこれ

global:
  # Set the storage backend
  #
  # Valid values are:
  # - s3: Store persistent data in AWS S3 (configure in S3 section)
  # - azure: Store persistent data in Azure's object storage
  # - gcs: Store persistent data in Google Cloud Storage
  # - minio: Store persistent data on in-cluster Minio server
  storage: minio

  # Set the location of Workflow's PostgreSQL database
  #
  # Valid values are:
  # - on-cluster: Run PostgreSQL within the Kubernetes cluster (credentials are generated
  #   automatically; backups are sent to object storage
  #   configured above)
  # - off-cluster: Run PostgreSQL outside the Kubernetes cluster (configure in database section)
  database_location: "on-cluster"

  # Set the location of Workflow's logger-specific Redis instance
  #
  # Valid values are:
  # - on-cluster: Run Redis within the Kubernetes cluster
  # - off-cluster: Run Redis outside the Kubernetes cluster (configure in loggerRedis section)
  logger_redis_location: "on-cluster"

  # Set the location of Workflow's influxdb cluster
  #
  # Valid values are:
  # - on-cluster: Run Influxdb within the Kubernetes cluster
  # - off-cluster: Influxdb is running outside of the cluster and credentials and connection information will be provided.
  influxdb_location: "on-cluster"
  # Set the location of Workflow's grafana instance
  #
  # Valid values are:
  # - on-cluster: Run Grafana within the Kubernetes cluster
  # - off-cluster: Grafana is running outside of the cluster
  grafana_location: "on-cluster"

  # Set the location of Workflow's Registry
  #
  # Valid values are:
  # - on-cluster: Run registry within the Kubernetes cluster
  # - off-cluster: Use registry outside the Kubernetes cluster (example: dockerhub,quay.io,self-hosted)
  # - ecr: Use Amazon's ECR
  # - gcr: Use Google's GCR
  registry_location: "on-cluster"
  # The host port to which registry proxy binds to
  host_port: 5555
  # Prefix for the imagepull secret created when using private registry
  secret_prefix: "private-registry"


s3:
  # Your AWS access key. Leave it empty if you want to use IAM credentials.
  accesskey: ""
  # Your AWS secret key. Leave it empty if you want to use IAM credentials.
  secretkey: ""
  # Any S3 region
  region: "us-west-1"
  # Your buckets.
  registry_bucket: "your-registry-bucket-name"
  database_bucket: "your-database-bucket-name"
  builder_bucket: "your-builder-bucket-name"

azure:
  accountname: "YOUR ACCOUNT NAME"
  accountkey: "YOUR ACCOUNT KEY"
  registry_container: "your-registry-container-name"
  database_container: "your-database-container-name"
  builder_container: "your-builder-container-name"

gcs:
  # key_json is expanded into a JSON file on the remote server. It must be
  # well-formatted JSON data.
  key_json: <base64-encoded JSON data>
  registry_bucket: "your-registry-bucket-name"
  database_bucket: "your-database-bucket-name"
  builder_bucket: "your-builder-bucket-name"

swift:
  username: "Your OpenStack Swift Username"
  password: "Your OpenStack Swift Password"
  authurl: "Swift auth URL for obtaining an auth token"
  # Your OpenStack tenant name if you are using auth version 2 or 3.
  tenant: ""
  authversion: "Your OpenStack swift auth version"
  registry_container: "your-registry-container-name"
  database_container: "your-database-container-name"
  builder_container: "your-builder-container-name"

# Set the default (global) way of how Application (your own) images are
# pulled from within the Controller.
# This can be configured per Application as well in the Controller.
#
# This affects pull apps and git push (slugrunner images) apps
#
# Values values are:
# - Always
# - IfNotPresent
controller:
  app_pull_policy: "IfNotPresent"
  # Possible values are:
  # enabled - allows for open registration
  # disabled - turns off open registration
  # admin_only - allows for registration by an admin only.
  registration_mode: "admin_only"

database:
  # The username and password to be used by the on-cluster database.
  # If left empty they will be generated using randAlphaNum
  username: ""
  password: ""
  # Configure the following ONLY if using an off-cluster PostgreSQL database
  postgres:
    name: "database name"
    username: "database username"
    password: "database password"
    host: "database host"
    port: "database port"

redis:
  # Configure the following ONLY if using an off-cluster Redis instance for logger
  db: "0"
  host: "redis host"
  port: "redis port"
  password: "redis password" # "" == no password

fluentd:
  syslog:
    # Configure the following ONLY if using Fluentd to send log messages to both
    # the Logger component and external syslog endpoint
    # external syslog endpoint url
    host: ""
    # external syslog endpoint port
    port: ""

monitor:
  grafana:
    user: "admin"
    password: "admin"
    # Configure the following ONLY if you want persistence for on-cluster grafana
    # GCP PDs and EBS volumes are supported only
    persistence:
      enabled: false # Set to true to enable persistence
      size: 5Gi # PVC size

  influxdb:
    # Configure the following ONLY if using an off-cluster Influx database
    url: "my.influx.url"
    database: "kubernetes"
    user: "user"
    password: "password"
    # Configure the following ONLY if you want persistence for on-cluster influxdb
    # GCP PDs and EBS volumes are supported only
    persistence:
      enabled: false # Set to true to enable persistence
      size: 20Gi # PVC size

registry-token-refresher:
  # Time in minutes after which the token should be refreshed.
  # Leave it empty to use the default provider time.
  token_refresh_time: ""
  off_cluster_registry:
    hostname: ""
    organization: ""
    username: ""
    password: ""
  ecr:
    # Your AWS access key. Leave it empty if you want to use IAM credentials.
    accesskey: ""
    # Your AWS secret key. Leave it empty if you want to use IAM credentials.
    secretkey: ""
    # Any S3 region
    region: "us-west-2"
    registryid: ""
    hostname: ""
  gcr:
    key_json: <base64-encoded JSON data>
    hostname: ""

router:
  dhparam: ""
  # Any custom router annotations(https://github.com/deis/router#annotations)
  # which need to be applied can be specified as key-value pairs under "deployment_annotations"
  deployment_annotations:
    #<example-key>: <example-value>

  # Any custom annotations for k8s services like http://kubernetes.io/docs/user-guide/services/#ssl-support-on-aws
  # which need to be applied can be specified as key-value pairs under "service_annotations"
  service_annotations:
    #<example-key>: <example-value>

  # Enable to pin router pod hostPort when using vagrant
  host_port:
    enabled: false

  # Service type default to LoadBalancer
  # service_type: LoadBalancer

workflow-manager:
  versions_api_url: https://versions.deis.com
  doctor_api_url: https://doctor.deis.com

このvalues.ymlの内容をproductionのための設定に変えていって、helm installとやる予定。