Deisをproduction運用するための設定の仕方調査
背景
Deisはhelmコマンドでインストールすれば簡単に入るけど、そのままだとデータ保存先がephemeralなものになってしまうのでダメ。他にもproduction運用のためにした方が設定があるのでまとめる。
調べ方
- ほとんどdeisの公式ページに書いてあった
- installing-workflow
- production-deployments
- cloud sql postgres/connect-container-engine
変えなきゃいけなさそうなところ
- kubernetesのバージョン
- クラスターインスタンスの推奨スペック
- object storage はminioから
GCS
へ - postgres は
Cloud SQL for postgresql
orpostgresql on GCE
- docker registryは
GCR
へ - firewallの設定。そのままだと0.0.0.0/0からつながってしまう。
- deis registerできるのをadminだけにする。controllerの設定
- Grafanaのsign upをdisableにする。
- SSLの設定。そのままだとオレオレ認証のまま?
- monitorのinfluxDBの設定。そのままだとmetricsが永続化されない。
- DNS
- routerをscaleさせた方がいいとある -> 今回はいらないかも?
変え方
kubernetsの色んなリソースをまとめたものをChartという。まぁざっくりいうとyumとかhomebrew的なもの。 実はChartには海図という意味があり、kubernetesがギリシャ語で船長みたいな意味で、helmが舵という意味なので、kubernetesがchartをもとにhelmをとると考えればなんとなく意味が分かる。
Chartはある決まったstructureをしてるんだけど、それのうちパラメータを設定できるのがvalues.ymlというもの。CLI経由で--set
で一つ一つパラメータを渡すこともできるけどめんどいから基本やらない。ただ、--set
でoverrideするとか追加するとかはやりそうな気がする。
で、DeisもChartで管理されてるので、当然values.yml経由でパラメータを変更する。
values.ymlの持ってきかたは、helm inspect
サブコマンドで取得できる。valuesをつけなければ、values以外の基本的な情報を取得できたりする。ちなみに、当然だけど、deis/workflow以外のものの場合も同様で、deis/workflow
の部分をstable/mysql
とかにすれば、情報がとれる。
helm inspect values deis/workflow | sed -n '1!p' > values.yml
values.ymlの内容はこれ
global: # Set the storage backend # # Valid values are: # - s3: Store persistent data in AWS S3 (configure in S3 section) # - azure: Store persistent data in Azure's object storage # - gcs: Store persistent data in Google Cloud Storage # - minio: Store persistent data on in-cluster Minio server storage: minio # Set the location of Workflow's PostgreSQL database # # Valid values are: # - on-cluster: Run PostgreSQL within the Kubernetes cluster (credentials are generated # automatically; backups are sent to object storage # configured above) # - off-cluster: Run PostgreSQL outside the Kubernetes cluster (configure in database section) database_location: "on-cluster" # Set the location of Workflow's logger-specific Redis instance # # Valid values are: # - on-cluster: Run Redis within the Kubernetes cluster # - off-cluster: Run Redis outside the Kubernetes cluster (configure in loggerRedis section) logger_redis_location: "on-cluster" # Set the location of Workflow's influxdb cluster # # Valid values are: # - on-cluster: Run Influxdb within the Kubernetes cluster # - off-cluster: Influxdb is running outside of the cluster and credentials and connection information will be provided. influxdb_location: "on-cluster" # Set the location of Workflow's grafana instance # # Valid values are: # - on-cluster: Run Grafana within the Kubernetes cluster # - off-cluster: Grafana is running outside of the cluster grafana_location: "on-cluster" # Set the location of Workflow's Registry # # Valid values are: # - on-cluster: Run registry within the Kubernetes cluster # - off-cluster: Use registry outside the Kubernetes cluster (example: dockerhub,quay.io,self-hosted) # - ecr: Use Amazon's ECR # - gcr: Use Google's GCR registry_location: "on-cluster" # The host port to which registry proxy binds to host_port: 5555 # Prefix for the imagepull secret created when using private registry secret_prefix: "private-registry" s3: # Your AWS access key. Leave it empty if you want to use IAM credentials. accesskey: "" # Your AWS secret key. Leave it empty if you want to use IAM credentials. secretkey: "" # Any S3 region region: "us-west-1" # Your buckets. registry_bucket: "your-registry-bucket-name" database_bucket: "your-database-bucket-name" builder_bucket: "your-builder-bucket-name" azure: accountname: "YOUR ACCOUNT NAME" accountkey: "YOUR ACCOUNT KEY" registry_container: "your-registry-container-name" database_container: "your-database-container-name" builder_container: "your-builder-container-name" gcs: # key_json is expanded into a JSON file on the remote server. It must be # well-formatted JSON data. key_json: <base64-encoded JSON data> registry_bucket: "your-registry-bucket-name" database_bucket: "your-database-bucket-name" builder_bucket: "your-builder-bucket-name" swift: username: "Your OpenStack Swift Username" password: "Your OpenStack Swift Password" authurl: "Swift auth URL for obtaining an auth token" # Your OpenStack tenant name if you are using auth version 2 or 3. tenant: "" authversion: "Your OpenStack swift auth version" registry_container: "your-registry-container-name" database_container: "your-database-container-name" builder_container: "your-builder-container-name" # Set the default (global) way of how Application (your own) images are # pulled from within the Controller. # This can be configured per Application as well in the Controller. # # This affects pull apps and git push (slugrunner images) apps # # Values values are: # - Always # - IfNotPresent controller: app_pull_policy: "IfNotPresent" # Possible values are: # enabled - allows for open registration # disabled - turns off open registration # admin_only - allows for registration by an admin only. registration_mode: "admin_only" database: # The username and password to be used by the on-cluster database. # If left empty they will be generated using randAlphaNum username: "" password: "" # Configure the following ONLY if using an off-cluster PostgreSQL database postgres: name: "database name" username: "database username" password: "database password" host: "database host" port: "database port" redis: # Configure the following ONLY if using an off-cluster Redis instance for logger db: "0" host: "redis host" port: "redis port" password: "redis password" # "" == no password fluentd: syslog: # Configure the following ONLY if using Fluentd to send log messages to both # the Logger component and external syslog endpoint # external syslog endpoint url host: "" # external syslog endpoint port port: "" monitor: grafana: user: "admin" password: "admin" # Configure the following ONLY if you want persistence for on-cluster grafana # GCP PDs and EBS volumes are supported only persistence: enabled: false # Set to true to enable persistence size: 5Gi # PVC size influxdb: # Configure the following ONLY if using an off-cluster Influx database url: "my.influx.url" database: "kubernetes" user: "user" password: "password" # Configure the following ONLY if you want persistence for on-cluster influxdb # GCP PDs and EBS volumes are supported only persistence: enabled: false # Set to true to enable persistence size: 20Gi # PVC size registry-token-refresher: # Time in minutes after which the token should be refreshed. # Leave it empty to use the default provider time. token_refresh_time: "" off_cluster_registry: hostname: "" organization: "" username: "" password: "" ecr: # Your AWS access key. Leave it empty if you want to use IAM credentials. accesskey: "" # Your AWS secret key. Leave it empty if you want to use IAM credentials. secretkey: "" # Any S3 region region: "us-west-2" registryid: "" hostname: "" gcr: key_json: <base64-encoded JSON data> hostname: "" router: dhparam: "" # Any custom router annotations(https://github.com/deis/router#annotations) # which need to be applied can be specified as key-value pairs under "deployment_annotations" deployment_annotations: #<example-key>: <example-value> # Any custom annotations for k8s services like http://kubernetes.io/docs/user-guide/services/#ssl-support-on-aws # which need to be applied can be specified as key-value pairs under "service_annotations" service_annotations: #<example-key>: <example-value> # Enable to pin router pod hostPort when using vagrant host_port: enabled: false # Service type default to LoadBalancer # service_type: LoadBalancer workflow-manager: versions_api_url: https://versions.deis.com doctor_api_url: https://doctor.deis.com
このvalues.ymlの内容をproductionのための設定に変えていって、helm installとやる予定。