ConfigMap的熱更新 · kubernetes完整攻略

# ConfigMap的熱更新 ConfigMap是用來存儲配置文件的kubernetes資源對象，所有的配置內容都存儲在etcd中，下文主要是探究 ConfigMap 的創建和更新流程，以及對 ConfigMap 更新后容器內掛載的內容是否同步更新的測試。 ## 測試示例假設我們在 `default` namespace 下有一個名為 `nginx-config` 的 ConfigMap，可以使用 `kubectl`命令來獲取： ```bash $ kubectl get configmap nginx-config NAME DATA AGE nginx-config 1 99d ``` 獲取該ConfigMap的內容。 ```bash kubectl get configmap nginx-config -o yaml ``` ```bash apiVersion: v1 data: nginx.conf: |- worker_processes 1; events { worker_connections 1024; } http { sendfile on; server { listen 80; # a test endpoint that returns http 200s location / { proxy_pass http://httpstat.us/200; proxy_set_header X-Real-IP $remote_addr; } } server { listen 80; server_name api.hello.world; location / { proxy_pass http://l5d.default.svc.cluster.local; proxy_set_header Host $host; proxy_set_header Connection ""; proxy_http_version 1.1; more_clear_input_headers 'l5d-ctx-*' 'l5d-dtab' 'l5d-sample'; } } server { listen 80; server_name www.hello.world; location / { # allow 'employees' to perform dtab overrides if ($cookie_special_employee_cookie != "letmein") { more_clear_input_headers 'l5d-ctx-*' 'l5d-dtab' 'l5d-sample'; } # add a dtab override to get people to our beta, world-v2 set $xheader ""; if ($cookie_special_employee_cookie ~* "dogfood") { set $xheader "/host/world => /srv/world-v2;"; } proxy_set_header 'l5d-dtab' $xheader; proxy_pass http://l5d.default.svc.cluster.local; proxy_set_header Host $host; proxy_set_header Connection ""; proxy_http_version 1.1; } } } kind: ConfigMap metadata: creationTimestamp: 2017-08-01T06:53:17Z name: nginx-config namespace: default resourceVersion: "14925806" selfLink: /api/v1/namespaces/default/configmaps/nginx-config uid: 18d70527-7686-11e7-bfbd-8af1e3a7c5bd ``` ConfigMap中的內容是存儲到etcd中的，然后查詢etcd： ```bash ETCDCTL_API=3 etcdctl get /registry/configmaps/default/nginx-config -w json|python -m json.tool ``` 注意使用 v3 版本的 etcdctl API，下面是輸出結果： ```json { "count": 1, "header": { "cluster_id": 12091028579527406772, "member_id": 16557816780141026208, "raft_term": 36, "revision": 29258723 }, "kvs": [ { "create_revision": 14925806, "key": "L3JlZ2lzdHJ5L2NvbmZpZ21hcHMvZGVmYXVsdC9uZ2lueC1jb25maWc=", "mod_revision": 14925806, "value": "azhzAAoPCgJ2MRIJQ29uZmlnTWFwEqQMClQKDG5naW54LWNvbmZpZxIAGgdkZWZhdWx0IgAqJDE4ZDcwNTI3LTc2ODYtMTFlNy1iZmJkLThhZjFlM2E3YzViZDIAOABCCwjdyoDMBRC5ss54egASywsKCm5naW54LmNvbmYSvAt3b3JrZXJfcHJvY2Vzc2VzIDE7CgpldmVudHMgeyB3b3JrZXJfY29ubmVjdGlvbnMgMTAyNDsgfQoKaHR0cCB7CiAgICBzZW5kZmlsZSBvbjsKCiAgICBzZXJ2ZXIgewogICAgICAgIGxpc3RlbiA4MDsKCiAgICAgICAgIyBhIHRlc3QgZW5kcG9pbnQgdGhhdCByZXR1cm5zIGh0dHAgMjAwcwogICAgICAgIGxvY2F0aW9uIC8gewogICAgICAgICAgICBwcm94eV9wYXNzIGh0dHA6Ly9odHRwc3RhdC51cy8yMDA7CiAgICAgICAgICAgIHByb3h5X3NldF9oZWFkZXIgIFgtUmVhbC1JUCAgJHJlbW90ZV9hZGRyOwogICAgICAgIH0KICAgIH0KCiAgICBzZXJ2ZXIgewoKICAgICAgICBsaXN0ZW4gODA7CiAgICAgICAgc2VydmVyX25hbWUgYXBpLmhlbGxvLndvcmxkOwoKICAgICAgICBsb2NhdGlvbiAvIHsKICAgICAgICAgICAgcHJveHlfcGFzcyBodHRwOi8vbDVkLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw7CiAgICAgICAgICAgIHByb3h5X3NldF9oZWFkZXIgSG9zdCAkaG9zdDsKICAgICAgICAgICAgcHJveHlfc2V0X2hlYWRlciBDb25uZWN0aW9uICIiOwogICAgICAgICAgICBwcm94eV9odHRwX3ZlcnNpb24gMS4xOwoKICAgICAgICAgICAgbW9yZV9jbGVhcl9pbnB1dF9oZWFkZXJzICdsNWQtY3R4LSonICdsNWQtZHRhYicgJ2w1ZC1zYW1wbGUnOwogICAgICAgIH0KICAgIH0KCiAgICBzZXJ2ZXIgewoKICAgICAgICBsaXN0ZW4gODA7CiAgICAgICAgc2VydmVyX25hbWUgd3d3LmhlbGxvLndvcmxkOwoKICAgICAgICBsb2NhdGlvbiAvIHsKCgogICAgICAgICAgICAjIGFsbG93ICdlbXBsb3llZXMnIHRvIHBlcmZvcm0gZHRhYiBvdmVycmlkZXMKICAgICAgICAgICAgaWYgKCRjb29raWVfc3BlY2lhbF9lbXBsb3llZV9jb29raWUgIT0gImxldG1laW4iKSB7CiAgICAgICAgICAgICAgbW9yZV9jbGVhcl9pbnB1dF9oZWFkZXJzICdsNWQtY3R4LSonICdsNWQtZHRhYicgJ2w1ZC1zYW1wbGUnOwogICAgICAgICAgICB9CgogICAgICAgICAgICAjIGFkZCBhIGR0YWIgb3ZlcnJpZGUgdG8gZ2V0IHBlb3BsZSB0byBvdXIgYmV0YSwgd29ybGQtdjIKICAgICAgICAgICAgc2V0ICR4aGVhZGVyICIiOwoKICAgICAgICAgICAgaWYgKCRjb29raWVfc3BlY2lhbF9lbXBsb3llZV9jb29raWUgfiogImRvZ2Zvb2QiKSB7CiAgICAgICAgICAgICAgc2V0ICR4aGVhZGVyICIvaG9zdC93b3JsZCA9PiAvc3J2L3dvcmxkLXYyOyI7CiAgICAgICAgICAgIH0KCiAgICAgICAgICAgIHByb3h5X3NldF9oZWFkZXIgJ2w1ZC1kdGFiJyAkeGhlYWRlcjsKCgogICAgICAgICAgICBwcm94eV9wYXNzIGh0dHA6Ly9sNWQuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbDsKICAgICAgICAgICAgcHJveHlfc2V0X2hlYWRlciBIb3N0ICRob3N0OwogICAgICAgICAgICBwcm94eV9zZXRfaGVhZGVyIENvbm5lY3Rpb24gIiI7CiAgICAgICAgICAgIHByb3h5X2h0dHBfdmVyc2lvbiAxLjE7CiAgICAgICAgfQogICAgfQp9GgAiAA==", "version": 1 } ] } ``` 其中的value就是 `nginx.conf` 配置文件的內容。可以使用base64解碼查看具體值，關于etcdctl的使用請參考[使用etcdctl訪問kuberentes數據](../guide/using-etcdctl-to-access-kubernetes-data.md)。 ## 代碼 ConfigMap 結構體的定義： ```go // ConfigMap holds configuration data for pods to consume. type ConfigMap struct { metav1.TypeMeta `json:",inline"` // Standard object's metadata. // More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata // +optional metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` // Data contains the configuration data. // Each key must be a valid DNS_SUBDOMAIN with an optional leading dot. // +optional Data map[string]string `json:"data,omitempty" protobuf:"bytes,2,rep,name=data"` } ``` 在 `staging/src/k8s.io/client-go/kubernetes/typed/core/v1/configmap.go` 中ConfigMap 的接口定義： ```go // ConfigMapInterface has methods to work with ConfigMap resources. type ConfigMapInterface interface { Create(*v1.ConfigMap) (*v1.ConfigMap, error) Update(*v1.ConfigMap) (*v1.ConfigMap, error) Delete(name string, options *meta_v1.DeleteOptions) error DeleteCollection(options *meta_v1.DeleteOptions, listOptions meta_v1.ListOptions) error Get(name string, options meta_v1.GetOptions) (*v1.ConfigMap, error) List(opts meta_v1.ListOptions) (*v1.ConfigMapList, error) Watch(opts meta_v1.ListOptions) (watch.Interface, error) Patch(name string, pt types.PatchType, data []byte, subresources ...string) (result *v1.ConfigMap, err error) ConfigMapExpansion } ``` 在 `staging/src/k8s.io/client-go/kubernetes/typed/core/v1/configmap.go` 中創建 ConfigMap 的方法如下: ```go // Create takes the representation of a configMap and creates it. Returns the server's representation of the configMap, and an error, if there is any. func (c *configMaps) Create(configMap *v1.ConfigMap) (result *v1.ConfigMap, err error) { result = &v1.ConfigMap{} err = c.client.Post(). Namespace(c.ns). Resource("configmaps"). Body(configMap). Do(). Into(result) return } ``` 通過 RESTful 請求在 etcd 中存儲 ConfigMap 的配置，該方法中設置了資源對象的 namespace 和 HTTP 請求中的 body，執行后將請求結果保存到 result 中返回給調用者。 **注意 Body 的結構** ```java // Body makes the request use obj as the body. Optional. // If obj is a string, try to read a file of that name. // If obj is a []byte, send it directly. // If obj is an io.Reader, use it directly. // If obj is a runtime.Object, marshal it correctly, and set Content-Type header. // If obj is a runtime.Object and nil, do nothing. // Otherwise, set an error. ``` 創建 ConfigMap RESTful 請求中的的 Body 中包含 `ObjectMeta` 和 `namespace`。 HTTP 請求中的結構體： ```go // Request allows for building up a request to a server in a chained fashion. // Any errors are stored until the end of your call, so you only have to // check once. type Request struct { // required client HTTPClient verb string baseURL *url.URL content ContentConfig serializers Serializers // generic components accessible via method setters pathPrefix string subpath string params url.Values headers http.Header // structural elements of the request that are part of the Kubernetes API conventions namespace string namespaceSet bool resource string resourceName string subresource string timeout time.Duration // output err error body io.Reader // This is only used for per-request timeouts, deadlines, and cancellations. ctx context.Context backoffMgr BackoffManager throttle flowcontrol.RateLimiter } ``` ## 測試分別測試使用 ConfigMap 掛載 Env 和 Volume 的情況。 ### 更新使用ConfigMap掛載的Env 使用下面的配置創建 nginx 容器測試更新 ConfigMap 后容器內的環境變量是否也跟著更新。 ```yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-nginx spec: replicas: 1 template: metadata: labels: run: my-nginx spec: containers: - name: my-nginx image: harbor-001.jimmysong.io/library/nginx:1.9 ports: - containerPort: 80 envFrom: - configMapRef: name: env-config --- apiVersion: v1 kind: ConfigMap metadata: name: env-config namespace: default data: log_level: INFO ``` 獲取環境變量的值 ```bash $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` env|grep log_level log_level=INFO ``` 修改 ConfigMap ```bash $ kubectl edit configmap env-config ``` 修改 `log_level` 的值為 `DEBUG`。再次查看環境變量的值。 ```bash $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` env|grep log_level log_level=INFO ``` 實踐證明修改 ConfigMap 無法更新容器中已注入的環境變量信息。 ### 更新使用ConfigMap掛載的Volume 使用下面的配置創建 nginx 容器測試更新 ConfigMap 后容器內掛載的文件是否也跟著更新。 ```yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-nginx spec: replicas: 1 template: metadata: labels: run: my-nginx spec: containers: - name: my-nginx image: harbor-001.jimmysong.io/library/nginx:1.9 ports: - containerPort: 80 volumeMounts: - name: config-volume mountPath: /etc/config volumes: - name: config-volume configMap: name: special-config --- apiVersion: v1 kind: ConfigMap metadata: name: special-config namespace: default data: log_level: INFO ``` ```bash $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` cat /etc/config/log_level INFO ``` 修改 ConfigMap ```bash $ kubectl edit configmap special-config ``` 修改 `log_level` 的值為 `DEBUG`。等待大概10秒鐘時間，再次查看環境變量的值。 ```bash $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` cat /tmp/log_level DEBUG ``` 我們可以看到使用 ConfigMap 方式掛載的 Volume 的文件中的內容已經變成了 `DEBUG`。 Known Issue：如果使用ConfigMap的**subPath**掛載為Container的Volume，Kubernetes不會做自動熱更新: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically ## ConfigMap 更新后滾動更新 Pod 更新 ConfigMap 目前并不會觸發相關 Pod 的滾動更新，可以通過修改 pod annotations 的方式強制觸發滾動更新。 ```bash $ kubectl patch deployment my-nginx --patch '{"spec": {"template": {"metadata": {"annotations": {"version/config": "20180411" }}}}}' ``` 這個例子里我們在 `.spec.template.metadata.annotations` 中添加 `version/config`，每次通過修改 `version/config` 來觸發滾動更新。 ## 總結更新 ConfigMap 后： - 使用該 ConfigMap 掛載的 Env **不會**同步更新 - 使用該 ConfigMap 掛載的 Volume 中的數據需要一段時間（實測大概10秒）才能同步更新 ENV 是在容器啟動的時候注入的，啟動之后 kubernetes 就不會再改變環境變量的值，且同一個 namespace 中的 pod 的環境變量是不斷累加的，參考 [Kubernetes中的服務發現與docker容器間的環境變量傳遞源碼探究](https://jimmysong.io/posts/exploring-kubernetes-env-with-docker/)。為了更新容器中使用 ConfigMap 掛載的配置，需要通過滾動更新 pod 的方式來強制重新掛載 ConfigMap。 ## 參考 - [Kubernetes 1.7 security in practice](https://acotten.com/post/kube17-security) - [ConfigMap | kubernetes handbook - jimmysong.io](https://jimmysong.io/kubernetes-handbook/concepts/configmap.html) - [創建高可用ectd集群 | Kubernetes handbook - jimmysong.io](https://jimmysong.io/kubernetes-handbook/practice/etcd-cluster-installation.html) - [Kubernetes中的服務發現與docker容器間的環境變量傳遞源碼探究](https://jimmysong.io/posts/exploring-kubernetes-env-with-docker/) - [Automatically Roll Deployments When ConfigMaps or Secrets change](https://github.com/kubernetes/helm/blob/master/docs/charts_tips_and_tricks.md#automatically-roll-deployments-when-configmaps-or-secrets-change)