自定義指標HPA · kubernetes完整攻略

# 自定義指標HPA Kubernetes中不僅支持CPU、內存為指標的HPA，還支持自定義指標的HPA，例如QPS。本文中使用的yaml文件見[manifests/HPA](https://github.com/rootsongjc/kubernetes-handbook/tree/master/manifests/HPA)。 ## 設置自定義指標 **kubernetes1.6** > 在kubernetes1.6集群中配置自定義指標的HPA的說明已廢棄。在設置定義指標HPA之前需要先進行如下配置： - 將heapster的啟動參數 `--api-server` 設置為 true - 啟用custom metric API - 將kube-controller-manager的啟動參數中`--horizontal-pod-autoscaler-use-rest-clients`設置為true，并指定`--master`為API server地址，如`--master=http://172.20.0.113:8080` 在kubernetes1.5以前很容易設置，參考[1.6以前版本的kubernetes中開啟自定義HPA](https://medium.com/@marko.luksa/kubernetes-autoscaling-based-on-custom-metrics-without-using-a-host-port-b783ed6241ac)，而在1.6中因為取消了原來的annotation方式設置custom metric，只能通過API server和kube-aggregator來獲取custom metric，因為只有兩種方式來設置了，一是直接通過API server獲取heapster的metrics，二是部署[kube-aggragator](https://github.com/kubernetes/kube-aggregator)來實現。我們將在kubernetes1.8版本的kubernetes中，使用聚合的API server來實現自定義指標的HPA。 **kuberentes1.7+** 確認您的kubernetes版本在1.7或以上，修改以下配置： - 將kube-controller-manager的啟動參數中`--horizontal-pod-autoscaler-use-rest-clients`設置為true，并指定`--master`為API server地址，如`--master=http://172.20.0.113:8080 - 修改kube-apiserver的配置文件apiserver，增加一條配置`--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem --requestheader-allowed-names=aggregator --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=/etc/kubernetes/ssl/kubernetes.pem --proxy-client-key-file=/etc/kubernetes/ssl/kubernetes-key.pem`，用來配置aggregator的CA證書。已經內置了`apiregistration.k8s.io/v1beta1` API，可以直接定義APIService，如： ```yaml apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: name: v1.custom-metrics.metrics.k8s.io spec: insecureSkipTLSVerify: true group: custom-metrics.metrics.k8s.io groupPriorityMinimum: 1000 versionPriority: 5 service: name: api namespace: custom-metrics version: v1alpha1 ``` **部署Prometheus** 使用[prometheus-operator.yaml](https://github.com/rootsongjc/kubernetes-handbook/blob/master/manifests/HPA/prometheus-operator.yaml)文件部署Prometheus operator。 **注意：**將鏡像修改為你自己的鏡像倉庫地址。這產生一個自定義的API：<http://172.20.0.113:8080/apis/custom-metrics.metrics.k8s.io/v1alpha1>，可以通過瀏覽器訪問，還可以使用下面的命令可以檢查該API： ```bash $ kubectl get --raw=apis/custom-metrics.metrics.k8s.io/v1alpha1 {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom-metrics.metrics.k8s.io/v1alpha1","resources":[{"name":"jobs.batch/http_requests","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/http_requests","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"jobs.batch/up","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/up","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"services/scrape_samples_scraped","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/scrape_samples_scraped","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/scrape_duration_seconds","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"services/scrape_duration_seconds","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/http_requests","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/scrape_samples_post_metric_relabeling","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"jobs.batch/scrape_samples_scraped","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"jobs.batch/scrape_duration_seconds","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/scrape_duration_seconds","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/scrape_samples_post_metric_relabeling","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"services/scrape_samples_post_metric_relabeling","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"services/up","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/scrape_samples_scraped","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"services/http_requests","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"jobs.batch/scrape_samples_post_metric_relabeling","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/up","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]}]} ``` ## 參考 - [1.6以前版本的kubernetes中開啟自定義HPA](https://medium.com/@marko.luksa/kubernetes-autoscaling-based-on-custom-metrics-without-using-a-host-port-b783ed6241ac) - [1.7版本的kubernetes中啟用自定義HPA](https://docs.bitnami.com/kubernetes/how-to/configure-autoscaling-custom-metrics/) - [Horizontal Pod Autoscaler Walkthrough](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/) - [Kubernetes 1.8: Now with 100% Daily Value of Custom Metrics](https://blog.openshift.com/kubernetes-1-8-now-custom-metrics/) - [Arbitrary/Custom Metrics in the Horizontal Pod Autoscaler#117](https://github.com/kubernetes/features/issues/117)