使用ingress 訪問dashboard v1.8.3 · kubernets native cloud 最佳實踐總結

# Dashboard功能介紹 Kubernetes Dashboard 是一個管理Kubernetes集群的全功能Web界面，旨在以UI的方式完全替代命令行工具（kubectl 等）。 kubespray 默認已經部署了dashboard，我們只需要簡單修改kubeconfig 文件，然后在通過ingress 添加路由就能正常訪問。 ### 修改dashboard文件添加nodeport訪問方式修改kubespray 提供的dashboard.yml 將service type設置為NodePort，修改后的yaml文件見kubernetes-dashboard.yaml，然后就可以部署新版本的dashboard了。 kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard kubernetes.io/cluster-service: "true" name: kubernetes-dashboard namespace: kube-system spec: type: NodePort ports: - port: 443 targetPort: 8443 selector: k8s-app: kubernetes-dashboard 先刪除原有dashboard，然后在重新創建dashboard kubectl create -f dashboard.yml 獲取dashboard的外網訪問端口 kubectl -n kube-system get svc kubernetes-dashboard NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes-dashboard NodePort 10.233.38.80 <none> 443:31782/TCP 1h 訪問集群中的任何一個節點，即可打開dashboard登陸頁面，如 https://172.20.0.113:32324/ （請使用https訪問），支持使用kubeconfig和token兩種的認證方式：選擇本地的kubeconfig文件以登陸集群，kubeconfig文件中包括登陸的用戶名、證書和token信息。 ![dashboard](https://box.kancloud.cn/2f31c641269ea61bb51f51c96ab7ed91_1628x806.png) 登陸之后首先看到的界面是這樣的：這是因為該用戶沒有對default命名空間的訪問權限。 ![](https://box.kancloud.cn/8de3574ba85f99b082bc1522e8f23511_1199x643.jpg) # 配置Dashboard 登錄權限登陸 dashboard 的時候支持 kubeconfig 和 token 兩種認證方式，kubeconfig 中也依賴 token 字段，所以生成 token 這一步是必不可少的。下文分兩塊來講解兩種登陸認證方式： - 使用 kubeconfig - 使用token登錄 ## 使用kubeconfig 登錄dashboard 登陸dashboard的時候可以指定kubeconfig文件來認證用戶權限，如何生成登陸dashboard時指定的kubeconfig文件請參考[創建用戶認證授權的kubeconfig文件](https://jimmysong.io/kubernetes-handbook/guide/kubectl-user-authentication-authorization.html) 使用kubespray部署集群默認已經給我們創建了kubeconfig 文件，文件路徑/etc/kubernetes/admin.conf。我們只需要在文件最后一行添加token 配置及給管理者用戶綁定權限。 admin.conf文件內容參考如下： apiVersion: v1 kind: Config current-context: admin-cluster.local preferences: {} clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMrVENDQWVHZ0F3SUJBZ0lKQU03VFpMRlhGZkdaTUEwR0NTcUdTSWIzRFFFQk server: https://10.34.11.12:6443 name: cluster.local contexts: - context: cluster: cluster.local user: admin-cluster.local name: admin-cluster.local users: - name: admin-cluster.local user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ4akNDQXRxZ0F3SUJBZ0lKQU5leE1YMUZORHJsTUEwR0NTcUdTSWIzRFFFQkN client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBbjhudGxyQUR4aTFpbytlWEVwU1NYU1ZmMXZqUTV token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNj ### 創建管理用戶及綁定用戶權限 #### 創建用戶 cat admin-user.yaml apiVersion: v1 kind: ServiceAccount metadata: name: admin-cluster.local namespace: kube-system --- ### 綁定RBAC用戶角色 apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: admin-cluster.local roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: admin-cluster.local namespace: kube-system 執行kubectl create命令 kubectl create -f admin-user.yaml ### 獲取token 現在我們需要找到新創建的用戶的Token，以便用來登錄dashboard： kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-cluster.local | awk '{print $1}') 輸出類似： Name: admin-cluster.local-token-xpm5v Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name=admin-cluster.local kubernetes.io/service-account.uid=0610610c-84e7-11e8-98de-00163e02d9ff Type: kubernetes.io/service-account-token Data ==== ca.crt: 1090 bytes namespace: 11 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNj 把查詢到的token字段添加到admin.conf 文件中，登入后顯示如下： ![](https://box.kancloud.cn/21ba985a60fab39d1dfcc6f3a4d8ef1d_1897x845.png) ## 使用token方式登錄dashboard 使用token登錄方式和使用kubeconfig方式非常類似，同樣需要先創建管理用戶及綁定權限。詳細方法可以參考上文。當我們創建好管理用戶后需要查詢創建用戶的token：查詢方法 kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-cluster.local | awk '{print $1}') 輸出類似：token是一段很長的字符 Name: admin-cluster.local-token-xpm5v Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name=admin-cluster.local kubernetes.io/service-account.uid=0610610c-84e7-11e8-98de-00163e02d9ff Type: kubernetes.io/service-account-token Data ==== ca.crt: 1090 bytes namespace: 11 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNj 在 dashboard 登錄頁面上使用上面輸出中的那個非常長的字符串進行 **base64 解碼后作為 token 登錄**，即可以擁有管理員權限操作整個kubernetes集群中的對象。當然您也可以將這串 token 進行 base64 解碼后，加到 admin 用戶的kubeconfig文件中，繼續使用kubeconfig登錄，兩種認證方式任您選擇。 > 注意：定要將 kubectl 的輸出中的 token 值進行 base64 解碼，在線解碼工具 [base64decode](https://www.base64decode.org/)，Linux 和 Mac 有自帶的 base64 命令也可以直接使用，輸入 base64 是進行編碼，Linux 中base64 -d 表示解碼，Mac 中使用 base64 -D。也可以使用 jsonpath 的方式直接獲取 token 的值，如： kubectl -n kube-system get secret admin-cluster.local-token-xpm5v -o jsonpath={.data.token}|base64 -d 結果輸出： eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNj 最后只需要把解碼后的token值填入登錄界面即可登錄。 ### backend https 部署dashboard 創建密鑰 ks8-v1.10 最新版創建秘鑰方法與老版本v1.7.6有所差異 kubectl create secret generic jiedai361-certs --from-file=tls.crt=jiedai361.com.pem --from-file=tls.key=jiedai361.com.key -n kube-system 后端dashboard 使用https ，系統默認有創建kubernetes-dashboard-certs 證書，需要先手動刪除然后重新創建證書：kubernetes-dashboard-certs kubectl delete secret kubernetes-dashboard-certs -n kube-system kubectl delete secret kubernetes-dashboard-key-holder -n kube-system kubectl create secret generic kubernetes-dashboard-certs --from-file=tls.crt=jiedai361.com.pem --from-file=tls.key=jiedai361.com.key -n kube-system ### 添加ingress ，backend 轉發為https apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: ingress.kubernetes.io/ssl-passthrough: "true" # 開啟https 透傳 nginx.org/ssl-backends: "kubernetes-dashboard" kubernetes.io/ingress.allow-http: "false" nginx.ingress.kubernetes.io/secure-backends: "true" # 后端backend 使用https name: dashboard-ingress namespace: kube-system spec: tls: - hosts: - ftc-dashboard.jiedai361.com secretName: kubernetes-dashboard-certs rules: - host: ftc-dashboard.jiedai361.com http: paths: - path: / backend: serviceName: kubernetes-dashboard servicePort: 443 [ingress annotations 注解地址 ](https://github.com/kubernetes/ingress-nginx/blob/722027723f4b08f902f684ccc7c7bb203e42c36a/docs/user-guide/nginx-configuration/annotations.md) ### backend http 部署dashboard 使用kubespray 部署新版dashborad v1.8.3 默認使用https 。也可以個人修改dashboard-http.yaml。使用http:9090 ，添加NodePort:30090 cat /etc/kubernetes/dashboard-http.yml # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Configuration to deploy release version of the Dashboard UI compatible with # Kubernetes 1.8. # # Example usage: kubectl create -f <this_file> # ------------------- Dashboard Secret ------------------- # apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-certs namespace: kube-system type: Opaque --- # ------------------- Dashboard Service Account ------------------- # apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-system --- # ------------------- Dashboard Role & Role Binding ------------------- # kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: kubernetes-dashboard-minimal namespace: kube-system rules: # Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret. - apiGroups: [""] resources: ["secrets"] verbs: ["create"] # Allow Dashboard to create 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] verbs: ["create"] # Allow Dashboard to get, update and delete Dashboard exclusive secrets. - apiGroups: [""] resources: ["secrets"] resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"] verbs: ["get", "update", "delete"] # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] resourceNames: ["kubernetes-dashboard-settings"] verbs: ["get", "update"] # Allow Dashboard to get metrics from heapster. - apiGroups: [""] resources: ["services"] resourceNames: ["heapster"] verbs: ["proxy"] - apiGroups: [""] resources: ["services/proxy"] resourceNames: ["heapster", "http:heapster:", "https:heapster:"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kubernetes-dashboard-minimal namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubernetes-dashboard-minimal subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kube-system --- # ------------------- Dashboard Deployment ------------------- # kind: Deployment apiVersion: apps/v1beta2 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-system spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard spec: serviceAccountName: kubernetes-dashboard containers: - name: kubernetes-dashboard image: dl-harbor.dianrong.com/google_containers/kubernetes-dashboard-amd64:v1.8.3 ports: - containerPort: 9090 protocol: TCP args: #- --auto-generate-certificates # Uncomment the following line to manually specify Kubernetes API server Host # If not specified, Dashboard will attempt to auto discover the API server and connect # to it. Uncomment only if the default does not work. #- --apiserver-host=http://10.0.1.168:8080 volumeMounts: - name: kubernetes-dashboard-certs mountPath: /certs # Create on-disk volume to store exec logs - mountPath: /tmp name: tmp-volume livenessProbe: httpGet: scheme: HTTP path: / port: 9090 initialDelaySeconds: 30 timeoutSeconds: 30 volumes: - name: kubernetes-dashboard-certs secret: secretName: kubernetes-dashboard-certs - name: tmp-volume emptyDir: {} serviceAccountName: kubernetes-dashboard # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule --- # ------------------- Dashboard Service ------------------- # kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-system spec: ports: - port: 9090 targetPort: 9090 selector: k8s-app: kubernetes-dashboard # ------------------------------------------------------------ kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-external namespace: kube-system spec: ports: - port: 9090 targetPort: 9090 nodePort: 30090 type: NodePort selector: k8s-app: kubernetes-dashboard ### Ingress separated between HTTP and HTTPS - HTTP only config ``` apiVersion: extensions/v1beta1 kind: Ingress metadata: name: foo-http annotations: kubernetes.io/ingress.allow-http: "true" ingress.kubernetes.io/ssl-passthrough: "false" spec: rules: - host: foo.com http: paths: - backend: serviceName: foo-service servicePort: 80 ``` - SSL passtrough config ``` apiVersion: extensions/v1beta1 kind: Ingress metadata: name: foo-https annotations: kubernetes.io/ingress.allow-http: "false" ingress.kubernetes.io/ssl-passthrough: "true" spec: rules: - host: foo.com http: paths: - backend: serviceName: foo-service servicePort: 443 ``` ### 創建私有registry secret kubectl create secret docker-registry regcred --docker-server='dl-harbor.dianrong.com' --docker-username='admin' --docker-password='Dianrong@huyi1987' --docker-email='yi.hu@dianrong.com' ### 創建雙向證書 kubectl create secret generic corp.dalianyun-secret --from-file=tls.crt=corp.dalianyun.com.pem.crt --from-file=corp.dalianyun.com.pem.key -n ftc-demo kubectl create secret generic ca-tls-chain --from-file=ca-chain.cert.pem -n ftc-demo