使用rbd-provisioner提供rbd持久化存儲 · kubernetes完整攻略

# 使用rbd-provisioner提供rbd持久化存儲 rbd-provisioner為kubernetes 1.5+版本提供了類似于`kubernetes.io/rbd`的ceph rbd持久化存儲動態配置實現。一些用戶會使用kubeadm來部署集群，或者將kube-controller-manager以容器的方式運行。這種方式下，kubernetes在創建使用ceph rbd pv/pvc時沒任何問題，但使用dynamic provisioning自動管理存儲生命周期時會報錯。提示`"rbd: create volume failed, err: failed to create rbd image: executable file not found in $PATH:"`。問題來自gcr.io提供的kube-controller-manager容器鏡像未打包ceph-common組件，缺少了rbd命令，因此無法通過rbd命令為pod創建rbd image，查了github的相關文章，目前kubernetes官方在kubernetes-incubator/external-storage項目通過External Provisioners的方式來解決此類問題。本文主要針對該問題，通過rbd-provisioner的方式，解決ceph rbd的dynamic provisioning問題。 * 參考鏈接[RBD Volume Provisioner for Kubernetes 1.5+](https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd) ## 部署rbd-provisioner 首先得在kubernetes集群中安裝rbd-provisioner，github倉庫鏈接[https://github.com/kubernetes-incubator/external-storage](https://github.com/kubernetes-incubator/external-storage) ```bash [root@k8s01 ~]# git clone https://github.com/kubernetes-incubator/external-storage.git [root@k8s01 ~]# cd external-storage/ceph/rbd/deploy [root@k8s01 deploy]# NAMESPACE=kube-system [root@k8s01 deploy]# sed -r -i "s/namespace: [^ ]+/namespace: $NAMESPACE/g" ./rbac/clusterrolebinding.yaml ./rbac/rolebinding.yaml [root@k8s01 deploy]# kubectl -n $NAMESPACE apply -f ./rbac ``` * 根據自己需要，修改rbd-provisioner的namespace；部署完成后檢查rbd-provisioner deployment，確保已經正常部署； ```bash [root@k8s01 ~]# kubectl describe deployments.apps -n kube-system rbd-provisioner Name: rbd-provisioner Namespace: kube-system CreationTimestamp: Sat, 13 Oct 2018 20:08:45 +0800 Labels: app=rbd-provisioner Annotations: deployment.kubernetes.io/revision: 1 kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"name":"rbd-provisioner","namespace":"kube-system"},"s... Selector: app=rbd-provisioner Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType: Recreate MinReadySeconds: 0 Pod Template: Labels: app=rbd-provisioner Service Account: rbd-provisioner Containers: rbd-provisioner: Image: quay.io/external_storage/rbd-provisioner:latest Port: <none> Host Port: <none> Environment: PROVISIONER_NAME: ceph.com/rbd Mounts: <none> Volumes: <none> Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable OldReplicaSets: <none> NewReplicaSet: rbd-provisioner-db574c5c (1/1 replicas created) Events: <none> ``` ## 創建storageclass 部署完rbd-provisioner，還需要創建StorageClass。創建SC前，我們還需要創建相關用戶的secret； ```bash [root@k8s01 ~]# vi secrets.yaml apiVersion: v1 kind: Secret metadata: name: ceph-admin-secret namespace: kube-system type: "kubernetes.io/rbd" data: # ceph auth get-key client.admin | base64 key: QVFCdng4QmJKQkFsSFJBQWl1c1o0TGdOV250NlpKQ1BSMHFCa1E9PQ== --- apiVersion: v1 kind: Secret metadata: name: ceph-secret namespace: kube-system type: "kubernetes.io/rbd" data: # ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=kube' # ceph auth get-key client.kube | base64 key: QVFCTHdNRmJueFZ4TUJBQTZjd1MybEJ2Q0JUcmZhRk4yL2tJQVE9PQ== [root@k8s01 ~]# kubectl create -f secrets.yaml [root@k8s01 ~]# vi secrets-default.yaml apiVersion: v1 kind: Secret metadata: name: ceph-secret type: "kubernetes.io/rbd" data: # ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=kube' # ceph auth get-key client.kube | base64 key: QVFCTHdNRmJueFZ4TUJBQTZjd1MybEJ2Q0JUcmZhRk4yL2tJQVE9PQ== [root@k8s01 ~]# kubectl create -f secrets-default.yaml -n default ``` * 創建secret保存client.admin和client.kube用戶的key，client.admin和client.kube用戶的secret可以放在kube-system namespace，但如果其他namespace需要使用ceph rbd的dynamic provisioning功能的話，要在相應的namespace創建secret來保存client.kube用戶key信息； ```bash [root@k8s01 ~]# vi ceph-rbd-sc.yaml apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: name: ceph-rbd annotations: storageclass.beta.kubernetes.io/is-default-class: "true" provisioner: ceph.com/rbd parameters: monitors: 172.16.16.81,172.16.16.82,172.16.16.83 adminId: admin adminSecretName: ceph-admin-secret adminSecretNamespace: kube-system pool: rbd userId: kube userSecretName: ceph-secret fsType: ext4 imageFormat: "2" imageFeatures: "layering" [root@k8s01 ~]# kubectl create -f ceph-rbd-sc.yaml ``` * 其他設置和普通的ceph rbd StorageClass一致，但provisioner需要設置為`ceph.com/rbd`，不是默認的`kubernetes.io/rbd`，這樣rbd的請求將由rbd-provisioner來處理； * 考慮到兼容性，建議盡量關閉rbd image feature，并且kubelet節點的ceph-common版本盡量和ceph服務器端保持一致，我的環境都使用的L版本； ## 測試ceph rbd自動分配在kube-system和default namespace分別創建pod，通過啟動一個busybox實例，將ceph rbd鏡像掛載到`/usr/share/busybox`； ```bash [root@k8s01 ~]# vi test-pod.yaml apiVersion: v1 kind: Pod metadata: name: ceph-pod1 spec: containers: - name: ceph-busybox image: busybox command: ["sleep", "60000"] volumeMounts: - name: ceph-vol1 mountPath: /usr/share/busybox readOnly: false volumes: - name: ceph-vol1 persistentVolumeClaim: claimName: ceph-claim --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: ceph-claim spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi [root@k8s01 ~]# kubectl create -f test-pod.yaml -n kube-system pod/ceph-pod1 created persistentvolumeclaim/ceph-claim created [root@k8s01 ~]# kubectl create -f test-pod.yaml -n default pod/ceph-pod1 created persistentvolumeclaim/ceph-claim created ``` 檢查pv和pvc的創建狀態，是否都已經創建； ```bash [root@k8s01 ~]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-claim Bound pvc-ee0f1c35-cef7-11e8-8484-005056a33f16 2Gi RWO ceph-rbd 25s [root@k8s01 ~]# kubectl get pvc -n kube-system NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-claim Bound pvc-ea377cad-cef7-11e8-8484-005056a33f16 2Gi RWO ceph-rbd 36s [root@k8s01 ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-ea377cad-cef7-11e8-8484-005056a33f16 2Gi RWO Delete Bound kube-system/ceph-claim ceph-rbd 40s pvc-ee0f1c35-cef7-11e8-8484-005056a33f16 2Gi RWO Delete Bound default/ceph-claim ceph-rbd 32s ``` 在ceph服務器上，檢查rbd鏡像創建情況和鏡像的信息； ```bash [root@k8s01 ~]# rbd ls --pool rbd kubernetes-dynamic-pvc-ea390cbf-cef7-11e8-aa22-0a580af40202 kubernetes-dynamic-pvc-eef5814f-cef7-11e8-aa22-0a580af40202 [root@k8s01 ~]# rbd info rbd/kubernetes-dynamic-pvc-ea390cbf-cef7-11e8-aa22-0a580af40202 rbd image 'kubernetes-dynamic-pvc-ea390cbf-cef7-11e8-aa22-0a580af40202': size 2048 MB in 512 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.456876b8b4567 format: 2 features: layering flags: create_timestamp: Sat Oct 13 22:54:41 2018 [root@k8s01 ~]# rbd info rbd/kubernetes-dynamic-pvc-eef5814f-cef7-11e8-aa22-0a580af40202 rbd image 'kubernetes-dynamic-pvc-eef5814f-cef7-11e8-aa22-0a580af40202': size 2048 MB in 512 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.ad6c6b8b4567 format: 2 features: layering flags: create_timestamp: Sat Oct 13 22:54:49 2018 ``` 檢查busybox內的文件系統掛載和使用情況，確認能正常工作； ```bash [root@k8s01 ~]# kubectl exec -it ceph-pod1 mount |grep rbd /dev/rbd0 on /usr/share/busybox type ext4 (rw,seclabel,relatime,stripe=1024,data=ordered) [root@k8s01 ~]# kubectl exec -it -n kube-system ceph-pod1 mount |grep rbd /dev/rbd0 on /usr/share/busybox type ext4 (rw,seclabel,relatime,stripe=1024,data=ordered) [root@k8s01 ~]# kubectl exec -it -n kube-system ceph-pod1 df |grep rbd /dev/rbd0 1998672 6144 1976144 0% /usr/share/busybox [root@k8s01 ~]# kubectl exec -it ceph-pod1 df |grep rbd /dev/rbd0 1998672 6144 1976144 0% /usr/share/busybox ``` 測試刪除pod能否自動刪除pv和pvc，生產環境中謹慎，設置好回收策略； ```bash [root@k8s01 ~]# kubectl delete -f test-pod.yaml pod "ceph-pod1" deleted persistentvolumeclaim "ceph-claim" deleted [root@k8s01 ~]# kubectl delete -f test-pod.yaml -n kube-system pod "ceph-pod1" deleted persistentvolumeclaim "ceph-claim" deleted [root@k8s01 ~]# kubectl get pv No resources found. [root@k8s01 ~]# kubectl get pvc No resources found. [root@k8s01 ~]# kubectl get pvc -n kube-system No resources found. ``` ceph服務器上的rbd image也已清除，自動回收成功； ```bash [root@k8s01 ~]# rbd ls --pool rbd ``` * 確認之前創建的rbd images都已經刪除； ## 總結大部分情況下，我們無需使用rbd provisioner來提供ceph rbd的dynamic provisioning能力。經測試，在OpenShift、Rancher、SUSE CaaS以及本Handbook的二進制文件方式部署，在安裝好ceph-common軟件包的情況下，定義StorageClass時使用`kubernetes.io/rbd`即可正常使用ceph rbd provisioning功能。 ## 參考 - [RBD Volume Provisioner for Kubernetes 1.5+](https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd)