本文翻譯自nfs-client-provisioner的[說明文檔](https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client)，本文將介紹使用nfs-client-provisioner這個應用，利用NFS Server給Kubernetes作為持久存儲的后端，并且動態提供PV。前提條件是有已經安裝好的NFS服務器，并且NFS服務器與Kubernetes的Slave節點都能網絡連通。 所有下文用到的文件來自于`git clone https://github.com/kubernetes-incubator/external-storage.git`的nfs-client目錄。 ### nfs-client-provisioner nfs-client-provisioner 是一個Kubernetes的簡易NFS的外部provisioner，本身不提供NFS，需要現有的NFS服務器提供存儲 * PV以 `${namespace}-${pvcName}-${pvName}`的命名格式提供（在NFS服務器上） * PV回收的時候以 `archieved-${namespace}-${pvcName}-${pvName}` 的命名格式（在NFS服務器上） ---------------------- ### 安裝部署 修改deployment文件并部署 `deploy/deployment.yaml` 需要修改的地方只有NFS服務器所在的IP地址（10.10.10.60），以及NFS服務器共享的路徑（`/ifs/kubernetes`），兩處都需要修改為你實際的NFS服務器和共享目錄 kind: Deployment apiVersion: extensions/v1beta1 metadata: name: nfs-client-provisioner spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: quay.io/external_storage/nfs-client-provisioner:latest volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: 10.10.10.60 - name: NFS_PATH value: /ifs/kubernetes volumes: - name: nfs-client-root nfs: server: 10.10.10.60 path: /ifs/kubernetes 修改StorageClass文件并部署 `deploy/class.yaml` 此處可以不修改，或者修改provisioner的名字，需要與上面的deployment的PROVISIONER\_NAME名字一致。 ~~~ apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: managed-nfs-storage provisioner: fuseim.pri/ifs ~~~ ### 如果啟用了RBAC 需要執行如下的命令來授權。 nfs-rbac.yaml 權限管理 kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner 授權 ~~~ $ kubectl create -f deploy/auth/serviceaccount.yaml serviceaccount "nfs-client-provisioner" created $ kubectl create -f deploy/auth/clusterrole.yaml clusterrole "nfs-client-provisioner-runner" created $ kubectl create -f deploy/auth/clusterrolebinding.yaml clusterrolebinding "run-nfs-client-provisioner" created $ kubectl patch deployment nfs-client-provisioner -p '{"spec":{"template":{"spec":{"serviceAccount":"nfs-client-provisioner"}}}}' ~~~ 如果您的集群啟用了RBAC，或者您正在運行OpenShift，則必須授權provisioner。 如果你在非默認的“default”名稱空間/項目之外部署，可以編輯`deploy/auth/clusterrolebinding.yaml`或編輯`oadm policy“指令。 ---- ### 問題描述: ????kubernetes掛載nfs發現發現pod一直處于containercreating狀態，describe pod 發現報錯如下： [root@master nginx-ip\]# kubectl describe pod nginx-r67f7 Name:? ? ? ? ? ?nginx-r67f7 Namespace:? ? ? default Node:? ? ? ? ? ?master/10.168.xx.xx Start Time:? ? ?Sun, 10 Jun 2018 09:47:50 +0800 Labels:? ? ? ? ?name=nginx Annotations:? ? Status:? ? ? ? ?Pending IP:? ? ? ? ? ? ? Controlled By:? ReplicationController/nginx Containers: ? nginx: ? ? Container ID:? ? ? ? Image:? ? ? ? ? reg.xx.xx/library/nginx-ip:0.4 ? ? Image ID:? ? ? ? ? ? Port:? ? ? ? ? ?80/TCP ? ? State:? ? ? ? ? Waiting ? ? ? Reason:? ? ? ?ContainerCreating ? ? Ready:? ? ? ? ? False ? ? Restart Count:? 0 ? ? Environment:? ? ? ? Mounts: ? ? ? /usr/share/nginx/html from httpd-storage (rw) ? ? ? /var/run/secrets/kubernetes.io/serviceaccount from default-token-drh57 (ro) Conditions: ? Type? ? ? ? ? Status ? Initialized? ?True? ? Ready? ? ? ? ?False? Volumes: ? httpd-storage: ? ? Type:? ? ? NFS (an NFS mount that lasts the lifetime of a pod) ? ? Server:? ? 10.168.xx.xx ? ? Path:? ? ? /data/web ? ? ReadOnly:? false ? default-token-drh57: ? ? Type:? ? ? ? Secret (a volume populated by a Secret) ? ? SecretName:? default-token-drh57 ? ? Optional:? ? false QoS Class:? ? ? ?BestEffort Node-Selectors:? Tolerations:? ? ?node.kubernetes.io/not-ready:NoExecute for 300s ? ? ? ? ? ? ? ? ?node.kubernetes.io/unreachable:NoExecute for 300s Events: ? Type? ? ?Reason? ? ? ?Age? ?From? ? ? ? ? ? ? ? ? ? ?Message ? —-? ? ?——? ? ? ?—-? —-? ? ? ? ? ? ? ? ? ? ?——- ? Warning? FailedMount? 18s? ?kubelet, master? MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage Output: Running scope as unit run-34311.scope. mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web, ? ? ? ?missing codepage or helper program, or other error ? ? ? ?(for several filesystems (e.g. nfs, cifs) you might ? ? ? ?need a /sbin/mount. helper program) ? ? ? ?In some cases useful info is found in syslog – try ? ? ? ?dmesg | tail or so. ? Normal? ?SuccessfulMountVolume? 18s? kubelet, master? MountVolume.SetUp succeeded for volume "default-token-drh57" ? Warning? FailedMount? ? ? ? ? ? 17s? kubelet, master? MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage Output: Running scope as unit run-34316.scope. mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web, ? ? ? ?missing codepage or helper program, or other error ? ? ? ?(for several filesystems (e.g. nfs, cifs) you might ? ? ? ?need a /sbin/mount. helper program) ? ? ? ?In some cases useful info is found in syslog – try ? ? ? ?dmesg | tail or so. ? Warning? FailedMount? 16s? kubelet, master? MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage Output: Running scope as unit run-34325.scope. mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web, ? ? ? ?missing codepage or helper program, or other error ? ? ? ?(for several filesystems (e.g. nfs, cifs) you might ? ? ? ?need a /sbin/mount. helper program) ? ? ? ?In some cases useful info is found in syslog – try ? ? ? ?dmesg | tail or so. ? Warning? FailedMount? 14s? kubelet, master? MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage Output: Running scope as unit run-34341.scope. mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web, ? ? ? ?missing codepage or helper program, or other error ? ? ? ?(for several filesystems (e.g. nfs, cifs) you might ? ? ? ?need a /sbin/mount. helper program) ? ? ? ?In some cases useful info is found in syslog – try ? ? ? ?dmesg | tail or so. ? Warning? FailedMount? 10s? kubelet, master? MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage Output: Running scope as unit run-34364.scope. mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web, ? ? ? ?missing codepage or helper program, or other error ? ? ? ?(for several filesystems (e.g. nfs, cifs) you might ? ? ? ?need a /sbin/mount. helper program) ? ? ? ?In some cases useful info is found in syslog – try ? ? ? ?dmesg | tail or so. ? Warning? FailedMount? 2s? kubelet, master? MountVolume.SetUp failed for volume "httpd-storage" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: –description=Kubernetes transient mount for /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage –scope — mount -t nfs 10.168.xx.xx:/data/web /var/lib/kubelet/pods/4868802e-6c50-11e8-b8b6-b083feea4d48/volumes/kubernetes.io~nfs/httpd-storage Output: Running scope as unit run-34394.scope. mount: wrong fs type, bad option, bad superblock on 10.168.xx.xx:/data/web, ? ? ? ?missing codepage or helper program, or other error ? ? ? ?(for several filesystems (e.g. nfs, cifs) you might ? ? ? ?need a /sbin/mount. helper program) ? ? ? ?In some cases useful info is found in syslog – try ? ? ? ?dmesg | tail or so. 看這個報錯想是容器沒有安裝nfs-common? nfs-utils軟件包安裝導致，容器安裝完這兩個包發現，還是報這個錯誤，經過不懈的google，終于發現需要將這兩個軟件包安裝到宿主機上即可。 yum install nfs-common? nfs-utils -y