使用kubeadm在Ubuntu Server 16.04上快速構建測試集群 · kubernetes完整攻略

# 用kubeadm在Ubuntu上快速構建Kubernetes測試集群本文將介紹如何在Ubuntu server 16.04版本上安裝kubeadm，并利用kubeadm快速的在Ubuntu server 版本 16.04上構建一個kubernetes的基礎的測試集群，用來做學習和測試用途，當前（2018-04-14）最新的版本是1.10.1。參考文檔包括kubernetes官方網站的[kubeadm安裝文檔](https://kubernetes.io/docs/setup/independent/install-kubeadm/)以及[利用kubeadm創建集群](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)這兩個文檔。生產用途的環境，需要考慮各個組件的高可用，建議參考Kubernetes的官方的相關的安裝文檔。 ## 概述本次安裝建議至少4臺服務器或者虛擬機，每臺服務器4G內存，2個CPU核心以上，基本架構為1臺master節點，3臺slave節點。整個安裝過程將在Ubuntu服務器上安裝完kubeadm，以及安裝kubernetes的基本集群，包括canal網絡，另后臺存儲可參考本書的最佳實踐中的存儲管理內容。本次安裝一共4個節點，節點信息如下: | 角色 | 主機名 | IP地址 | |----------|----------- |------------| | Master | Ubuntu-master | 192.168.5.200 | | Slave | ubuntu-1 | 192.168.5.201 | | Slave | ubuntu-2 | 192.168.5.202 | | Slave | ubuntu-3 | 192.168.5.203 | ## 準備工作 - 默認方式安裝Ubuntu Server 版本 16.04 - 配置主機名映射，每個節點 ```bash # cat /etc/hosts 127.0.0.1 localhost 192.168.0.200 Ubuntu-master 192.168.0.201 Ubuntu-1 192.168.0.202 Ubuntu-2 192.168.0.203 Ubuntu-3 ``` * 如果連接gcr網站不方便，無法下載鏡像，會導致安裝過程卡住，可以下載我導出的鏡像包，[我導出的鏡像網盤鏈接](https://pan.baidu.com/s/1ZJFRt_UNCQvwcu9UENr_gw)，解壓縮以后是多個個tar包，使用```docker load< xxxx.tar``` 導入各個文件即可）。 ## 在所有節點上安裝kubeadm 查看apt安裝源如下配置，使用阿里云的系統和kubernetes的源。 ```bash $ cat /etc/apt/sources.list # 系統安裝源 deb http://mirrors.aliyun.com/ubuntu/ xenial main restricted deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted deb http://mirrors.aliyun.com/ubuntu/ xenial universe deb http://mirrors.aliyun.com/ubuntu/ xenial-updates universe deb http://mirrors.aliyun.com/ubuntu/ xenial multiverse deb http://mirrors.aliyun.com/ubuntu/ xenial-updates multiverse deb http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse # kubeadm及kubernetes組件安裝源 deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main ``` 安裝docker，可以使用系統源的的docker.io軟件包，版本1.13.1，我的系統里是已經安裝好最新的版本了。 ```bash # apt-get install docker.io Reading package lists... Done Building dependency tree Reading state information... Done docker.io is already the newest version (1.13.1-0ubuntu1~16.04.2). 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. ``` 更新源，可以不理會gpg的報錯信息。 ```bash # apt-get update Hit:1 http://mirrors.aliyun.com/ubuntu xenial InRelease Hit:2 http://mirrors.aliyun.com/ubuntu xenial-updates InRelease Hit:3 http://mirrors.aliyun.com/ubuntu xenial-backports InRelease Get:4 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease [8,993 B] Ign:4 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease Fetched 8,993 B in 0s (20.7 kB/s) Reading package lists... Done W: GPG error: https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6A030B21BA07F4FB W: The repository 'https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease' is not signed. N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use. N: See apt-secure(8) manpage for repository creation and user configuration details. ``` 強制安裝kubeadm，kubectl，kubelet軟件包。 ```bash # apt-get install -y kubelet kubeadm kubectl --allow-unauthenticated Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: kubernetes-cni socat The following NEW packages will be installed: kubeadm kubectl kubelet kubernetes-cni socat 0 upgraded, 5 newly installed, 0 to remove and 4 not upgraded. Need to get 56.9 MB of archives. After this operation, 410 MB of additional disk space will be used. WARNING: The following packages cannot be authenticated! kubernetes-cni kubelet kubectl kubeadm Authentication warning overridden. Get:1 http://mirrors.aliyun.com/ubuntu xenial/universe amd64 socat amd64 1.7.3.1-1 [321 kB] Get:2 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.6.0-00 [5,910 kB] Get:3 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubelet amd64 1.10.1-00 [21.1 MB] Get:4 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubectl amd64 1.10.1-00 [8,906 kB] Get:5 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubeadm amd64 1.10.1-00 [20.7 MB] Fetched 56.9 MB in 5s (11.0 MB/s) Use of uninitialized value $_ in lc at /usr/share/perl5/Debconf/Template.pm line 287. Selecting previously unselected package kubernetes-cni. (Reading database ... 191799 files and directories currently installed.) Preparing to unpack .../kubernetes-cni_0.6.0-00_amd64.deb ... Unpacking kubernetes-cni (0.6.0-00) ... Selecting previously unselected package socat. Preparing to unpack .../socat_1.7.3.1-1_amd64.deb ... Unpacking .... .... ``` kubeadm安裝完以后，就可以使用它來快速安裝部署Kubernetes集群了。 ## 使用kubeadm安裝Kubernetes集群在做好了準備工作之后，下面介紹如何使用 kubeadm 安裝 Kubernetes 集群，我們將首先安裝 master 節點，然后將 slave 節點一個個加入到集群中去。 ### 使用kubeadmin初始化master節點因為使用要使用canal，因此需要在初始化時加上網絡配置參數,設置kubernetes的子網為10.244.0.0/16，注意此處不要修改為其他地址，因為這個值與后續的canal的yaml值要一致，如果修改，請一并修改。這個下載鏡像的過程涉及翻墻，因為會從gcr的站點下載容器鏡像。。。（如果大家翻墻不方便的話，可以用我在上文準備工作中提到的導出的鏡像）。如果有能夠連接gcr站點的網絡，那么整個安裝過程非常簡單。 ```bash # kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.0.200 [init] Using Kubernetes version: v1.10.1 [init] Using Authorization modes: [Node RBAC] [preflight] Running pre-flight checks. [WARNING FileExisting-crictl]: crictl not found in system path Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl [preflight] Starting the kubelet service [certificates] Generated ca certificate and key. [certificates] Generated apiserver certificate and key. [certificates] apiserver serving cert is signed for DNS names [ubuntu-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.200] [certificates] Generated apiserver-kubelet-client certificate and key. [certificates] Generated etcd/ca certificate and key. [certificates] Generated etcd/server certificate and key. [certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1] [certificates] Generated etcd/peer certificate and key. [certificates] etcd/peer serving cert is signed for DNS names [ubuntu-master] and IPs [192.168.0.200] [certificates] Generated etcd/healthcheck-client certificate and key. [certificates] Generated apiserver-etcd-client certificate and key. [certificates] Generated sa key and public key. [certificates] Generated front-proxy-ca certificate and key. [certificates] Generated front-proxy-client certificate and key. [certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf" [controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml" [controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml" [controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml" [etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml" [init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests". [init] This might take a minute or longer if the control plane images have to be pulled. [apiclient] All control plane components are healthy after 28.003828 seconds [uploadconfig]?Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [markmaster] Will mark node ubuntu-master as master by adding a label and a taint [markmaster] Master ubuntu-master tainted and labelled with key/value: node-role.kubernetes.io/master="" [bootstraptoken] Using token: rw4enn.mvk547juq7qi2b5f [bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: kube-dns [addons] Applied essential addon: kube-proxy Your Kubernetes master has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join 192.168.0.200:6443 --token rw4enn.mvk547juq7qi2b5f --discovery-token-ca-cert-hash sha256:ba260d5191213382a806a9a7d92c9e6bb09061847c7914b1ac584d0c69471579 ``` 執行如下命令來配置kubectl。 ```bash mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` 這樣master的節點就配置好了，并且可以使用kubectl來進行各種操作了，根據上面的提示接著往下做，將slave節點加入到集群。 ### Slave節點加入集群在slave節點執行如下的命令,將slave節點加入集群，正常的返回信息如下： ```bash #kubeadm join 192.168.0.200:6443 --token rw4enn.mvk547juq7qi2b5f --discovery-token-ca-cert-hash sha256:ba260d5191213382a806a9a7d92c9e6bb09061847c7914b1ac584d0c69471579 [preflight] Running pre-flight checks. [WARNING FileExisting-crictl]: crictl not found in system path Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl [discovery] Trying to connect to API Server "192.168.0.200:6443" [discovery] Created cluster-info discovery client, requesting info from "https://192.168.0.200:6443" [discovery] Requesting info from "https://192.168.0.200:6443" again to validate TLS against the pinned public key [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.0.200:6443" [discovery] Successfully established connection with API Server "192.168.0.200:6443" This node has joined the cluster: * Certificate signing request was sent to master and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the master to see this node join the cluster. ``` 等待節點加入完畢。加入中狀態。 ```bash # kubectl get node NAME STATUS ROLES AGE VERSION ubuntu-1 NotReady <none> 6m v1.10.1 ubuntu-2 NotReady <none> 6m v1.10.1 ubuntu-3 NotReady <none> 6m v1.10.1 ubuntu-master NotReady master 10m v1.10.1 ``` 在master節點查看信息如下狀態為節點加入完畢。 ```bash root@Ubuntu-master:~# kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE etcd-ubuntu-master 1/1 Running 0 21m 192.168.0.200 ubuntu-master kube-apiserver-ubuntu-master 1/1 Running 0 21m 192.168.0.200 ubuntu-master kube-controller-manager-ubuntu-master 1/1 Running 0 22m 192.168.0.200 ubuntu-master kube-dns-86f4d74b45-wkfk2 0/3 Pending 0 22m <none> <none> kube-proxy-6ddb4 1/1 Running 0 22m 192.168.0.200 ubuntu-master kube-proxy-7ngb9 1/1 Running 0 17m 192.168.0.202 ubuntu-2 kube-proxy-fkhhx 1/1 Running 0 18m 192.168.0.201 ubuntu-1 kube-proxy-rh4lq 1/1 Running 0 18m 192.168.0.203 ubuntu-3 kube-scheduler-ubuntu-master 1/1 Running 0 21m 192.168.0.200 ubuntu-master ``` kubedns組件需要在網絡插件完成安裝以后會自動安裝完成。 ## 安裝網絡插件canal 從[canal官方文檔參考](https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/canal/)，如下網址下載2個文件并且安裝，其中一個是配置canal的RBAC權限，一個是部署canal的DaemonSet。 ```bash # kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/canal/rbac.yaml clusterrole.rbac.authorization.k8s.io "calico" created clusterrole.rbac.authorization.k8s.io "flannel" created clusterrolebinding.rbac.authorization.k8s.io "canal-flannel" created clusterrolebinding.rbac.authorization.k8s.io "canal-calico" created ``` ```bash # kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/canal/canal.yaml configmap "canal-config" created daemonset.extensions "canal" created customresourcedefinition.apiextensions.k8s.io "felixconfigurations.crd.projectcalico.org" created customresourcedefinition.apiextensions.k8s.io "bgpconfigurations.crd.projectcalico.org" created customresourcedefinition.apiextensions.k8s.io "ippools.crd.projectcalico.org" created customresourcedefinition.apiextensions.k8s.io "clusterinformations.crd.projectcalico.org" created customresourcedefinition.apiextensions.k8s.io "globalnetworkpolicies.crd.projectcalico.org" created customresourcedefinition.apiextensions.k8s.io "networkpolicies.crd.projectcalico.org" created serviceaccount "canal" created ``` 查看canal的安裝狀態。 ```bash # kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE canal-fc94k 3/3 Running 10 4m 192.168.0.201 ubuntu-1 canal-rs2wp 3/3 Running 10 4m 192.168.0.200 ubuntu-master canal-tqd4l 3/3 Running 10 4m 192.168.0.202 ubuntu-2 canal-vmpnr 3/3 Running 10 4m 192.168.0.203 ubuntu-3 etcd-ubuntu-master 1/1 Running 0 28m 192.168.0.200 ubuntu-master kube-apiserver-ubuntu-master 1/1 Running 0 28m 192.168.0.200 ubuntu-master kube-controller-manager-ubuntu-master 1/1 Running 0 29m 192.168.0.200 ubuntu-master kube-dns-86f4d74b45-wkfk2 3/3 Running 0 28m 10.244.2.2 ubuntu-3 kube-proxy-6ddb4 1/1 Running 0 28m 192.168.0.200 ubuntu-master kube-proxy-7ngb9 1/1 Running 0 24m 192.168.0.202 ubuntu-2 kube-proxy-fkhhx 1/1 Running 0 24m 192.168.0.201 ubuntu-1 kube-proxy-rh4lq 1/1 Running 0 24m 192.168.0.203 ubuntu-3 kube-scheduler-ubuntu-master 1/1 Running 0 28m 192.168.0.200 ubuntu-master ``` 可以看到canal和kube-dns都已經運行正常，一個基本功能正常的測試環境就部署完畢了。此時查看集群的節點狀態，版本為最新的版本v1.10.1。 ```bash # kubectl get node NAME STATUS ROLES AGE VERSION ubuntu-1 Ready <none> 27m v1.10.1 ubuntu-2 Ready <none> 27m v1.10.1 ubuntu-3 Ready <none> 27m v1.10.1 ubuntu-master Ready master 31m v1.10.1 ``` 讓master也運行pod（默認master不運行pod）,這樣在測試環境做是可以的，不建議在生產環境如此操作。 ```bash #kubectl taint nodes --all node-role.kubernetes.io/master- node "ubuntu-master" untainted taint "node-role.kubernetes.io/master:" not found taint "node-role.kubernetes.io/master:" not found taint "node-role.kubernetes.io/master:" not found ``` 后續如果想要集群其他功能啟用，請參考后續文章。 ## 參考 - [Overview of kubeadm](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/)