## 部署高可用etcd集群 kubernetes 系統使用`etcd`存儲所有的數據，我們這里部署3個節點的etcd 集群，這3個節點分別用192.168.10.65、 192.168.10.64、 192.168.10.63，分別命名為`kube-node-65`、`kube-node-64`、`kube-node-63`: * kube-node-65 / 192.168.10.65 * kube-node-64 / 192.168.10.64 * kube-node-63 / 192.168.10.63 #### 定義環境變量 ```shell $ export NODE_NAME=kube-node-65 # 當前部署的機器名稱(隨便定義，只要能區分不同機器即可) $ export NODE_IP=192.168.10.65 # 當前部署的機器IP $ export NODE_IPS="192.168.10.65 192.168.10.64 192.168.10.63" # etcd 集群所有機器 IP # etcd 集群間通信的IP和端口 $ export ETCD_NODES=etcd01=kube-node-65=https://192.168.10.65:2380,kube-node-64=https://192.168.10.64:2380,kube-node-63=https://192.168.10.63:2380 $ # 導入用到的其它全局變量：ETCD_ENDPOINTS、FLANNEL_ETCD_PREFIX、CLUSTER_CIDR $ source /usr/k8s/bin/env.sh ``` #### 下載etcd二進制文件 ```shell $ wget https://github.com/coreos/etcd/releases/download/v3.2.9/etcd-v3.2.9-linux-amd64.tar.gz $ tar -xvf etcd-v3.2.9-linux-amd64.tar.gz $ sudo mv etcd-v3.2.9-linux-amd64/etcd* /usr/k8s/bin/ ``` #### 創建TLS密鑰和證書 > 為了保證通信安全，客戶端與etcd集群、etcd集群之間的通信需要使用TLS加密。 **創建etcd證書簽名請求：** ```json $ cat > etcd-csr.json <<EOF { "CN": "etcd", "hosts": [ "127.0.0.1", "${NODE_IP}" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF ``` **生成etcd證書和密鑰** ```shell $ cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \ -ca-key=/etc/kubernetes/ssl/ca-key.pem \ -config=/etc/kubernetes/ssl/ca-config.json \ -profile=kubernetes etcd-csr.json | cfssljson -bare etcd $ ls etcd* etcd.csr etcd-csr.json etcd-key.pem etcd.pem $ sudo mkdir -p /etc/etcd/ssl $ sudo mv etcd*.pem /etc/etcd/ssl/ ``` #### 創建etcd的systemd unit文件 ```shell $ sudo mkdir -p /var/lib/etcd # 必須要先創建工作目錄 $ cat > etcd.service <<EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/usr/k8s/bin/etcd \\ --name=${NODE_NAME} \\ --cert-file=/etc/etcd/ssl/etcd.pem \\ --key-file=/etc/etcd/ssl/etcd-key.pem \\ --peer-cert-file=/etc/etcd/ssl/etcd.pem \\ --peer-key-file=/etc/etcd/ssl/etcd-key.pem \\ --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\ --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\ --initial-advertise-peer-urls=https://${NODE_IP}:2380 \\ --listen-peer-urls=https://${NODE_IP}:2380 \\ --listen-client-urls=https://${NODE_IP}:2379,http://127.0.0.1:2379 \\ --advertise-client-urls=https://${NODE_IP}:2379 \\ --initial-cluster-token=etcd-cluster-0 \\ --initial-cluster=${ETCD_NODES} \\ --initial-cluster-state=new \\ --data-dir=/var/lib/etcd Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF ``` **如kube-node-65機器生成的文件：** ```shell [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/usr/k8s/bin/etcd \ --name=kube-node-65 \ --cert-file=/etc/etcd/ssl/etcd.pem \ --key-file=/etc/etcd/ssl/etcd-key.pem \ --peer-cert-file=/etc/etcd/ssl/etcd.pem \ --peer-key-file=/etc/etcd/ssl/etcd-key.pem \ --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --initial-advertise-peer-urls=https://192.168.10.65:2380 \ --listen-peer-urls=https://192.168.10.65:2380 \ --listen-client-urls=https://192.168.10.65:2379,http://127.0.0.1:2379 \ --advertise-client-urls=https://192.168.10.65:2379 \ --initial-cluster-token=etcd-cluster-0 \ --initial-cluster=kube-node-65=https://192.168.10.65:2380,kube-node-64=https://192.168.10.64:2380,kube-node-63=https://192.168.10.63:2380 \ --initial-cluster-state=new \ --data-dir=/var/lib/etcd Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target ``` #### 啟動etcd服務 ```shell $ sudo mv etcd.service /etc/systemd/system/ $ sudo systemctl daemon-reload $ sudo systemctl enable etcd $ sudo systemctl start etcd $ sudo systemctl status etcd ``` > 最先啟動的etcd 進程會卡住一段時間，等待其他節點啟動加入集群，在所有的etcd 節點重復上面的步驟，直到所有的機器etcd 服務都已經啟動。 #### 驗證etcd服務 部署完etcd集群后，在任一etcd節點上執行下面命令： ```shell for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/k8s/bin/etcdctl \ --endpoints=https://${ip}:2379 \ --cacert=/etc/kubernetes/ssl/ca.pem \ --cert=/etc/etcd/ssl/etcd.pem \ --key=/etc/etcd/ssl/etcd-key.pem \ endpoint health; done ``` 輸出如下 ```yaml https://192.168.10.65:2379 is healthy: successfully committed proposal: took = 963.711μs https://192.168.10.64:2379 is healthy: successfully committed proposal: took = 2.420937ms https://192.168.10.63:2379 is healthy: successfully committed proposal: took = 2.555773ms ``` > 上面的信息顯示3個節點上的etcd均為healthy，則表示集群服務正常