[TOC] ## 擴容 1. 查看集群狀態 ```shell $ etcdctl --cacert /data/etcd/certs/ca.pem --cert /data/etcd/certs/etcd.pem --key /data/etcd/certs/etcd-key.pem --endpoints=https://192.168.31.95:2379,https://192.168.31.253:2379 -w table endpoint status +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://192.168.31.95:2379 | a691716a7d43ab3b | 3.4.16 | 7.1 MB | false | false | 422 | 6655386 | 6655386 | | | https://192.168.31.253:2379 | 10f52b9841a7c5f5 | 3.4.16 | 20 MB | true | false | 422 | 6655386 | 6655386 | | +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ ``` 2. 集群添加節點 ```shell $ etcdctl --cacert /data/etcd/certs/ca.pem --cert /data/etcd/certs/etcd.pem --key /data/etcd/certs/etcd-key.pem --endpoints=https://192.168.31.95:2379,https://192.168.31.253:2379 member add etcd02 --peer-urls=https://192.168.31.78:2380 Member 196434b337233d6a added to cluster f14a737ced9d0df5 ETCD_NAME="etcd02" ETCD_INITIAL_CLUSTER="etcd03=https://192.168.31.253:2380,etcd02=https://192.168.31.78:2380,etcd01=https://192.168.31.95:2380" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.31.78:2380" ETCD_INITIAL_CLUSTER_STATE="existing" ``` 3. 安裝etcd ```shell $ scp -r /data/etcd/ 192.168.31.78:/data $ rm -rf /data/etcd/data/* $ cat > /data/etcd/conf/etcd.conf <<-EOF #[Member] ETCD_NAME="etcd02" ETCD_DATA_DIR="/data/etcd/data/" ETCD_LISTEN_PEER_URLS="https://192.168.31.78:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.31.78:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.31.78:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.31.78:2379" ETCD_INITIAL_CLUSTER="etcd01=https://192.168.31.95:2380,etcd02=https://192.168.31.78:2380,etcd03=https://192.168.31.253:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="existing" EOF $ cat > /usr/lib/systemd/system/etcd.service <<-EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify EnvironmentFile=/data/etcd/conf/etcd.conf ExecStart=/data/etcd/bin/etcd \ --cert-file=/data/etcd/certs/etcd.pem \ --key-file=/data/etcd/certs/etcd-key.pem \ --peer-cert-file=/data/etcd/certs/etcd.pem \ --peer-key-file=/data/etcd/certs/etcd-key.pem \ --trusted-ca-file=/data/etcd/certs/ca.pem \ --peer-trusted-ca-file=/data/etcd/certs/ca.pem LimitNOFILE=65536 Restart=always RestartSec=30 StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target EOF ``` > 說明：配置文件修改 `ETCD_NAME`、`ETCD_INITIAL_CLUSTER_STATE` 以及所有涉及的IP地址 4. 啟動etcd ```shell $ systemctl daemon-reload $ systemctl start etcd ``` 5. 檢查 ```shell # 有出現 `finished scheduled compaction` 即可 $ systemctl status etcd.service ● etcd.service - Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2022-01-27 16:21:15 CST; 1min 59s ago Docs: https://github.com/coreos Main PID: 61831 (etcd) Tasks: 21 Memory: 72.3M CGroup: /system.slice/etcd.service └─61831 /data/etcd/bin/etcd --cert-file=/data/etcd/certs/etcd.pem --key-file=/data/etcd/certs/etcd-key.pem --peer-cert-file=/data/etcd/certs/etcd.pem --pee... Jan 27 16:21:08 k8s-node02 etcd[61831]: established a TCP streaming connection with peer 10f52b9841a7c5f5 (stream MsgApp v2 reader) Jan 27 16:21:08 k8s-node02 etcd[61831]: established a TCP streaming connection with peer a691716a7d43ab3b (stream MsgApp v2 reader) Jan 27 16:21:08 k8s-node02 etcd[61831]: established a TCP streaming connection with peer a691716a7d43ab3b (stream Message reader) Jan 27 16:21:15 k8s-node02 etcd[61831]: publish error: etcdserver: request timed out, possibly due to connection lost Jan 27 16:21:15 k8s-node02 etcd[61831]: published {Name:etcd02 ClientURLs:[https://192.168.31.78:2379]} to cluster f14a737ced9d0df5 Jan 27 16:21:15 k8s-node02 etcd[61831]: ready to serve client requests Jan 27 16:21:15 k8s-node02 systemd[1]: Started Etcd Server. Jan 27 16:21:15 k8s-node02 etcd[61831]: serving client requests on 192.168.31.78:2379 Jan 27 16:22:22 k8s-node02 etcd[61831]: store.index: compact 5848210 Jan 27 16:22:22 k8s-node02 etcd[61831]: finished scheduled compaction at 5848210 (took 13.220051ms) $ etcdctl --cacert /data/etcd/certs/ca.pem --cert /data/etcd/certs/etcd.pem --key /data/etcd/certs/etcd-key.pem --endpoints=https://192.168.31.95:2379,https://192.168.31.78:2379,https://192.168.31.253:2379 -w table endpoint health +-----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +-----------------------------+--------+-------------+-------+ | https://192.168.31.253:2379 | true | 13.970304ms | | | https://192.168.31.95:2379 | true | 15.791288ms | | | https://192.168.31.78:2379 | true | 14.993178ms | | +-----------------------------+--------+-------------+-------+ $ etcdctl --cacert /data/etcd/certs/ca.pem --cert /data/etcd/certs/etcd.pem --key /data/etcd/certs/etcd-key.pem --endpoints=https://192.168.31.95:2379,https://192.168.31.78:2379,https://192.168.31.253:2379 -w table member list +------------------+---------+--------+-----------------------------+-----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+--------+-----------------------------+-----------------------------+------------+ | 10f52b9841a7c5f5 | started | etcd03 | https://192.168.31.253:2380 | https://192.168.31.253:2379 | false | | 196434b337233d6a | started | etcd02 | https://192.168.31.78:2380 | https://192.168.31.78:2379 | false | | a691716a7d43ab3b | started | etcd01 | https://192.168.31.95:2380 | https://192.168.31.95:2379 | false | +------------------+---------+--------+-----------------------------+-----------------------------+------------+ ``` ## 縮容 1. 查看成員ID信息 ```shell $ etcdctl --cacert /data/etcd/certs/ca.pem --cert /data/etcd/certs/etcd.pem --key /data/etcd/certs/etcd-key.pem --endpoints=https://192.168.31.95:2379,https://192.168.31.78:2379,https://192.168.31.253:2379 -w table member list +------------------+---------+--------+-----------------------------+-----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+--------+-----------------------------+-----------------------------+------------+ | 10f52b9841a7c5f5 | started | etcd03 | https://192.168.31.253:2380 | https://192.168.31.253:2379 | false | | a691716a7d43ab3b | started | etcd01 | https://192.168.31.95:2380 | https://192.168.31.95:2379 | false | | f833bfe4fb9c10d3 | started | etcd02 | https://192.168.31.78:2380 | https://192.168.31.78:2379 | false | +------------------+---------+--------+-----------------------------+-----------------------------+------------+ ``` 2. 停止etcd服務 ```shell $ systemctl stop etcd ``` 3. 刪除成員 ```shell $ etcdctl --cacert /data/etcd/certs/ca.pem --cert /data/etcd/certs/etcd.pem --key /data/etcd/certs/etcd-key.pem --endpoints=https://192.168.31.95:2379,https://192.168.31.78:2379,https://192.168.31.253:2379 -w table endpoint status {"level":"warn","ts":"2022-01-27T15:55:58.403+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://192.168.31.78:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.31.78:2379: connect: connection refused\""} Failed to get the status of endpoint https://192.168.31.78:2379 (context deadline exceeded) +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://192.168.31.95:2379 | a691716a7d43ab3b | 3.4.16 | 7.1 MB | false | false | 422 | 6654918 | 6654918 | | | https://192.168.31.253:2379 | 10f52b9841a7c5f5 | 3.4.16 | 20 MB | true | false | 422 | 6654931 | 6654931 | | +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ $ etcdctl --cacert /data/etcd/certs/ca.pem --cert /data/etcd/certs/etcd.pem --key /data/etcd/certs/etcd-key.pem --endpoints=https://192.168.31.95:2379,https://192.168.31.78:2379,https://192.168.31.253:2379 -w table member list +------------------+---------+--------+-----------------------------+-----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+--------+-----------------------------+-----------------------------+------------+ | 10f52b9841a7c5f5 | started | etcd03 | https://192.168.31.253:2380 | https://192.168.31.253:2379 | false | | a691716a7d43ab3b | started | etcd01 | https://192.168.31.95:2380 | https://192.168.31.95:2379 | false | | f833bfe4fb9c10d3 | started | etcd02 | https://192.168.31.78:2380 | https://192.168.31.78:2379 | false | +------------------+---------+--------+-----------------------------+-----------------------------+------------+ $ etcdctl --cacert /data/etcd/certs/ca.pem --cert /data/etcd/certs/etcd.pem --key /data/etcd/certs/etcd-key.pem --endpoints=https://192.168.31.95:2379,https://192.168.31.78:2379,https://192.168.31.253:2379 member remove f833bfe4fb9c10d3 Member f833bfe4fb9c10d3 removed from cluster f14a737ced9d0df5 ``` 4. 清理數據目錄 ```shell rm -rf /data/etcd /usr/lib/systemd/system/etcd.service ``` 5. 驗證 ```shell # 查看成員 --endpoints 寫全，也只能看到兩個成員 $ etcdctl --cacert /data/etcd/certs/ca.pem --cert /data/etcd/certs/etcd.pem --key /data/etcd/certs/etcd-key.pem --endpoints=https://192.168.31.95:2379,https://192.168.31.78:2379,https://192.168.31.253:2379 -w table member list +------------------+---------+--------+-----------------------------+-----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+--------+-----------------------------+-----------------------------+------------+ | 10f52b9841a7c5f5 | started | etcd03 | https://192.168.31.253:2380 | https://192.168.31.253:2379 | false | | a691716a7d43ab3b | started | etcd01 | https://192.168.31.95:2380 | https://192.168.31.95:2379 | false | +------------------+---------+--------+-----------------------------+-----------------------------+------------+ # 檢查成員健康 --endpoints 不寫刪除的成員 $ etcdctl --cacert /data/etcd/certs/ca.pem --cert /data/etcd/certs/etcd.pem --key /data/etcd/certs/etcd-key.pem --endpoints=https://192.168.31.95:2379,https://192.168.31.253:2379 -w table endpoint status +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://192.168.31.95:2379 | a691716a7d43ab3b | 3.4.16 | 7.1 MB | false | false | 422 | 6655386 | 6655386 | | | https://192.168.31.253:2379 | 10f52b9841a7c5f5 | 3.4.16 | 20 MB | true | false | 422 | 6655386 | 6655386 | | +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ ```