## 部署Flannel網絡 kubernetes 要求集群內各節點能通過Pod 網段互聯互通，下面我們來使用Flannel 在所有節點上創建互聯互通的Pod 網段的步驟。 > 需要在所有的Node節點安裝。 #### 環境變量 ```shell $ export NODE_IP=192.168.10.61 # 當前部署node節點的ip # 導入全局變量 $ source /usr/k8s/bin/env.sh ``` #### 創建TLS密鑰和證書 etcd集群啟用了雙向TLS認證，所以需要flanneld指定與etcd集群通信的CA和密鑰。 創建flanneld證書簽名請求： ```shell $ cat > flanneld-csr.json <<EOF { "CN": "flanneld", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF ``` 生成flanneld證書和私鑰： ```shell $ cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \ -ca-key=/etc/kubernetes/ssl/ca-key.pem \ -config=/etc/kubernetes/ssl/ca-config.json \ -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld $ ls flanneld* flanneld.csr flanneld-csr.json flanneld-key.pem flanneld.pem $ sudo mkdir -p /etc/flanneld/ssl $ sudo mv flanneld*.pem /etc/flanneld/ssl ``` #### 向etcd寫入集群Pod網段信息 > 該步驟只需在第一次部署Flannel 網絡時執行，后續在其他節點上部署Flanneld 時無需再寫入該信息 ```shell $ etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/flanneld/ssl/flanneld.pem \ --key-file=/etc/flanneld/ssl/flanneld-key.pem \ set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}' # 得到如下反饋信息 {"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}} ``` - 寫入的 Pod 網段(${CLUSTER_CIDR}，172.30.0.0/16) 必須與`kube-controller-manager` 的 `--cluster-cidr` 選項值一致； #### 安裝和配置flanneld ```shell $ mkdir flannel $ wget https://github.com/coreos/flannel/releases/download/v0.9.0/flannel-v0.9.0-linux-amd64.tar.gz $ tar -xzvf flannel-v0.9.0-linux-amd64.tar.gz -C flannel $ sudo cp flannel/{flanneld,mk-docker-opts.sh} /usr/k8s/bin ``` 創建flanneld的systemd unit文件 ```shell $ cat > flanneld.service << EOF [Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service [Service] Type=notify ExecStart=/usr/k8s/bin/flanneld \\ -etcd-cafile=/etc/kubernetes/ssl/ca.pem \\ -etcd-certfile=/etc/flanneld/ssl/flanneld.pem \\ -etcd-keyfile=/etc/flanneld/ssl/flanneld-key.pem \\ -etcd-endpoints=${ETCD_ENDPOINTS} \\ -etcd-prefix=${FLANNEL_ETCD_PREFIX} ExecStartPost=/usr/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker Restart=on-failure [Install] WantedBy=multi-user.target RequiredBy=docker.service EOF ``` 192.168.10.61 / kube-node-61的flanneld.service文件如下 ```shell [Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service [Service] Type=notify ExecStart=/usr/k8s/bin/flanneld \ -etcd-cafile=/etc/kubernetes/ssl/ca.pem \ -etcd-certfile=/etc/flanneld/ssl/flanneld.pem \ -etcd-keyfile=/etc/flanneld/ssl/flanneld-key.pem \ -etcd-endpoints=https://192.168.10.65:2379,https://192.168.10.64:2379,https://192.168.10.63:2379 \ -etcd-prefix=/kubernetes/network ExecStartPost=/usr/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker Restart=on-failure [Install] WantedBy=multi-user.target RequiredBy=docker.service ``` #### 啟動flanneld ```shell $ sudo cp flanneld.service /etc/systemd/system/ $ sudo systemctl daemon-reload $ sudo systemctl enable flanneld $ sudo systemctl start flanneld $ systemctl status flanneld ``` #### 檢查flanneld服務 ```shell ifconfig flanneld.1 ``` **檢查分配給各flanneld的Pod網段信息** ```shell # 查看集群 Pod 網段(/16) $ etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/flanneld/ssl/flanneld.pem \ --key-file=/etc/flanneld/ssl/flanneld-key.pem \ get ${FLANNEL_ETCD_PREFIX}/config { "Network": "172.30.0.0/16", "SubnetLen": 24, "Backend": { "Type": "vxlan" } } # 查看已分配的 Pod 子網段列表(/24) $ etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/flanneld/ssl/flanneld.pem \ --key-file=/etc/flanneld/ssl/flanneld-key.pem \ ls ${FLANNEL_ETCD_PREFIX}/subnets /kubernetes/network/subnets/172.30.77.0-24 # 查看某一 Pod 網段對應的 flanneld 進程監聽的 IP 和網絡參數 $ etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/flanneld/ssl/flanneld.pem \ --key-file=/etc/flanneld/ssl/flanneld-key.pem \ get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.77.0-24 {"PublicIP":"192.168.1.137","BackendType":"vxlan","BackendData":{"VtepMAC":"62:fc:03:83:1b:2b"}} ``` **確保各節點間Pod網段能互聯互通** ```shell $ etcdctl \ --endpoints=${ETCD_ENDPOINTS} \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/flanneld/ssl/flanneld.pem \ --key-file=/etc/flanneld/ssl/flanneld-key.pem \ ls ${FLANNEL_ETCD_PREFIX}/subnets ```