一、自簽APIServer SSL證書 1、 進入TLS目錄 1.1、執行shell： ``` generate_k8s_cert.sh ``` shell命令的內容是： ``` cfssl gencert -initca ca-csr.json | cfssljson -bare ca - cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy ``` 1.2、 server-csr.json node節點不用配置 ``` { ??? "CN": "kubernetes", ??? "hosts": [ ????? "10.0.0.1", ????? "127.0.0.1", ????? "kubernetes", ????? "kubernetes.default", ????? "kubernetes.default.svc", ????? "kubernetes.default.svc.cluster", ????? "kubernetes.default.svc.cluster.local", ????? "192.168.72.166", ????? "192.168.72.167", ????? "192.168.72.170", ????? "192.168.72.171", ????? "192.168.72.172" ??? ], ??? "key": { ??????? "algo": "rsa", ??????? "size": 2048 ??? }, ??? "names": [ ??????? { ??????????? "C": "CN", ??????????? "L": "BeiJing", ??????????? "ST": "BeiJing", ??????????? "O": "k8s", ??????????? "OU": "System" ??????? } ??? ] } ``` 1.3、 kube-proxy-csr.json ``` { ? "CN": "system:kube-proxy", ? "hosts": [], ? "key": { ??? "algo": "rsa", ??? "size": 2048 ? }, ? "names": [ ??? { ????? "C": "CN", ????? "L": "BeiJing", ????? "ST": "BeiJing", ????? "O": "k8s", ????? "OU": "System" ??? } ? ] } ``` 2、生產證書 ``` ./generate_k8s_cert.sh ``` 二、部署Master組件 1、下載 https://github.com/kubernetes/kubernetes/releases/tag/v1.17.4 https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.17.md#server-binaries https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/ 2、Master組件 2.1、 kube-apiserver 1） kube-apiserver.service ``` [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target ``` 2）kube-apiserver.conf ``` KUBE_APISERVER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --etcd-servers=https://192.168.254.201:2379,https://192.168.254.203:2379,https://192.168.254.204:2379 \ --bind-address=192.168.254.201 \ #監聽的地址 --secure-port=6443 \ --advertise-address=192.168.254.201 \ #通告的地址 --allow-privileged=true \ #所有權限 --service-cluster-ip-range=10.0.0.0/24 \ #service的ip范圍 --service-node-port-range=1-65535 \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \ #準入的插件 --authorization-mode=RBAC,Node \ #授權的模式 --enable-bootstrap-token-auth=true \ #自動頒發證書 --token-auth-file=/opt/kubernetes/cfg/token.csv \ --service-node-port-range=30000-32767 \ #暴露的端口范圍 --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \ --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \ --tls-cert-file=/opt/kubernetes/ssl/server.pem? \ --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \ --client-ca-file=/opt/kubernetes/ssl/ca.pem \ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \ --etcd-cafile=/opt/etcd/ssl/ca.pem \ --etcd-certfile=/opt/etcd/ssl/server.pem \ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log" ``` 2.2、kube-controller-manager 1）kube-controller-manager ``` [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target ``` 2）kube-controller-manager.conf ``` KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --leader-elect=true \ #集群的選舉 --master=127.0.0.1:8080 \ --address=127.0.0.1 \ --allocate-node-cidrs=true \ --cluster-cidr=10.244.0.0/16 \ --service-cluster-ip-range=10.0.0.0/24?\ --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem? \ --root-ca-file=/opt/kubernetes/ssl/ca.pem \ --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \ --experimental-cluster-signing-duration=87600h0m0s" ``` 2.3、 kube-scheduler 1）kube-scheduler.service ``` [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target ``` 2）kube-scheduler.conf ``` KUBE_SCHEDULER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --leader-elect \ --master=127.0.0.1:8080 \ --address=127.0.0.1" ``` 3、實驗 3.1、上傳tar并且解壓 3.2、拷貝證書到ssl里面 我們拷貝生成的證書到kubernetes中的ssl文件中 ``` cp /yhj/TLS/k8s/*.pem /yhj/kubernetes/ssl/ ```  3.3、cp -rf kubernetes /opt  3.4、拷貝service到執行目錄下 ``` cp kube-apiserver.service kube-controller-manager.service kube-scheduler.service /usr/lib/systemd/system ```  3.5、啟動 ``` # systemctl start kube-apiserver # systemctl start kube-controller-manager # systemctl start kube-scheduler # systemctl enable kube-apiserver # systemctl enable kube-controller-manager # systemctl enable kube-scheduler #systemctl restart kube-apiserver for i in $(ls /opt/kubernetes/bin);do systemctl enable $i;done ``` 3.6、日志查看 ``` less /opt/kubernetes/logs ``` ``` cd /opt/kubernetes/logs ```  ``` less kube-apiserver.INFO ``` 3.7、開機啟動 subtopic 3.8、kubectl get node 3.9、查看 ``` cd /opt/kubernetes/bin/ ```  ``` ./kubectl get cs ```  3.10、驗證 ``` ps -ef | grep kube ps -ef | grep kube | wc -4 ``` 4、 授權啟用TLS Bootstrapping 4.1、文件拷貝  4.2、cat /opt/kubernetes/cfg/token.csv 格式：token,用戶,uid,用戶組 c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper" 4.3、給kubelet-bootstrap授權： ``` kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap ``` 4.4、token也可自行生成替換： ``` head -c 16 /dev/urandom | od -An -t x | tr -d ' ' ``` 注意：我們如果需要最新的，我們只需要下載最新版，然后替換掉bin目錄中的文件即可