9、高可用部署 · Kubernetes部署和實戰

一、部署master組件 1、基本環境搭建 ``` scp -r /opt/kubernetes root@192.168.254.202:/opt scp -r /opt/etcd/ssl root@192.168.254.202:/opt/etcd scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.254.202:/usr/lib/systemd/system scp /usr/bin/kubectl root@192.168.254.202:/usr/bin ``` 2、修改配置文件 ``` # cat /opt/kubernetes/cfg/kube-apiserver.conf? KUBE_APISERVER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --etcd-servers=https://192.168.254.201:2379,https://192.168.254.202:2379,https://192.168.254.203:2379 \ --bind-address=192.168.254.202?\ --secure-port=6443 \ --advertise-address=192.168.254.202?\ ``` 3、啟動 ``` systemctl start kube-apiserver systemctl start kube-controller-manager systemctl start kube-scheduler systemctl enable kube-apiserver systemctl enable kube-controller-manager systemctl enable kube-scheduler systemctl daemon-reload ``` ``` for i in $(ls /opt/kubernetes/bin/);do systemctl start $i;systemctl enable $i;done ``` 4、驗證 4.1、查看 ``` ps -ef | grep kube ``` ![](https://img.kancloud.cn/5c/74/5c7402835a5396b9537b944c39ec5193_946x174.png) 4.2、執行命令 ``` kubectl get node ``` 二、部署nginx負載均衡 1、下載 http://nginx.org/packages/rhel/7/x86\_64/RPMS/ http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm 2、建立nginx的yum倉庫 rpm -ivh nginx-release-centos-7-0.el7.ngx.noarch.rpm 3、下載并安裝nginx yum install nginx 4、修改配置文件 ``` # vim /etc/nginx/nginx.conf …… stream { ??? log_format? main? '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent'; ??? access_log? /var/log/nginx/k8s-access.log? main; ??? upstream k8s-apiserver { ??????????????? server?192.168.254.201:6443; ??????????????? server?192.168.254.202:6443; ??????????? } ??? server { ?????? listen 6443; ?????? proxy_pass k8s-apiserver; ??? } } ``` 5、啟動nginx服務 ``` systemctl start nginx ``` 或 ``` service nginx start ``` 6、開機啟動 ``` systemctl enable nginx ``` 7、驗證 ![](https://img.kancloud.cn/d5/f7/d5f7540f9cc04c679e939d79c03f120c_1572x579.png) 查看進程 ``` ps -ef | grep nginx ``` 三、 keepalived高可用 1、主節點 1.1、安裝 yum install -y keepalived 1.2、修改配置文件 ``` # vi /etc/keepalived/keepalived.conf global_defs {? ?? notification_email {? ???? acassen@firewall.loc? ???? failover@firewall.loc? ???? sysadmin@firewall.loc? ?? }? ?? notification_email_from Alexandre.Cassen@firewall.loc ? ?? smtp_server 127.0.0.1? ?? smtp_connect_timeout 30? ?? router_id NGINX_MASTER }? vrrp_script check_nginx { ??? script "/etc/keepalived/check_nginx.sh" } vrrp_instance VI_1 {? ??? state MASTER? ??? interface?ens33 ??? virtual_router_id 51 # VRRP 路由 ID實例，每個實例是唯一的? ??? priority?100 ???# 優先級，備服務器設置 90? ??? advert_int 1 ?? # 指定VRRP 心跳包通告間隔時間，默認1秒? ??? authentication {? ??????? auth_type PASS ????? ??????? auth_pass 1111? ??? } ? ??? virtual_ipaddress {? 192.168.254.200/24 ??? }? ??? track_script { ??????? check_nginx ??? }? } # cat /etc/keepalived/check\_nginx.sh? #!/bin/bash count=$(ps -ef |grep nginx |egrep -cv "grep|$$") if [ "$count" -eq 0 ];then ??? exit 1 else ??? exit 0 fi ``` 1.3、執行 ``` chmod +x check\_nginx.sh systemctl start keepalived systemctl enable keepalived ``` 2、備用節點 2.1、安裝 yum install -y keepalived 2.2、修改配置文件 ``` # cat /etc/keepalived/keepalived.conf? global_defs {? ?? notification_email {? ???? acassen@firewall.loc? ???? failover@firewall.loc? ???? sysadmin@firewall.loc? ?? }? ?? notification_email_from Alexandre.Cassen@firewall.loc?? ?? smtp_server 127.0.0.1? ?? smtp_connect_timeout 30? ?? router_id NGINX_BACKUP }? vrrp_script check_nginx { ??? script "/etc/keepalived/check_nginx.sh" } vrrp_instance VI_1 {? ??? state BACKUP? ??? interface ens33 ??? virtual_router_id 51 # VRRP 路由 ID實例，每個實例是唯一的? ??? priority 90??? # 優先級，備服務器設置 90? ??? advert_int 1??? # 指定VRRP 心跳包通告間隔時間，默認1秒? ??? authentication {? ??????? auth_type PASS?????? ??????? auth_pass 1111? ??? }?? ??? virtual_ipaddress {? ??????? 192.168.254.200/24 ??? }? ??? track_script { ??????? check_nginx ??? }? } # cat /etc/keepalived/check_nginx.sh? #!/bin/bash count=$(ps -ef |grep nginx |egrep -cv "grep|$$") if [ "$count" -eq 0 ];then ??? exit 1 else ??? exit 0 fi ``` 3.2、執行 ``` chmod +x check\_nginx.sh systemctl start keepalived systemctl enable keepalived ``` 3、驗證 3.1、查看ip地址 ``` ip addr ``` 3.2、查看進程 ``` ps -ef | grep keep* ``` 3.3、 tail -f /var/log/messages 3.4、停止nginx ``` systemctl stop nginx ``` 四、修改node鏈接vip 1、將Node連接VIP： ``` # cd /opt/kubernetes/cfg # grep 192 * bootstrap.kubeconfig:??? server: https://192.168.254.201:6443 kubelet.kubeconfig:??? server: https://192.168.254.2016443 kube-proxy.kubeconfig:??? server: https://192.168.254.201:6443 ``` 批量修改： ``` sed -i 's#192.168.254.201#192.168.254.200#g' * ``` 2.、執行 ``` systemctl restart kubelet systemctl restart kube-proxy ``` 3、驗證 ![](https://img.kancloud.cn/1a/88/1a882a97ebe6949cad465d51caaa4aad_1026x164.png) 查看日志 ``` tail -f /var/log/nginx/k8s-access.log ``` 五、注意 [https://blog.csdn.net/u010801994/article/details/86691777](https://blog.csdn.net/u010801994/article/details/86691777) ``` kubectl get node ``` ![](https://img.kancloud.cn/02/be/02bef2655acaa1003f40b617ccd86648_837x105.png) ``` kubectl describe node? k8s-node1 ``` ![](https://img.kancloud.cn/2b/1e/2b1e2ca22c8bae0a4c4d7998321cb6bf_1470x206.png) ?由錯誤日志可知是節點中的Cgroup不支持子系統pids所致。于是用uname -r查看內核版本： [root@localhost ~]# uname -r? 3.10.0-327.el7.x86_64 然后查看該內核所支持的CGROUP，發現的確不支持PIDS [root@localhost ~]# cat /boot/config-3.10.0-327.el7.x86_64? | grep CGROUP CONFIG_CGROUPS=y #CONFIG_CGROUP\_DEBUG is not set CONFIG_CGROUP_FREEZER=y CONFIG_CGROUP_DEVICE=y CONFIG_CGROUP_CPUACCT=y CONFIG_CGROUP_HUGETLB=y CONFIG_CGROUP_PERF=y CONFIG_CGROUP_SCHED=y CONFIG_BLK_CGROUP=y #CONFIG_DEBUG_BLK_CGROUP is not set CONFIG_NETFILTER_XT_MATCH_CGROUP=m CONFIG_NET_CLS_CGROUP=y CONFIG_NETPRIO_CGROUP=m 然后在運行yum update -y后，使用yum list kernel命令查看當前安裝的內核. [root@lbw-master ~]# yum list kernel Installed Packages kernel.x86_64?????3.10.0-327.el7?????????? @anaconda kernel.x86_64?????3.10.0-862.3.2.el7 ??????? @updates kernel.x86_64?????3.10.0-957.21.3.el7 ???? @updates kernel.x86_64?????3.10.0-957.27.2.el7???????@updates ?查看新版內核所支持的CGOURP [root@lbw-master ~]# cat /boot/config-3.10.0-957.27.2.el7.x86_64 | grep CGROUP CONFIG_CGROUPS=y #CONFIG_CGROUP_DEBUG is not set CONFIG_CGROUP_FREEZER=y CONFIG_CGROUP_PIDS=y CONFIG_CGROUP_DEVICE=y CONFIG_CGROUP_CPUACCT=y CONFIG_CGROUP_HUGETLB=y CONFIG_CGROUP_PERF=y CONFIG_CGROUP_SCHED=y CONFIG_BLK_CGROUP=y #CONFIG_DEBUG_BLK_CGROUP is not set CONFIG_NETFILTER_XT_MATCH_CGROUP=m CONFIG_NET_CLS_CGROUP=y CONFIG_NETPRIO_CGROUP=y 發現的確有PIDS支持。于是接下來就是想辦法將內核進行升級了。用以下命令查看所有可用的內核 [root@lbw-master ~]# awk -F\\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg 0 : CentOS Linux (3.10.0-957.27.2.el7.x86_64) 7 (Core) 1 : CentOS Linux (3.10.0-957.21.3.el7.x86_64) 7 (Core) 2 : CentOS Linux (3.10.0-862.3.2.el7.x86_64) 7 (Core) 3 : CentOS Linux (3.10.0-327.el7.x86_64) 7 (Core) 4 : CentOS Linux (0-rescue-c4da2e677e384e85b9fd9f27eb3a9f8a) 7 (Core) 用grub2-set-default命令設置默認啟動內核。利用設為0表示使用上一個命令輸出的第一個內核。 grub2-set-default 0 然后用grub2-mkconfig命令生成配置文件并應用在grub.config文件中。 grub2-mkconfig -o /boot/grub2/grub.cfg 執行完畢后，用reboot命令重啟機器即可。