# Kubernetes搭建高可用集群
## 前言
之前我們搭建的集群,只有一個master節點,當master節點宕機的時候,通過node將無法繼續訪問,而master主要是管理作用,所以整個集群將無法提供服務

## 高可用集群
下面我們就需要搭建一個多master節點的高可用集群,不會存在單點故障問題
但是在node 和 master節點之間,需要存在一個 LoadBalancer組件,作用如下:
- 負載
- 檢查master節點的狀態

對外有一個統一的VIP:虛擬ip來對外進行訪問
## 高可用集群技術細節
高可用集群技術細節如下所示:

- keepalived:配置虛擬ip,檢查節點的狀態
- haproxy:負載均衡服務【類似于nginx】
- apiserver:
- controller:
- manager:
- scheduler:
## 高可用集群步驟
我們采用2個master節點,一個node節點來搭建高可用集群,下面給出了每個節點需要做的事情

## 初始化操作
我們需要在這三個節點上進行操作
```bash
# 關閉防火墻
systemctl stop firewalld
systemctl disable firewalld
# 關閉selinux
# 永久關閉
sed -i 's/enforcing/disabled/' /etc/selinux/config
# 臨時關閉
setenforce 0
# 關閉swap
# 臨時
swapoff -a
# 永久關閉
sed -ri 's/.*swap.*/#&/' /etc/fstab
# 根據規劃設置主機名【master1節點上操作】
hostnamectl set-hostname master1
# 根據規劃設置主機名【master2節點上操作】
hostnamectl set-hostname master1
# 根據規劃設置主機名【node1節點操作】
hostnamectl set-hostname node1
# r添加hosts
cat >> /etc/hosts << EOF
192.168.44.158 k8smaster
192.168.44.155 master01.k8s.io master1
192.168.44.156 master02.k8s.io master2
192.168.44.157 node01.k8s.io node1
EOF
# 將橋接的IPv4流量傳遞到iptables的鏈【3個節點上都執行】
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
# 生效
sysctl --system
# 時間同步
yum install ntpdate -y
ntpdate time.windows.com
```
## 部署keepAlived
下面我們需要在所有的master節點【master1和master2】上部署keepAlive
### 安裝相關包
```bash
# 安裝相關工具
yum install -y conntrack-tools libseccomp libtool-ltdl
# 安裝keepalived
yum install -y keepalived
```
### 配置master節點
添加master1的配置
```bash
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id k8s
}
vrrp_script check_haproxy {
script "killall -0 haproxy"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 250
advert_int 1
authentication {
auth_type PASS
auth_pass ceb1b3ec013d66163d6ab
}
virtual_ipaddress {
192.168.44.158
}
track_script {
check_haproxy
}
}
EOF
```
添加master2的配置
```bash
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id k8s
}
vrrp_script check_haproxy {
script "killall -0 haproxy"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 200
advert_int 1
authentication {
auth_type PASS
auth_pass ceb1b3ec013d66163d6ab
}
virtual_ipaddress {
192.168.44.158
}
track_script {
check_haproxy
}
}
EOF
```
### 啟動和檢查
在兩臺master節點都執行
```bash
# 啟動keepalived
systemctl start keepalived.service
# 設置開機啟動
systemctl enable keepalived.service
# 查看啟動狀態
systemctl status keepalived.service
```
啟動后查看master的網卡信息
```bash
ip a s ens33
```

## 部署haproxy
haproxy主要做負載的作用,將我們的請求分擔到不同的node節點上
### 安裝
在兩個master節點安裝 haproxy
```bash
# 安裝haproxy
yum install -y haproxy
# 啟動 haproxy
systemctl start haproxy
# 開啟自啟
systemctl enable haproxy
```
啟動后,我們查看對應的端口是否包含 16443
```bash
netstat -tunlp | grep haproxy
```

### 配置
兩臺master節點的配置均相同,配置中聲明了后端代理的兩個master節點服務器,指定了haproxy運行的端口為16443等,因此16443端口為集群的入口
```bash
cat > /etc/haproxy/haproxy.cfg << EOF
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes-apiserver
mode tcp
bind *:16443
option tcplog
default_backend kubernetes-apiserver
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
mode tcp
balance roundrobin
server master01.k8s.io 192.168.44.155:6443 check
server master02.k8s.io 192.168.44.156:6443 check
#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
bind *:1080
stats auth admin:awesomePassword
stats refresh 5s
stats realm HAProxy\ Statistics
stats uri /admin?stats
EOF
```
## 安裝Docker、Kubeadm、kubectl
所有節點安裝Docker/kubeadm/kubelet ,Kubernetes默認CRI(容器運行時)為Docker,因此先安裝Docker
### 安裝Docker
首先配置一下Docker的阿里yum源
```bash
cat >/etc/yum.repos.d/docker.repo<<EOF
[docker-ce-edge]
name=Docker CE Edge - \$basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/\$basearch/edge
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF
```
然后yum方式安裝docker
```bash
# yum安裝
yum -y install docker-ce
# 查看docker版本
docker --version
# 啟動docker
systemctl enable docker
systemctl start docker
```
配置docker的鏡像源
```bash
cat >> /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
```
然后重啟docker
```bash
systemctl restart docker
```
### 添加kubernetes軟件源
然后我們還需要配置一下yum的k8s軟件源
```bash
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
```
### 安裝kubeadm,kubelet和kubectl
由于版本更新頻繁,這里指定版本號部署:
```bash
# 安裝kubelet、kubeadm、kubectl,同時指定版本
yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0
# 設置開機啟動
systemctl enable kubelet
```
## 部署Kubernetes Master【master節點】
### 創建kubeadm配置文件
在具有vip的master上進行初始化操作,這里為master1
```bash
# 創建文件夾
mkdir /usr/local/kubernetes/manifests -p
# 到manifests目錄
cd /usr/local/kubernetes/manifests/
# 新建yaml文件
vi kubeadm-config.yaml
```
yaml內容如下所示:
```bash
apiServer:
certSANs:
- master1
- master2
- master.k8s.io
- 192.168.44.158
- 192.168.44.155
- 192.168.44.156
- 127.0.0.1
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "master.k8s.io:16443"
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.16.3
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.1.0.0/16
scheduler: {}
```
然后我們在 master1 節點執行
```bash
kubeadm init --config kubeadm-config.yaml
```
執行完成后,就會在拉取我們的進行了【需要等待...】

按照提示配置環境變量,使用kubectl工具
```bash
# 執行下方命令
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 查看節點
kubectl get nodes
# 查看pod
kubectl get pods -n kube-system
```
**按照提示保存以下內容,一會要使用:**
```bash
kubeadm join master.k8s.io:16443 --token jv5z7n.3y1zi95p952y9p65 \
--discovery-token-ca-cert-hash sha256:403bca185c2f3a4791685013499e7ce58f9848e2213e27194b75a2e3293d8812 \
--control-plane
```
> --control-plane : 只有在添加master節點的時候才有
查看集群狀態
```bash
# 查看集群狀態
kubectl get cs
# 查看pod
kubectl get pods -n kube-system
```
## 安裝集群網絡
從官方地址獲取到flannel的yaml,在master1上執行
```bash
# 創建文件夾
mkdir flannel
cd flannel
# 下載yaml文件
wget -c https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```
安裝flannel網絡
```bash
kubectl apply -f kube-flannel.yml
```
檢查
```bash
kubectl get pods -n kube-system
```
## master2節點加入集群
### 復制密鑰及相關文件
從master1復制密鑰及相關文件到master2
```bash
# ssh root@192.168.44.156 mkdir -p /etc/kubernetes/pki/etcd
# scp /etc/kubernetes/admin.conf root@192.168.44.156:/etc/kubernetes
# scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@192.168.44.156:/etc/kubernetes/pki
# scp /etc/kubernetes/pki/etcd/ca.* root@192.168.44.156:/etc/kubernetes/pki/etcd
```
### master2加入集群
執行在master1上init后輸出的join命令,需要帶上參數`--control-plane`表示把master控制節點加入集群
```bash
kubeadm join master.k8s.io:16443 --token ckf7bs.30576l0okocepg8b --discovery-token-ca-cert-hash sha256:19afac8b11182f61073e254fb57b9f19ab4d798b70501036fc69ebef46094aba --control-plane
```
檢查狀態
```bash
kubectl get node
kubectl get pods --all-namespaces
```
## 加入Kubernetes Node
在node1上執行
向集群添加新節點,執行在kubeadm init輸出的kubeadm join命令:
```bash
kubeadm join master.k8s.io:16443 --token ckf7bs.30576l0okocepg8b --discovery-token-ca-cert-hash sha256:19afac8b11182f61073e254fb57b9f19ab4d798b70501036fc69ebef46094aba
```
**集群網絡重新安裝,因為添加了新的node節點**
檢查狀態
```bash
kubectl get node
kubectl get pods --all-namespaces
```
## 測試kubernetes集群
在Kubernetes集群中創建一個pod,驗證是否正常運行:
```bash
# 創建nginx deployment
kubectl create deployment nginx --image=nginx
# 暴露端口
kubectl expose deployment nginx --port=80 --type=NodePort
# 查看狀態
kubectl get pod,svc
```
然后我們通過任何一個節點,都能夠訪問我們的nginx頁面
- Kubernetes簡介
- 搭建K8S集群前置知識
- 使用kubeadm方式搭建K8S集群
- 使用二進制方式搭建K8S集群
- Kubeadm和二進制方式對比
- Kubernetes集群管理工具kubectl
- Kubernetes集群YAML文件詳解
- Kubernetes核心技術Pod
- Kubernetes核心技術Controller
- Kubernetes核心技術Service
- Kubernetes控制器Controller詳解
- Kubernetes配置管理
- Kubernetes集群安全機制
- Kubernetes核心技術Ingress
- Kubernetes核心技術Helm
- Kubernetes持久化存儲
- Kubernetes集群資源監控
- Kubernetes搭建高可用集群
- Kubernetes容器交付介紹
- 使用kubeadm-ha腳本一鍵安裝K8S
- Kubernetes可視化界面kubesphere
- Kubernetes配置默認存儲類
- 使用Rancher搭建Kubernetes集群
- Kubernetes中的CRI