## 部署Node節點
**kubernetes Node節點包含如下組件:**
* flanneld
* docker
* kubelet
* kube-proxy
#### 環境變量
```shell
$ source /usr/k8s/bin/env.sh
$ export KUBE_APISERVER="https://${MASTER_URL}" #如果你沒有安裝`haproxy`的話,還是需要使用6443端口的哦
$ export NODE_IP=192.168.1.170 # 當前部署的節點 IP
```
#### 配置docker
```shell
$ cat /usr/lib/systemd/system/docker.service # 用systemctl status docker 命令可查看unit 文件路徑
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/bin/dockerd --log-level=info $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
```
**kube-node-61的docker.service文件如下**
```shell
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target rhel-push-plugin.socket registries.service
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer
[Service]
Type=notify
NotifyAccess=all
EnvironmentFile=-/run/containers/registries.conf
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \
--add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
--default-runtime=docker-runc \
--exec-opt native.cgroupdriver=cgroupfs \
--userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
$OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$ADD_REGISTRY \
$BLOCK_REGISTRY \
$INSECURE_REGISTRY\
$REGISTRIES
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
MountFlags=slave
KillMode=process
[Install]
WantedBy=multi-user.target
```
#### 啟動docker
```shell
$ sudo systemctl daemon-reload
$ sudo systemctl stop firewalld
$ sudo systemctl disable firewalld
$ sudo iptables -F && sudo iptables -X && sudo iptables -F -t nat && sudo iptables -X -t nat
$ sudo systemctl enable docker
$ sudo systemctl start docker
```
* 需要關閉 firewalld(centos7)/ufw(ubuntu16.04),否則可能會重復創建的 iptables 規則
* 最好清理舊的 iptables rules 和 chains 規則
* 執行命令:docker version,檢查docker服務是否正常
#### 安裝和配置kubelet
kubelet 啟動時向kube-apiserver 發送TLS bootstrapping 請求,需要先將bootstrap token 文件中的kubelet-bootstrap 用戶賦予system:node-bootstrapper 角色,然后kubelet 才有權限創建認證請求(certificatesigningrequests):
> kubelet就是運行在Node節點上的,所以這一步安裝是在所有的Node節點上,如果你想把你的Master也當做Node節點的話,當然也可以在Master節點上安裝的。
```shell
$ kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
```
* --user=kubelet-bootstrap 是文件 /etc/kubernetes/token.csv 中指定的用戶名,同時也寫入了文件 /etc/kubernetes/bootstrap.kubeconfig
另外1.8 版本中還需要為Node 請求創建一個RBAC 授權規則:
```shell
$ kubectl create clusterrolebinding kubelet-nodes --clusterrole=system:node --group=system:nodes
```
然后下載最新的kubelet 和kube-proxy 二進制文件(前面下載kubernetes 目錄下面其實也有):
```shell
$ wget https://dl.k8s.io/v1.8.2/kubernetes-server-linux-amd64.tar.gz
$ tar -xzvf kubernetes-server-linux-amd64.tar.gz
$ cd kubernetes
$ tar -xzvf kubernetes-src.tar.gz
$ sudo cp -r ./server/bin/{kube-proxy,kubelet} /usr/k8s/bin/
```
**創建kubelet bootstapping kubeconfig文件**
```shell
# 設置集群參數
$ kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig
# 設置客戶端認證參數
$ kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig
# 設置上下文參數
$ kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
# 設置默認上下文
$ kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
$ mv bootstrap.kubeconfig /etc/kubernetes/
```
* --embed-certs 為 true 時表示將 certificate-authority 證書寫入到生成的 bootstrap.kubeconfig 文件中;
* 設置 kubelet 客戶端認證參數時沒有指定秘鑰和證書,后續由 kube-apiserver 自動生成;
**創建kubelet的systemd unit文件**
```shell
$ sudo mkdir /var/lib/kubelet # 必須先創建工作目錄
$ cat > kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/k8s/bin/kubelet \
--fail-swap-on=false \
--cgroup-driver=cgroupfs \
--address=192.168.10.61 \
--hostname-override=192.168.10.61 \
--experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
--require-kubeconfig \
--cert-dir=/etc/kubernetes/ssl \
--cluster-dns=10.254.0.2 \
--cluster-domain=cluster.local. \
--hairpin-mode promiscuous-bridge \
--allow-privileged=true \
--serialize-image-pulls=false \
--logtostderr=true \
--v=2
ExecStartPost=/sbin/iptables -A INPUT -s 10.0.0.0/8 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -s 172.17.0.0/12 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -s 192.168.1.0/16 -p tcp --dport 4194 -j ACCEPT
ExecStartPost=/sbin/iptables -A INPUT -p tcp --dport 4194 -j DROP
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
```
> 請仔細閱讀下面的注意事項,不然可能會啟動失敗。
* --fail-swap-on參數,這個一定要注意,Kubernetes 1.8開始要求關閉系統的Swap,如果不關閉,默認配置下kubelet將無法啟動,也可以通過kubelet的啟動參數–fail-swap-on=false來避免該問題
* --cgroup-driver參數,kubelet 用來維護主機的的 cgroups 的,默認是cgroupfs,但是這個地方的值需要你根據docker 的配置來確定(docker info |grep cgroup)
* -address 不能設置為 127.0.0.1,否則后續 Pods 訪問 kubelet 的 API 接口時會失敗,因為 Pods 訪問的 127.0.0.1指向自己而不是 kubelet
* 如果設置了 --hostname-override 選項,則 kube-proxy 也需要設置該選項,否則會出現找不到 Node 的情況
* --experimental-bootstrap-kubeconfig 指向 bootstrap kubeconfig 文件,kubelet 使用該文件中的用戶名和 token 向 kube-apiserver 發送 TLS Bootstrapping 請求
* 管理員通過了 CSR 請求后,kubelet 自動在 --cert-dir 目錄創建證書和私鑰文件(kubelet-client.crt 和 kubelet-client.key),然后寫入 --kubeconfig 文件(自動創建 --kubeconfig 指定的文件)
* 建議在 --kubeconfig 配置文件中指定 kube-apiserver 地址,如果未指定 --api-servers 選項,則必須指定 --require-kubeconfig 選項后才從配置文件中讀取 kue-apiserver 的地址,否則 kubelet 啟動后將找不到 kube-apiserver (日志中提示未找到 API Server),kubectl get nodes 不會返回對應的 Node 信息
* --cluster-dns 指定 kubedns 的 Service IP(可以先分配,后續創建 kubedns 服務時指定該 IP),--cluster-domain 指定域名后綴,這兩個參數同時指定后才會生效
#### 啟動kubelet
```shell
$ sudo cp kubelet.service /etc/systemd/system/kubelet.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable kubelet
$ sudo systemctl start kubelet
$ systemctl status kubelet
```
**通過kubelet的TLS證書請求**
kubelet 首次啟動時向kube-apiserver 發送證書簽名請求,必須通過后kubernetes 系統才會將該 Node 加入到集群。查看未授權的CSR 請求:
```shell
$ kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr--k3G2G1EoM4h9w1FuJRjJjfbIPNxa551A8TZfW9dG-g 2m kubelet-bootstrap Pending
$ kubectl get nodes
No resources found.
```
通過CSR請求
```shell
$ kubectl certificate approve node-csr--k3G2G1EoM4h9w1FuJRjJjfbIPNxa551A8TZfW9dG-g
certificatesigningrequest "node-csr--k3G2G1EoM4h9w1FuJRjJjfbIPNxa551A8TZfW9dG-g" approved
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
192.168.1.170 Ready <none> 48s v1.8.1
```
自動生成了kubelet kubeconfig文件和公私鑰
```shell
$ ls -l /etc/kubernetes/kubelet.kubeconfig
-rw------- 1 root root 2280 Nov 7 10:26 /etc/kubernetes/kubelet.kubeconfig
$ ls -l /etc/kubernetes/ssl/kubelet*
-rw-r--r-- 1 root root 1046 Nov 7 10:26 /etc/kubernetes/ssl/kubelet-client.crt
-rw------- 1 root root 227 Nov 7 10:22 /etc/kubernetes/ssl/kubelet-client.key
-rw-r--r-- 1 root root 1115 Nov 7 10:16 /etc/kubernetes/ssl/kubelet.crt
-rw------- 1 root root 1675 Nov 7 10:16 /etc/kubernetes/ssl/kubelet.key
```
#### 配置kube-proxy
**創建kube-proxy證書簽名請求:**
```shell
$ cat > kube-proxy-csr.json <<EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
```
**生成kube-proxy客戶端證書和私鑰**
```shell
$ cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
-ca-key=/etc/kubernetes/ssl/ca-key.pem \
-config=/etc/kubernetes/ssl/ca-config.json \
-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
$ ls kube-proxy*
kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem
$ sudo mv kube-proxy*.pem /etc/kubernetes/ssl/
```
#### 創建kube-proxy kubeconfig文件
```shell
# 設置集群參數
$ kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
# 設置客戶端認證參數
$ kubectl config set-credentials kube-proxy \
--client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
--client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
# 設置上下文參數
$ kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
# 設置默認上下文
$ kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
$ mv kube-proxy.kubeconfig /etc/kubernetes/
```
#### 創建kube-proxy的systemd unit文件
```shell
$ sudo mkdir -p /var/lib/kube-proxy # 必須先創建工作目錄
$ cat > kube-proxy.service <<EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/k8s/bin/kube-proxy \\
--bind-address=${NODE_IP} \\
--hostname-override=${NODE_IP} \\
--cluster-cidr=${SERVICE_CIDR} \\
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \\
--logtostderr=true \\
--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
```
**kube-node-61機器的kube-proxy.service文件如下**
```shell
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/k8s/bin/kube-proxy \
--bind-address=192.168.10.61 \
--hostname-override=192.168.10.61 \
--cluster-cidr=10.254.0.0/16 \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
--logtostderr=true \
--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
```
#### 啟動kube-proxy
```shell
$ sudo cp kube-proxy.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl enable kube-proxy
$ sudo systemctl start kube-proxy
$ systemctl status kube-proxy
```
- Docker
- Docker入門
- docker管理UI
- 封裝各大數據組件
- 自主封裝
- 封裝hadoop
- 封裝spark
- 官方封裝
- 封裝hue
- 封裝jenkins
- Swarm
- Swarm入門
- Zookeeper on swarm
- Hue on swarm
- Grafana
- influxDB
- Prometheus
- cAdvisor
- kubernetes
- k8s入門
- k8s部署dashboard
- minikube
- 手動搭建k8s的高可用集群
- 01環境準備
- 02部署etcd集群
- 03配置kubelet
- 04部署flannel網絡
- 05部署master集群
- 06配置高可用
- 07部署node節點
- 08驗證集群
- Monitor
- swarm 監控
- influxDB+Grafana
- Prometheus+Grafana