1-1 k8s-1.14-基于calico · 運維工作筆記

#### 這是本人學習[部署落地+業務遷移玩轉k8s進階與企業級實踐技能](https://coding.imooc.com/learn/list/335.html "部署落地+業務遷移玩轉k8s進階與企業級實踐技能")的筆記，如果需要附件，請到首頁百度網盤地址獲取基礎環境 | 系統 | IP地址 | 節點角色 | CPU | 內存 | 主機名 | | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | | centos-7.7 |192.168.88.101 | Master | 2 | 2G | docker-2-12-101 | | centos-7.7 |192.168.88.102 | Master | 2 | 2G |docker-2-12-102 | | centos-7.7 |192.168.88.103 | Node | 2 | 2G | docker-2-12-103 | | centos-7.7 | 192.168.88.104 | Node | 2 | 2G | docker-2-12-104 | MasterVIP：192.168.88.188（APIServer）軟件環境 ``` kubernetes 1.14.10 etcd 3.3.10 coredns 1.3.1 calico 3.1.3 docker 18.09(驗證版本)，實際我用19.03.5 ``` 部署依賴 ``` yum update yum install -y conntrack ipvsadm ipset jq sysstat curl iptables libseccomp ``` 內核參數優化 ``` cat > /etc/sysctl.d/kubernetes.conf <<EOF net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1 vm.swappiness=0 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 EOF sysctl -p /etc/sysctl.d/kubernetes.conf ``` 關閉服務 ``` # 關閉防火墻 systemctl stop firewalld && systemctl disable firewalld # 重置iptables iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT # 關閉swap(基于性能考慮，初始化參數可以忽略) swapoff -a sed -i '/swap/s/^$.*$$/#\1/g' /etc/fstab # 關閉selinux setenforce 0 # 關閉dnsmasq(否則可能導致docker容器無法解析域名) service dnsmasq stop && systemctl disable dnsmasq ``` 初始化主機名 ``` cat >> /etc/hosts << EOF 192.168.88.101 main-101 c7-docker-101 192.168.88.102 main-102 c7-docker-102 192.168.88.103 node-103 c7-docker-103 192.168.88.104 node-104 c7-docker-104 EOF ``` 修改Docker的驅動模式為systemd，請先確認方式一沒有配置/etc/docker/daemon.json ``` cat /etc/docker/daemon.json { ..... "exec-opts": ["native.cgroupdriver=systemd"] ..... } ``` 安裝工具（所有節點） ``` # 配置阿里云yum源 cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF # 安裝 yum install -y --nogpgcheck kubelet-1.14.10 kubeadm-1.14.10 kubectl-1.14.10 # 啟動 systemctl enable kubelet && systemctl start kubelet ``` #### 部署Keepalived集群(任意兩臺Master) /opt/kubeadm-k8s1.14/configs和的keepalived配置文件和/opt/kubeadm-k8s1.14/scritps的腳本文件 ``` yum install -y keepalived ``` #### 初始化Master-1 修改kubeadm-config.yaml的k8s版本和VIP，并上傳到/root目錄 ``` apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration kubernetesVersion: v1.14.10 controlPlaneEndpoint: "192.168.88.188:6443" networking: # This CIDR is a Calico default. Substitute or remove for your CNI provider. podSubnet: "172.22.0.0/16" imageRepository: registry.aliyuncs.com/google_containers ``` 初始化 ``` cd ~ kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs # 配置文件中從阿里云拉取的容器鏡像，速度很快 # 1.16之后參數有變化 experimental-upload-certs更換為upload-certs ``` 拷貝配置，master執行 ``` mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config ``` ##### 初始化成功，記錄相關命令，用于其他節點加入集群 Master-2執行加入master節點命令 ``` kubeadm join 192.168.88.233:6443 --token xzp2kb.habisql3vkgyx02d \ --discovery-token-ca-cert-hash sha256:4526f6e8f08a5c5564e5488c5b939753ee26b7fd0c8ca81423af2d4a58c718a6 \ --experimental-control-plane --certificate-key 5d1af50558253c92b5d9df07a14144ffb37edeb2f0afef5af4dcc3fc022846b3 ``` 加入Node節點 ``` kubeadm join 192.168.88.233:6443 --token xzp2kb.habisql3vkgyx02d \ --discovery-token-ca-cert-hash sha256:4526f6e8f08a5c5564e5488c5b939753ee26b7fd0c8ca81423af2d4a58c718a6 ``` #### 初始化Calico網絡 ``` # 創建目錄（在配置了kubectl的節點上執行） mkdir -p /etc/kubernetes/addons # 上傳calico配置到配置好kubectl的節點（一個節點即可） cd /opt/kubernetes-ha-kubeadm/ scp target/addons/calico* 192.168.88.101:/etc/kubernetes/addons/ # 部署calico kubectl apply -f /etc/kubernetes/addons/calico-rbac-kdd.yaml kubectl apply -f /etc/kubernetes/addons/calico.yaml # 查看狀態 $ kubectl get pods -n kube-system # 由于沒有Node節點，部分節點可能失敗 ``` 加入Master和Work節點一段時間后，集群狀態如下 ``` #kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-node-6dxfm 2/2 Running 0 19m kube-system calico-node-d2fq6 2/2 Running 1 19m kube-system calico-node-kfj78 2/2 Running 0 16m kube-system calico-node-l6vgh 2/2 Running 2 20m kube-system calico-node-rqkrr 2/2 Running 0 19m kube-system calico-typha-666749994b-lnkbt 1/1 Running 0 20m kube-system coredns-8567978547-g7pkr 1/1 Running 4 40m kube-system coredns-8567978547-xzjkb 1/1 Running 4 40m kube-system etcd-docker-2-12-101 1/1 Running 0 40m kube-system etcd-m2 1/1 Running 0 19m kube-system etcd-m3 1/1 Running 0 16m kube-system kube-apiserver-docker-2-12-101 1/1 Running 0 40m kube-system kube-apiserver-m2 1/1 Running 0 19m kube-system kube-apiserver-m3 1/1 Running 0 16m kube-system kube-controller-manager-docker-2-12-101 1/1 Running 1 40m kube-system kube-controller-manager-m2 1/1 Running 0 19m kube-system kube-controller-manager-m3 1/1 Running 0 16m kube-system kube-proxy-2bx2q 1/1 Running 0 16m kube-system kube-proxy-8vwqn 1/1 Running 0 19m kube-system kube-proxy-cv7vg 1/1 Running 0 19m kube-system kube-proxy-mmh7f 1/1 Running 0 40m kube-system kube-proxy-pzk2r 1/1 Running 0 19m kube-system kube-scheduler-docker-2-12-101 1/1 Running 1 39m kube-system kube-scheduler-m2 1/1 Running 0 19m kube-system kube-scheduler-m3 1/1 Running 0 16m ``` 檢查集群狀態（Master） ``` curl -k https://localhost:6443/healthz ``` #### 集群可用性測試創建nginx ds ``` cat > nginx-ds.yml <<EOF apiVersion: v1 kind: Service metadata: name: nginx-ds labels: app: nginx-ds spec: type: NodePort selector: app: nginx-ds ports: - name: http port: 80 targetPort: 80 --- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: nginx-ds labels: addonmanager.kubernetes.io/mode: Reconcile spec: template: metadata: labels: app: nginx-ds spec: containers: - name: my-nginx image: nginx:1.7.9 ports: - containerPort: 80 EOF ``` 創建ds ``` kubectl create -f nginx-ds.yml ``` #### 檢查各種ip連通性 ``` # 檢查各 Node 上的 Pod IP 連通性 kubectl get pods -o wide # 在每個節點上ping pod ip ping <pod-ip> # 檢查service可達性 kubectl get svc # 在每個節點上訪問服務 curl <service-ip>:<port> # 在每個節點檢查node-port可用性 curl <node-ip>:<port> ``` #### 檢查dns可用性 ``` # 創建一個nginx pod $ cat > pod-nginx.yaml <<EOF apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 EOF # 創建pod kubectl create -f pod-nginx.yaml # 進入pod，查看dns kubectl exec nginx -i -t -- /bin/bash # 查看dns配置 cat /etc/resolv.conf # 查看名字是否可以正確解析 ping nginx-ds ``` #### 部署dashboard ``` # 上傳dashboard配置 scp target/addons/dashboard-all.yaml 192.168.88.101:/etc/kubernetes/addons/ # 創建服務 kubectl apply -f /etc/kubernetes/addons/dashboard-all.yaml # 查看服務運行情況 kubectl get deployment kubernetes-dashboard -n kube-system kubectl --namespace kube-system get pods -o wide kubectl get services kubernetes-dashboard -n kube-system netstat -ntlp|grep 30005 ``` 訪問Dashboard 如果第一次部署報錯，刪除pods后重新再創建，成功 ``` xxx namespaces is forbidden xxx ``` ``` https://192.168.88.101:30005 ``` 獲取Token ``` # 創建service account kubectl create sa dashboard-admin -n kube-system # 創建角色綁定關系 kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin # 查看dashboard-admin的secret名字 ADMIN_SECRET=$(kubectl get secrets -n kube-system | grep dashboard-admin | awk '{print $1}') # 打印secret的token kubectl describe secret -n kube-system ${ADMIN_SECRET} | grep -E '^token' | awk '{print $2}' ```