在ubuntu18.04上k8s集群 · ubuntu系統上kubernetes集群實踐

### 1.環境準備 * * * * * master：192.168.11.212 etcd master：192.168.11.213 etcd master：192.168.11.214 etcd node：192.168.11.220 node：192.168.11.221 node：192.168.11.222 haproxy:192.168.11.215 haproxy:192.168.11.216 keealived(vip):192.168.11.230 jenkins-master: jenkins-slave: jenkins-slave: harbo: harbo: zookeeper+kafka zookeeper+kafka zookeeper+kafka elk: elk: elk: * * * * * ### 2.以上環境均為ubuntu18.04系統因為centos上docker的devicemapper性能問題上產環境最好也使用ubuntu18.04服務器版 * * * * * ### 3.安裝master節點： * [ ] 創建目錄：mkdir -p /opt/kubernetes/{bin,ssl,cfg,log} * [ ] master和node節點必須都創建統一 * [ ] bin目錄：二進制可執行文件安放 * [ ] ssl：生成的證書安放 * [ ] cfg：配置文件及kubeconfig文件安放 * [ ] log：容器日志統一安放地點，便于為以后elf日志收集 * [ ] 在這里直接使用cfssl工具來實現證書配置,cfssl工具安裝如下： * * * * * ~~~ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 mv cfssl_linux-amd64 /opt/kubernetes/bin/cfssl mv cfssljson_linux-amd64 /opt/kubernetes/bin/cfssljson mv cfssl-certinfo_linux-amd64 /opt/kubernetes/bin/cfssl-certinfo ~~~ * [ ] 臨時導入環境變量：export PATH=/opt/kubernetes/bin:$PATH * [ ] 永久可以寫入/etc/profile 生成證書我們需要如下配置文件： ca-config.json文件內容如下： ```sh { "signing": { "default": { "expiry": "175200h" }, "profiles": { "kubernetes": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] }, "etcd": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } ``` 字段說明： * ca-config.json：可以定義多個Profiles，分別指定不同的過期時間、使用場景等參數；后續在簽名證書的時候使用某個Profile。這里定義了兩個Profile，一個用于kubernetes，一個用于etcd，我這里etcd沒有使用證書，所以另一個不使用。 * signing：表示該證書可用于簽名其他證書；生成的ca.pem證書中CA=TRUE * server auth：表示client可以使用該ca對server提供的證書進行驗證 * client auth：表示server可以用該ca對client提供的證書進行驗證 * * * * * ca-csr.json內容如下： ~~~ { "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Wuhan", "ST": "Hubei", "O": "k8s", "OU": "System" } ] } ~~~ ### 生成ca證書： ``` cfssl gencert --initca=true ca-csr.json | cfssljson --bare ca ``` ### 生成kubernetes證書 kubernetes-csr.json內容如下： ```sh { "CN": "kubernetes", "hosts": [ "127.0.0.1", "localhost", "10.1.61.175", "10.1.61.176", "10.1.61.177", "10.254.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Hubei", "L": "Wuhan", "O": "k8s", "OU": "System" } ] } 這個內容需要做下簡要說明：上面配置hosts字段中指定授權使用該證書的IP和域名列表，因為現在要生成的證書需要被Kubernetes Master集群各個節點使用，所以這里指定了各個節點的IP和hostname。生成kubernetes證書： ``` cfssl gencert --ca ca.pem --ca-key ca-key.pem --config ca-config.json --profile kubernetes kubernetes-csr.json | cfssljson --bare kubernetes ``` ### 生成kubectl證書 admin-csr.json內容如下： ``` ``` { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Hubei", "L": "Wuhan", "O": "system:masters", "OU": "System" } ] } ``` * kube-apiserver會提取**CN**作為客戶端的用戶名，這里是admin，將提取**O**作為用戶的屬組，這里是system:masters * 后續kube-apiserver使用RBAC對客戶端（如kubelet、kube-proxy、pod）請求進行授權 * apiserver預定義了一些RBAC使用的ClusterRoleBindings，例如cluster-admin將組system:masters與CluasterRole cluster-admin綁定，而cluster-admin擁有訪問apiserver的所有權限，因此admin用戶將作為集群的超級管理員。 ### 生成kubectl證書： ``` cfssl gencert --ca ca.pem --ca-key ca-key.pem --config ca-config.json --profile kubernetes admin-csr.json | cfssljson --bare admin ``` ### 生成kube-proxy證書 kube-proxy-csr.json內容如下： ``` { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Hubei", "L": "Wuhan", "O": "k8s", "OU": "System" } ] } ``` * CN指定該證書的user為system:kube-proxy * kube-apiserver預定義的RoleBinding cluster-admin將User system:kube-proxy與Role system:node-proxier綁定，該role授予了調用kube-apiserver Proxy相關API的權限； ### 生成kube-proxy證書： ``` cfssl gencert --ca ca.pem --ca-key ca-key.pem --config ca-config.json --profile kubernetes kube-proxy-csr.json | cfssljson --bare kube-proxy ``` 上面所有證書，都可以通過如下方法一下子全部生成: ``` cfssl gencert --initca=true ca-csr.json | cfssljson --bare ca for targetName in kubernetes admin kube-proxy; do cfssl gencert --ca ca.pem --ca-key ca-key.pem --config ca-config.json --profile kubernetes $targetName-csr.json | cfssljson --bare $targetName done ``` cfssl的用法中，--profile就用于指定ca-config里的哪個profiles 生成的證書列表如下： ``` ll *.pem total 48 -rw------- 1 kube kube 1679 Aug 30 16:49 admin-key.pem -rw-r--r-- 1 kube kube 1363 Aug 30 16:49 admin.pem -rw------- 1 kube kube 1675 Aug 30 16:49 ca-key.pem -rw-r--r-- 1 kube kube 1289 Aug 30 16:49 ca.pem -rw------- 1 kube kube 1679 Aug 30 16:49 kube-proxy-key.pem -rw-r--r-- 1 kube kube 1363 Aug 30 16:49 kube-proxy.pem -rw------- 1 kube kube 1679 Sep 13 13:46 kubernetes-key.pem -rw-r--r-- 1 kube kube 1586 Sep 13 13:46 kubernetes.pem 將生成好的證書移動到創建后的ssl目錄檢驗證書： ~~~ #以kubernetes證書為例 openssl x509 -noout -text -in kubernetes.pem Certificate: Data: Version: 3 (0x2) Serial Number: 7a:a2:fa:da:4c:7a:0d:7d:fa:c1:f4:a8:af:f7:77:24:04:54:19:3f Signature Algorithm: sha256WithRSAEncryption Issuer: C = CN, ST = Hubei, L = Wuhan, O = k8s, OU = System Validity Not Before: Aug 27 11:50:00 2018 GMT Not After : Aug 22 11:50:00 2038 GMT Subject: C = CN, ST = Hubei, L = Wuhan, O = k8s, OU = System, CN = kubernetes Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:b8:00:1e:bb:a8:75:2c:07:32:5b:da:d5:23:25: c2:0f:c9:10:08:5b:78:40:78:90:4a:59:e3:cc:64: 36:1a:29:c1:ea:fe:01:f4:88:2f:73:be:20:98:b9: 09:e9:c1:13:a7:b8:26:5f:54:52:21:0a:89:03:c8: d3:33:a1:be:20:bb:03:d7:5b:e4:19:46:e2:e9:67: e7:89:3a:68:2d:f9:c8:66:54:ce:dd:7d:99:fd:1b: a7:32:e2:44:b5:ba:14:f0:60:94:38:51:ff:2b:2c: fe:7c:f3:55:1b:4c:19:d8:ad:10:10:08:c3:db:2e: 65:46:36:e9:63:ea:7c:3a:75:b7:59:a5:90:7f:16: 2d:be:56:16:c8:f0:fe:40:6d:1e:bf:9f:ff:4c:9c: cb:57:4b:a9:04:7a:61:ce:9b:91:86:c2:19:1b:a5: be:82:b2:75:e5:8c:fb:65:ce:cf:ad:72:c6:6d:85: 19:c7:ce:a9:86:72:79:51:bf:4f:2f:c2:03:e8:34: 9a:12:8c:0b:57:ac:90:39:69:56:0e:00:3b:15:32: fd:fa:77:de:a8:7e:46:5e:86:e3:60:ac:41:56:80: 00:59:4c:a7:a1:f5:78:0f:1e:1c:a6:9e:7e:f8:93: c5:aa:f8:22:b0:c4:e3:f9:24:92:f8:b7:09:ad:e9: 76:c1 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: F8:B2:8A:9F:D7:42:A5:33:D1:A0:23:29:FD:42:06:4A:80:2F:1D:F6 X509v3 Authority Key Identifier: keyid:8E:DD:D0:C9:6B:3D:D8:CA:ED:5B:FD:86:48:65:AD:CC:D6:3F:B6:B6 X509v3 Subject Alternative Name: DNS:localhost, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:192.168.11.212, IP Address:192.168.11.213, IP Address:192.168.11.214, IP Address:192.168.11.215, IP Address:192.168.11.216, IP Address:192.168.11.222, IP Address:192.168.11.221, IP Address:192.168.11.220, IP Address:192.168.11.230, IP Address:172.16.0.1 Signature Algorithm: sha256WithRSAEncryption 11:4f:5c:44:5b:0c:d1:ca:d4:aa:d8:47:16:63:f9:4b:8f:b0: a7:7c:58:42:2f:ea:dd:80:b6:ae:0e:1d:8b:72:b7:40:ba:9e: a2:3b:9f:fb:04:10:4d:bd:59:0c:08:ea:2e:54:a8:0d:63:02: 6d:94:78:be:72:b2:2f:8d:b1:c2:c0:bf:a4:19:45:8d:b6:b4: d8:28:58:c6:e9:75:c8:4a:49:51:72:33:04:6e:52:25:60:57: cc:fe:0e:83:35:b8:cb:1d:28:ed:cd:9d:7b:5b:49:8b:3a:56: 09:3f:ea:80:8a:ca:bd:4f:d9:c4:f7:90:bb:f0:55:be:c6:86: bc:0a:7a:2c:41:a1:19:42:b3:51:ee:f9:7d:7b:70:f7:46:2b: 40:f0:25:e2:2d:f7:fc:00:50:7a:7f:48:e1:7d:81:2b:f6:dd: f4:59:35:df:f9:af:2c:be:c3:c3:19:7b:94:9f:94:ec:e9:05: 74:29:c7:e8:40:f2:0b:ac:8c:df:81:8e:d4:0c:aa:ad:71:49: 99:71:d6:b3:f3:28:92:e5:9d:d8:1f:ad:a4:6e:43:d3:67:40: 5a:64:26:d3:0b:0a:79:90:50:1f:13:c7:99:90:14:d2:d5:ad: 82:96:63:ca:3d:21:79:9c:a7:26:0f:a2:1c:5c:d4:b8:5c:13: fb:bf:87:cc ``` * 確認 Issuer 字段的內容和 ca-csr.json 一致； * 確認 Subject 字段的內容和 kubernetes-csr.json 一致； * 確認 X509v3 Subject Alternative Name 字段的內容和 kubernetes-csr.json 一致； * 確認 X509v3 Key Usage、Extended Key Usage 字段的內容和 ca-config.json 中 kubernetesprofile 一致； # 生成token及kubeconfig 在本次配置中，我們將會同時啟用證書認證，token認證，以及http basic認證。所以需要提前生成token認證文件，basic認證文件以及kubeconfig ## 生成客戶端使用的token ``` export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ') cat > bootstrap-token.csv <<EOF ${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF ``` 生成bashboard使用的http basic認證文件 ``` cat > basic_auth.csv <<EOF 123456,admin,1,"system:masters" EOF ``` 生成kubeconfig ``` export KUBE_APISERVER="https://keepalived的虛擬ip:6443" ``` **#### # 設置集群參數，即api-server的訪問方式，給集群起個名字就叫kubernetes** ``` kubectl config set-cluster kubernetes \ --certificate-authority=ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=bootstrap.kubeconfig ``` **#### # 設置客戶端認證參數，這里采用token認證** ``` kubectl config set-credentials kubelet-bootstrap \ --token=${BOOTSTRAP_TOKEN} \ --kubeconfig=bootstrap.kubeconfig ``` **#### # 設置上下文參數，用于連接用戶kubelet-bootstrap與集群kubernetes** ``` kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=bootstrap.kubeconfig ``` #### # 設置默認上下文 ``` kubectl config use-context default --kubeconfig=bootstrap.kubeconfig ``` #### kube-proxy的kubeconfig配置如下，與上面基本相同：** ``` # 設置集群參數 kubectl config set-cluster kubernetes \ --certificate-authority=ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-proxy.kubeconfig # 設置客戶端認證參數 kubectl config set-credentials kube-proxy \ --client-certificate=kube-proxy.pem \ --client-key=kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig # 設置上下文參數 kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig # 設置默認上下文 kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig ``` # 部署master master端涉及kube-apiserver, kube-controller-manager以及kube-scheduler三個組件。所有組件我們都使用二進制包的方式安裝。kubernetes源代碼地址：https://github.com/kubernetes/kubernetes 我們可以通過git clone的方式把源代碼下載到本地，并checkout出1.10版本。然后執行編譯，編譯之后，所有的二進制文件都未于源代碼目錄的_output目錄中。我們獲取我們所需要的二進制組件即可。另外需要說明的是，編譯需要依賴go開發環境。 ``` git clone https://github.com/kubernetes/kubernetes.git cd kubernetes git checkout release-1.11 make ``` 另外，我們還需要將前面生成的ca證書及key，kubernetes的證書及key以及kubectl的證書及key分發到各個master節點的/opt/kubernetes/ssl目錄中。我這里使用的的是下載編譯好的二進制安裝包 server 的 tarball kubernetes-server-linux-amd64.tar.gz 已經包含了 client(kubectl) 二進制文件，所以不用單獨下載kubernetes-client-linux-amd64.tar.gz文件； ``` wget https://dl.k8s.io/v1.11.1/kubernetes-server-linux-amd64.tar.gz tar -xzvf kubernetes-server-linux-amd64.tar.gz cd kubernetes tar -xzvf kubernetes-src.tar.gz cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /opt/kubernetes/bin/ chomd +x * ```