### Kubespray Arch  ### cluster HA-mode  [參考鏈接](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/ha-mode.md) ### 阿里云 容器服務 vs kubespray - 集群托管 vs 自行維護 - 容器鏡像 vs harbor - 應用配置管理 vs git+zookeeper+configmap - 阿里 SLB 藍綠發布 vs nginx-ingress - Nas oss vs ClusterFS [slb金絲雀發布最佳實踐](https://www.alibabacloud.com/help/zh/doc-detail/73980.htm?spm=a2c63.p38356.b99.246.20d65044xK7ALo) [alicloud-controller-manager](https://github.com/AliyunContainerService/alicloud-controller-manager) ## 應用部署 ### Jenkins 動態伸縮slave jenkins地址：http://jenkins-k8s.jiedai361.com * 服務高可用，當 Jenkins Master 出現故障時，Kubernetes 會自動創建一個新的 Jenkins Master 容器，并且將 Volume 分配給新創建的容器，保證數據不丟失，從而達到集群服務高可用。 * 動態伸縮，合理使用資源，每次運行 Job 時，會自動創建一個 Jenkins Slave，Job 完成后，Slave 自動注銷并刪除容器，資源自動釋放，而且 Kubernetes 會根據每個資源的使用情況，動態分配 Slave 到空閑的節點上創建，降低出現因某節點資源利用率高，還排隊等待在該節點的情況。 * 擴展性好，當 Kubernetes 集群的資源嚴重不足而導致 Job 排隊等待時，可以很容易的添加一個 Kubernetes Node 到集群中，從而實現擴展。  1. 創建應用jenkins job 2. 創建應用配置：啟動參數 setenv.sh 3. 創建應用環境變量：configmap 4. 創建服務路由ingress 啟動參數配置： setenv.sh cat FTC_DEMO_FTC-PAAS-CONTRACT-SERVICE/bin/setenv.sh JAVA_OPTS=" -DZK_URL=${DR_CFG_ZOOKEEPER_ENV_URL} -DAppLogs=/volume_logs/ -Xms2G -Xmx2G -Dspring.application.name=ftc-paas-contract-service ${JAVA_OPTS} " 應用環境變量：configmap cat ftc-paas-contract-service-configmap.yaml kind: ConfigMap apiVersion: v1 metadata: name: ftc-paas-contract-service-env-config namespace: ftc-demo data: CFG_ADDR: '10.34.11.186:4181' DR_CFG_ZOOKEEPER_ENV_URL: '10.34.11.186:4181' CFG_FILES: 'bin/setenv.sh' 創建服務路由ingress： cat app-ingress/ftc-paas-contract-service-ingres.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ftc-paas-contract-service-ingress namespace: ftc-demo annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: ftc-paas-contract-k8s-demo.dalianyun.com http: paths: - backend: serviceName: ftc-paas-contract-service servicePort: 8073 service 服務請求地址 k8s-demo set /config/contract/ftc-paas-contract-service/cfca.seal_endpoint http://ftc-paas-cfca-seal.ftc-demo.svc.cluster.local:8080/Seal/ set /config/contract/ftc-paas-contract-service/cfca.ra_endpoint http://ftc-paas-cfca-ra.ftc-demo.svc.cluster.local:8080/raWeb/CSHttpServlet set /config/contract/ftc-paas-contract-service/cfca.kt_ip ftc-paas-cfca-kt.ftc-demo.svc.cluster.local set /config/contract/ftc-paas-contract-service/cfca.kt_port 9040 [kubernetes 命名規范](https://wiki.dianrong.com/pages/viewpage.action?pageId=36098232) jenkins 應用部署 kind: Deployment metadata: name: ftc-paas-contract-service namespace: ftc-demo labels: app: ftc-paas-contract-service version: CC-321 AppEnv: demo spec: replicas: 2 selector: matchLabels: app: ftc-paas-contract-service template: metadata: labels: app: ftc-paas-contract-service spec: containers: - name: ftc-paas-contract-service image: dl-harbor.dianrong.com/ftc/ftc-paas-contract-service:6676f059485f2e38703b8aeaadf6614184fb01a3 ports: - containerPort: 8073 livenessProbe: httpGet: path: /ccc/api/health port: 8073 initialDelaySeconds: 90 timeoutSeconds: 5 periodSeconds: 5 readinessProbe: httpGet: path: /ccc/api/health port: 8073 initialDelaySeconds: 5 timeoutSeconds: 5 periodSeconds: 5 # configmap env env: - name: CFG_LABEL value: /instances/FTC_DEMO_FTC-PAAS-CONTRACT-SERVICE - name: CFG_ADDR valueFrom: configMapKeyRef: name: ftc-paas-contract-service-env-config key: CFG_ADDR - name: DR_CFG_ZOOKEEPER_ENV_URL valueFrom: configMapKeyRef: name: ftc-paas-contract-service-env-config key: DR_CFG_ZOOKEEPER_ENV_URL - name: CFG_FILES valueFrom: configMapKeyRef: name: ftc-paas-contract-service-env-config key: CFG_FILES # configMap volume volumeMounts: - name: applogs mountPath: /volume_logs/ volumes: - name: applogs hostPath: path: /opt/app_logs/ftc-paas-contract-service imagePullSecrets: - name: regcred # 設置jenkins全局變量 --- apiVersion: v1 kind: Service metadata: name: ftc-paas-contract-service namespace: ftc-demo labels: app: ftc-paas-contract-service spec: ports: - port: 8073 targetPort: 8073 selector: app: ftc-paas-contract-service ### 集群資源管理 - Node Taints: node.kubernetes.io=unschedulable:NoSchedule - LimitRange ftx-demo=limits.yaml apiVersion: v1 kind: LimitRange metadata: name: ftc-demo spec: limits: - max: cpu: "2" memory: 4Gi min: cpu: "1" memory: "2Gi" # maxLimitRequestRatio: # cpu: 3 # memory: 2 type: Pod - default: cpu: 2 memory: 4Gi defaultRequest: cpu: 1 memory: 2Gi max: cpu: "2" memory: 4Gi min: cpu: 500m memory: 1024Mi # maxLimitRequestRatio: # cpu: 4 # memory: 4 type: Container ### 近期任務計劃 - 應用日志搜集 - 集群監控 - helm 應用軟件倉庫 - 容器自動伸縮 - ### k8s 監控prometheus-operator [金融云grafana展示](http://grafana-demo.dalianyun.com/) [金融云prometheus 控制臺](http://prometheus-demo.dalianyun.com)