將原基礎的 `ingress-nginx` 一個副本提升到多個副本。然后再提供VIP進行訪問。 以下三種方式都可以實現高可用 1. LoadBalancer 2. nodeport + VIP 3. hostport + VIP - 其中 `LoadBalancer` 是在公有云上使用，不過自管集群也可以安裝 `Metallb` 也可以實現 `LoadBalancer` 的方式。 - `Metallb` 的官網為 https://metallb.universe.tf/installation/ 這里演示 `hostport + keepalived + nginx` 的組合方式。實現高可用和高并發。 ## 安裝nginx **創建目錄** ```shell mkdir -p /etc/nginx/{conf.d,stream} ``` **nginx主配置** ```shell cat <<-"EOF" | sudo tee /etc/nginx/nginx.conf > /dev/null user nginx; worker_processes auto; error_log /var/log/nginx/error.log notice; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; #tcp_nopush on; keepalive_timeout 65; #gzip on; include /etc/nginx/conf.d/*.conf; } stream { log_format proxy '$remote_addr $remote_port - [$time_local] $status $protocol ' '"$upstream_addr" "$upstream_bytes_sent" "$upstream_connect_time"'; include /etc/nginx/stream/*.conf; } EOF ``` **四層代理ingress服務** ```shell cat <<-"EOF" | sudo tee /etc/nginx/stream/ingress.conf > /dev/null upstream http { server 192.168.31.103:80 max_fails=3 fail_timeout=5s; server 192.168.31.79:80 max_fails=3 fail_timeout=5s; } server { listen 80; # proxy_protocol on; proxy_pass http; access_log /var/log/nginx/ingress_http_tcp_access.log proxy; error_log /var/log/nginx/ingress_http_tcp_error.log; } upstream https { server 192.168.31.103:443 max_fails=3 fail_timeout=5s; server 192.168.31.79:443 max_fails=3 fail_timeout=5s; } server { listen 443; # proxy_protocol on; proxy_pass https; access_log /var/log/nginx/ingress_https_tcp_access.log proxy; error_log /var/log/nginx/ingress_https_error.log; } EOF ``` > 注意：修改server替換成實際的 master節點 IP地址 **docker-compose配置** ```shell cat <<-EOF | sudo tee /etc/nginx/docker-compose.yaml > /dev/null version: "3" services: nginx: container_name: nginx image: nginx:1.21-alpine volumes: - "./stream:/etc/nginx/stream:ro" - "./conf.d:/etc/nginx/conf.d:ro" - "./nginx.conf:/etc/nginx/nginx.conf:ro" - "./logs:/var/log/nginx" - "/usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro" restart: always ports: - "6443:6443" - "80:80" - "443:443" EOF ``` **啟動nginx** ```shell docker-compose -f /etc/nginx/docker-compose.yaml up -d ``` ## 安裝keepalived **keepalived配置** ```shell $ sudo mkdir /etc/keepalived $ cat <<-EOF | sudo tee -a /etc/keepalived/keepalived.conf > /dev/null include /etc/keepalived/keepalived_ingress.conf EOF $ cat <<-EOF | sudo tee /etc/keepalived/keepalived_ingress.conf > /dev/null vrrp_script ingress { # 檢測腳本路徑 script "/etc/keepalived/chk_ingress.sh" # 執行檢測腳本的用戶 user root # 腳本調用之間的秒數 interval 1 # 轉換失敗所需的次數 fall 5 # 轉換成功所需的次數 rise 3 # 按此權重調整優先級 weight -50 } vrrp_instance ingress { # 狀態是主節點還是從節點 state MASTER # inside_network 的接口，由 vrrp 綁定。 interface eth0 # 虛擬路由id，根據該id進行組成主從架構 virtual_router_id 200 # 初始優先級 # 最后優先級權重計算方法 # (1) weight 為正數，priority - weight # (2) weight 為負數，priority + weight priority 200 # 加入集群的認證 authentication { auth_type PASS auth_pass pwd200 } # keepalivd配置成單播模式 ## 單播的源地址 unicast_src_ip 192.168.31.103 ## 單播的對端地址 unicast_peer { 192.168.31.79 } # vip 地址 virtual_ipaddress { 192.168.31.188 } # 健康檢查腳本 track_script { ingress } } EOF ``` **keepalived檢測腳本** ```shell $ cat <<-EOF | sudo tee /etc/keepalived/chk_ingress.sh > /dev/null #!/bin/sh count=\$(netstat -lntup | egrep ":443|:80" | wc -l) if [ "\$count" -ge 2 ];then # 退出狀態為0，代表檢查成功 exit 0 else # 退出狀態為1，代表檢查不成功 exit 1 fi EOF $ chmod +x /etc/keepalived/chk_ingress.sh ``` **keepalived的docker-compose** ```shell $ cat <<-EOF | sudo tee /etc/keepalived/docker-compose.yaml > /dev/null version: "3" services: keepalived: container_name: keepalived image: jiaxzeng/keepalived:2.2.7-alpine3.12 volumes: - "/usr/share/zoneinfo/Asia/Shanghai:/etc/localtime" - ".:/etc/keepalived" cap_add: - NET_ADMIN network_mode: "host" restart: always EOF ``` **啟動keepalived** ```shell docker-compose -f /etc/keepalived/docker-compose.yaml up -d ``` ## 修改ingress-nginx ```shell # 在 deploy 添加或修改replicas replicas: 2 # 在 deploy.spec.template.spec 下面添加affinity affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/name: ingress-nginx topologyKey: kubernetes.io/hostname ``` > 需要重啟ingress-nginx-controller容器 ## 附加iptables ```shell iptables -I INPUT -p tcp -m multiport --dports 80,443,8443 -m comment --comment "nginx ingress controller external ports" -j ACCEPT iptables -I INPUT -p tcp --dport 10086 -m comment --comment "haproxy stats ports" -j ACCEPT ``` > `80、443、8443` 是由 `ingress-nginx-controller` 暴露的端口