通過橫向擴展puppetmaster增加架構的靈活性 · Puppet運維實戰

**Puppet擴展篇6-通過橫向擴展puppetmaster增加架構的靈活性** puppetmaster橫向擴展將采用以下架構進行部署，也可以參考《puppet實戰》第246頁的內容。主機IP地址信息機用途表 **工作原理：** - 客戶端通過配置ca_server指定CA服務器，以達到獨立CA服務器的目的。 - CA服務器可以部署在多個機房。 - Master集群可以在同一機房配置負載均衡器，也可以使用DNS解析Puppet Master域名到不同機房的多臺服務器，通過DNS實現負載均衡。 ### 1、配置前的準備工作 **1.1、版本** puppet采用版本3.4.3負載均衡器采用nginx或haproxy進行測試部署 **1.2、主機名解析** ~~~ /etc/hosts 192.168.10.10 kspupt-ca1 192.168.10.9 kspupt-ca2 192.168.10.20 kspupt-ca 192.168.10.13 kspupt-lvs1 192.168.10.11 kspupt-m1 192.168.10.12 kspupt-m2 ~~~ **1.3、時間統一**（略） ### 2、PuppetCA認證服務器部署 PuppetCA的唯一目的就是簽署和撤銷證書。當PuppetCA服務不可用時，新的客戶端將不能獲得證書，從而會影響使用，而已簽發證書的客戶端缺不受影響。因此將CA進行獨立隊架構，這對容錯性而言是非常有必要的。 **2.1 安裝軟件包** ~~~ [root@kspupt-ca1 ~]# groupadd -g 3000 puppet [root@kspupt-ca1 ~]# useradd -u 3000 -g 3000 puppet [root@kspupt-ca1 ~]# yum install puppet puppet-server -y ~~~ **2.2 臨時配置VIP地址** ~~~ [root@kspupt-ca1 ~]# ip addr add 192.168.10.20/24 dev eth0 ~~~ **注：**后期CA配置成了高可用后，將VIP地址添加到高可用資源中即可，臨時先綁定在CA1上。 **2.3 生成證書** 使用puppet cert命令生成CA服務器與服務器域名證書。生成puppetca和puppetmaster兩個域名的授權證書文件。 ~~~ [root@kspupt-ca1 ~]# puppet cert --generate --dns_alt_names puppetca:puppet puppetca [root@kspupt-ca1 ~]# puppet cert --generate --dns_alt_names puppetmaster:puppet puppetmaster [root@kspupt-ca1 ~]# puppet cert --list --all 驗證 + "puppetca" (SHA256) 76:1D:C1:90:23:45:43:A2:41:4B:3B:92:32:C4:BE:31:38:61:5B:42:03:D0:22:28:53:5B:6F:5E:99:5A:B8:94 (alt names: "DNS:puppetca", "DNS:puppetca:puppet") + "puppetmaster" (SHA256) 0A:A2:DC:22:B8:4C:EB:31:B0:52:8F:B0:21:72:DD:EB:C7:B4:05:97:45:B3:EA:19:3A:28:69:29:04:35:0F:E7 (alt names: "DNS:puppetmaster", "DNS:puppetmaster:puppet") ~~~ **2.4 配置puppet.conf，添加標簽[master]** ~~~ [root@kspupt-ca1 ~]# vim /etc/puppet/puppet.conf [master] confdir = /etc/puppet certname = puppetca ca = true #開啟CA認證 ~~~ **2.5 啟動puppetmaster，CA部署完成** ~~~ [root@kspupt-ca1 ssl]# /etc/init.d/puppetmaster start [root@kspupt-ca1 ssl]# chkconfig puppetmaster on ~~~ **kspupt-ca2配置（略）** ### 3、PuppetMaster服務器部署 PuppetMaster服務器部署可采用默認的WebRick方式，也可以采用apache+passenger或nginx+passenger方式。 **3.1 WebRick方式：** **3.1.1 安裝軟件包** ~~~ [root@kspupt-m1 ~]# groupadd -g 3000 puppet [root@kspupt-m1 ~]# useradd -u 3000 -g 3000 puppet [root@kspupt-m1 ~]# yum install puppet puppet-server -y ~~~ **3.1.2 設置hosts文件** ~~~ [root@kspupt-m1 ~]# vim /etc/hosts 192.168.10.20 puppetca 192.168.10.11 puppetmaster ~~~ **3.1.3 創建證書目錄** ~~~ [root@kspupt-m1 ~]# mkdir /var/lib/puppet/ssl/{certs,ca,private_keys} -p ~~~ **3.1.4 將puppetca上生成的puppetmaster公鑰、私鑰和根證書復制到kspupt-m1** ~~~ [root@kspupt-m1 ssl]# scp -r root@192.168.10.39:/var/lib/puppet/ssl/ca/signed/puppetmaster.pem /var/lib/puppet/ssl/certs/puppetmaster.pem [root@kspupt-m1 ssl]# scp -r root@192.168.10.39:/var/lib/puppet/ssl/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem [root@kspupt-m1 ssl]# scp -r root@192.168.10.39:/var/lib/puppet/ssl/private_keys/puppetmaster.pem /var/lib/puppet/ssl/private_keys/puppetmaster.pem [root@kspupt-m1 gem]# scp -r root@192.168.10.39:/var/lib/puppet/ssl/ca/ca_crl.pem /var/lib/puppet/ssl/ca/ca_crl.pem ~~~ **3.1.5 配置puppet.conf，添加標簽[master]，關閉ca** ~~~ [root@kspupt-m1 ~]# vim /etc/puppet/puppet.conf [master] certname = puppetmaster ca = false #關閉CA認證 ~~~ **3.1.6 配置puppet.conf，修改標簽[agent]，增加server和ca_server字段** ~~~ [root@kspupt-m1 ~]# vim /etc/puppet/puppet.conf [agent] server = puppetmaster ca_server = puppetca ~~~ **3.1.7 啟動puppetmaster服務，Puppetmaster部署完成** ~~~ [root@kspupt-m1 ~]# /etc/init.d/puppetmaster start ~~~ **3.1.8 運行puppet命令進行本地證書申請** ~~~ [root@kspupt-m1 ~]# puppet agent -t Info: Creating a new SSL key for kspupt-m1 Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml Info: Creating a new SSL certificate request for kspupt-m1 Info: Certificate Request fingerprint (SHA256): 78:A5:F2:6C:F6:EE:0C:25:0C:EF:96:B8:B4:E6:78:74:A6:AA:67:81:6B:8F:36:AC:B2:37:B5:E0:C1:F0:11:67 Exiting; no certificate found and waitforcert is disabled ~~~ **3.1.9 登錄puppetca進行證書簽發** ~~~ [root@kspupt-ca ~]# puppet cert --sign kspupt-m1 Notice: Signed certificate request for kspupt-m1 Notice: Removing file Puppet::SSL::CertificateRequest kspupt-m1 at '/var/lib/puppet/ssl/ca/requests/kspupt-m1.pem' ~~~ **3.1.10 再次運行puppet命令進行測試連通性** ~~~ [root@kspupt-m1 ~]# puppet agent -t Info: Caching certificate for kspupt-m1 Info: Caching certificate_revocation_list for ca Info: Caching certificate for kspupt-m1 Info: Retrieving pluginfacts Info: Retrieving plugin Info: Caching catalog for kspupt-m1 Info: Applying configuration version '1409296030' Info: Creating state file /var/lib/puppet/state/state.yaml Notice: Finished catalog run in 0.02 seconds ~~~ **3.1.11 在kspupt-ca上申請本地證書** ~~~ [root@kspupt-ca ~]# vim /etc/puppet/puppet.conf [agent] server = puppetmaster ca_server = puppetca [root@kspupt-ca ~]# puppet agent -t [root@kspupt-ca ~]# puppet cert --sign kspupt-ca [root@kspupt-ca ~]# puppet agent -t ~~~ ### 3.2 Nginx+Passenger方式： **注：**可參考 [http://kisspuppet.com/2014/10/20/puppet_learning_ext4/](http://kisspuppet.com/2014/10/20/puppet_learning_ext4/) **3.2.1、安裝相關開發包** ~~~ [root@kspupt-m1 ~]# groupadd -g 3001 nginx [root@kspupt-m1 ~]# useradd -u 3001 -g 3001 nginx [root@kspupt-m1 ~]# yum install ruby-devel gcc make pcre-devel zlib-devel openssl-devel pam-devel curl-devel rpm-build ~~~ **3.2.2、安裝passenger（將gem軟件包copy到本地）** ~~~ [root@kspupt-m1 gem]# gem install rake rack passenger --no-rdoc --no-ri ~~~ **3.2.3、解壓nginx、pcre源碼包** ~~~ [root@kspupt-m1 gem]# tar xf pcre-8.32.tar.gz -C /usr/local/src/ [root@kspupt-m1 gem]# tar xf nginx-1.4.2.tar.gz -C /usr/local/src/ ~~~ **3.2.4、編譯并安裝nginx** ~~~ [root@kspupt-m1 ~]# cd /usr/local/src/nginx-1.4.2/ [root@kspupt-m1 nginx-1.4.2]# ./configure --user=nginx --group=nginx --prefix=/etc/nginx --with-http_stub_status_module --with-http_ssl_module --with-pcre=/usr/local/src/pcre-8.32 --add-module=`passenger-config --root`/ext/nginx [root@kspupt-m1 nginx-1.4.2]# make && make install ~~~ **3.2.5、與passenger結合** ~~~ [root@kspupt-m1 nginx-1.4.2]# mkdir -p /etc/puppet/rack/public [root@kspupt-m1 nginx-1.4.2]# cp /usr/share/puppet/ext/rack/config.ru /etc/puppet/rack/ [root@kspupt-m1 nginx-1.4.2]# chown -R puppet. /etc/puppet/rack/ ~~~ **3.2.6、復制啟動腳本到** ~~~ [root@kspupt-m1 init.d]# cp /root/gem/nginx /etc/init.d/ [root@kspupt-m1 ~]# chmod a+x /etc/init.d/nginx ~~~ **3.2.7、配置nginx** ~~~ [root@kspupt-m1 gem]# vim /etc/nginx/conf/nginx.conf user nginx nginx; worker_processes 1; pid /var/run/nginx.pid; events { worker_connections 1024; } http { passenger_root /usr/lib/ruby/gems/1.8/gems/passenger-4.0.19; passenger_ruby /usr/bin/ruby; include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; server { listen 8140 ssl; server_name puppetmaster; passenger_enabled on; passenger_set_cgi_param HTTP_X_CLIENT_DN $ssl_client_s_dn; passenger_set_cgi_param HTTP_X_CLIENT_VERIFY $ssl_client_verify; proxy_buffer_size 4000k; proxy_buffering on; proxy_buffers 32 1280k; proxy_busy_buffers_size 17680k; client_max_body_size 10m; client_body_buffer_size 4096k; access_log /var/log/nginx/puppet_access.log; error_log /var/log/nginx/puppet_error.log; root /etc/puppet/rack/public; ssl off; ssl_session_timeout 5m; ssl_certificate /var/lib/puppet/ssl/certs/puppetmaster.pem; ssl_certificate_key /var/lib/puppet/ssl/private_keys/puppetmaster.pem; ssl_client_certificate /var/lib/puppet/ssl/certs/ca.pem; ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem; ssl_verify_client optional; ssl_ciphers SSLv2:-LOW:-EXPORT:RC4+RSA; ssl_prefer_server_ciphers on; ssl_verify_depth 1; ssl_session_cache shared:SSL:128m; # File sections location /production/file_content/files/ { types { } default_type application/x-raw; alias /etc/puppet/files/; } } } ~~~ **3.2.8、配置puppet.conf** ~~~ [root@kspupt-m1 ~]# vim /etc/puppet/puppet.conf [master] certname = puppetmaster ca = false ssl_client_verify_header = HTTP_X_CLIENT_VERIFY ssl_client_header = HTTP_X_CLIENT_DN ~~~ **3.2.9、啟動nginx** ~~~ [root@kspupt-m1 gem]# mkdir /var/log/nginx/ [root@kspupt-m1 nginx-1.4.2]# /etc/init.d/puppetmaster stop [root@kspupt-m1 nginx-1.4.2]# chkconfig puppetmaster off [root@kspupt-m1 nginx-1.4.2]# /etc/init.d/nginx start [root@kspupt-m1 nginx-1.4.2]# chkconfig nginx on ~~~ **3.2.10、測試** 在多個節點發起puppet agent -t命令動作 ~~~ [root@kspupt-ca ~]# puppet agent -t [root@kspupt-m1 ~]# puppet agent -t [root@kspupt-m1 ~]# tailf /var/log/nginx/puppet_access.log ~~~ **tkpupt-m2安裝(略)** ### 4 Puppet LB負載均衡器部署 **4.1 puppet認證建立** **4.1.1、安裝軟件包** ~~~ [root@kspupt-lvs1 ~]# groupadd -g 3000 puppet [root@kspupt-lvs1 ~]# useradd -u 3000 -g 3000 puppet [root@kspupt-lvs1 ~]# yum install puppet ~~~ **4.1.2、編輯hosts文件** ~~~ [root@kspupt-lvs1 ~]# vim /etc/hosts 192.168.10.20 puppetca 192.168.10.11 puppetmaster 192.168.10.13 kspupt-lvs1 ~~~ **4.1.3、創建證書目錄** ~~~ [root@kspupt-lvs1 ~]# mkdir /var/lib/puppet/ssl/{certs,ca,private_keys} -p ~~~ **4.1.4、將kspupt-ca上生成的puppetmaster公鑰、私鑰和根證書復制到kspupt-lvs1** ~~~ [root@kspupt-lvs1 ssl]# scp -r root@192.168.10.10:/var/lib/puppet/ssl/ca/signed/puppetmaster.pem /var/lib/puppet/ssl/certs/puppetmaster.pem [root@kspupt-lvs1 ssl]# scp -r root@192.168.10.10:/var/lib/puppet/ssl/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem [root@kspupt-lvs1 ssl]# scp -r root@192.168.10.10:/var/lib/puppet/ssl/private_keys/puppetmaster.pem /var/lib/puppet/ssl/private_keys/puppetmaster.pem [root@kspupt-lvs1 ssl]# scp -r root@192.168.10.10:/var/lib/puppet/ssl/ca/ca_crl.pem /var/lib/puppet/ssl/ca/ ~~~ **4.1.5、配置puppet.conf，修改標簽[agent]，增加server和ca_server字段** ~~~ [root@kspupt-lvs1 ~]# vim /etc/puppet/puppet.conf [agent] server = puppetmaster ca_server = puppetca ~~~ **4.1.6、運行puppet命令進行本地證書申請** ~~~ [root@kspupt-lvs1 ~]# puppet agent -t ~~~ **4.1.7、登錄kspupt-ca進行證書簽發** ~~~ [root@kspupt-ca1 ~]# puppet cert --sign kspupt-lvs1 ~~~ **4.1.8、再次運行puppet命令進行測試連通性** ~~~ [root@kspupt-lvs1 ~]# puppet agent -t Info: Retrieving pluginfacts Info: Retrieving plugin Info: Caching catalog for kspupt-lvs1 Info: Applying configuration version '1409210667' ~~~ **4.2 安裝并配置nginx負載均衡器** **4.2.1、安裝nginx軟件** ~~~ [root@kspupt-lvs1 ~]# groupadd -g 3001 nginx [root@kspupt-lvs1 ~]# useradd -u 3001 -g 3001 nginx [root@kspupt-lvs1 ~]# yum install nginx ~~~ **4.2.2、臨時設置VIP地址（后面通過高可用軟件代替）** ~~~ [root@kspupt-lvs1 ~]# ip addr add 192.168.10.18/24 dev eth0 ~~~ **4.2.3、配置nginx虛擬主機，添加upstrem** ~~~ [root@kspupt-lvs1 ~]# vim /etc/nginx/conf.d/puppetmaster.conf upstream puppet-master { server 192.168.10.11:8140; server 192.168.10.12:8140; } server { listen 8140 ssl; server_name puppetmaster; access_log /var/log/nginx/puppet_access.log; error_log /var/log/nginx/puppet_error.log; ssl_protocols SSLv3 TLSv1; ssl_ciphers ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP; proxy_set_header X-SSL-Subject $ssl_client_s_dn; proxy_set_header X-Client-DN $ssl_client_s_dn; proxy_set_header X-Client-Verify $ssl_client_verify; client_max_body_size 100m; client_body_buffer_size 1024k; proxy_buffer_size 100m; proxy_buffers 8 100m; proxy_busy_buffers_size 100m; proxy_temp_file_write_size 100m; proxy_read_timeout 500; ssl on; ssl_session_timeout 5m; ssl_certificate /var/lib/puppet/ssl/certs/puppetmaster.pem; ssl_certificate_key /var/lib/puppet/ssl/private_keys/puppetmaster.pem; ssl_client_certificate /var/lib/puppet/ssl/certs/ca.pem; ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem; ssl_verify_client optional; ssl_prefer_server_ciphers on; ssl_verify_depth 1; ssl_session_cache shared:SSL:128m; location / { proxy_redirect off; proxy_pass https://puppet-master; } } ~~~ **4.2.4、編輯hosts文件，puppetmaster解析指向VIP** ~~~ [root@kspupt-lvs1 ~]# vim /etc/hosts 192.168.10.20 puppetca 192.168.10.18 puppetmaster 192.168.10.13 kspupt-lvs1 ~~~ **4.2.5、修改kspupt-ca和kspupt-m1的hosts文件puppetmaster解析** ~~~ [root@kspupt-ca1 ~]# vim /etc/hosts 192.168.10.20 puppetca 192.168.10.18 puppetmaster [root@kspupt-m1 ~]# vim /etc/hosts 192.168.10.20 puppetca 192.168.10.18 puppetmaster ~~~ **4.2.6、啟動nginx服務器** ~~~ [root@kspupt-lvs1 ~]# /etc/init.d/nginx start ~~~ **4.2.7、再次運行puppet命令進行測試連通性** ~~~ [root@kspupt-ca1 ~]# puppet agent -t [root@kspupt-m1 ~]# puppet agent -t [root@kspupt-lvs1 ~]# puppet agent -t [root@kspupt-m1 ~]# tailf /var/log/nginx/puppet_access.log [root@kspupt-lvs1 ~]# tailf /var/log/nginx/puppet_access.log ~~~ **kspupt-lvs2(略)** **4.3 HAproxy負載均衡配置參考** ~~~ [root@kspupt-lvs2 ~]# cat /etc/haproxy/haproxy.cfg listen admin_stats bind 0.0.0.0:8080 mode http stats refresh 5s stats enable stats hide-version stats realm Haproxy\ Statistics stats uri /haproxy stats auth admin:password listen puppetmaster *:8140 mode tcp option ssl-hello-chk # option tcplog #balance source # balance roundrobin balance source server kspupt-m1 kspupt-m1:8140 check inter 2000 fall 3 server kspupt-m2 kspupt-m2:8140 check inter 2000 fall 3 ~~~