#### MCollective架構篇7-多MQ下MCollective高可用部署
存在這樣一種場景，當你的puppet基于mcollective環境搭建完成之后，需要考慮MQ的高可用，否則，MQ掛掉之后就不能用mco命令進行推送了哦。如何做MQ的高可用呢，其實有兩種方法：方法一：兩臺MQ做集群，通過復制隊列信息進行同步，節點訪問可通過浮動IP進行。方法二：兩臺MQ獨立，在MC Server端做failover，通過rabbtimq的plugins參數實現，可設置自動檢測，切換時間等等。
# 一、配置Rabbitmq
安裝（略），可參考[http://kisspuppet.com/2013/11/10/mcollective-middleware/](http://kisspuppet.com/2013/11/10/mcollective-middleware/)或[http://rsyslog.org/2013/11/10/mcollective-middleware/](http://rsyslog.org/2013/11/10/mcollective-middleware/)
### 1. 開啟插件rabbitmq_stomp
~~~
[root@linuxmaster1poc ~]# rabbitmq-plugins enable rabbitmq_stomp
The following plugins have been enabled:
rabbitmq_stomp
Plugin configuration has changed. Restart RabbitMQ for changes to take effect.
~~~
### 2. 添加tcp監聽端口和范圍
~~~
[root@linuxmaster1poc ~]# vim /etc/rabbitmq/rabbitmq.config
[
{rabbitmq_stomp, [{tcp_listeners, [61613]}]}
].
~~~
**備注：**可參考 [http://www.rabbitmq.com/stomp.html](http://www.rabbitmq.com/stomp.html)
### 3. 創建賬戶并設置權限
如果你以前配置過，建議將配置清空
~~~
[root@linuxmaster1poc ~]# rabbitmqctl stop_app
Stopping node rabbit@linuxmaster1poc ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl reset
Resetting node rabbit@linuxmaster1poc ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl start_app
Starting node rabbit@linuxmaster1poc ...
...done.
~~~
刪除默認用戶guest，添加三個用戶（web_admin-http訪問用，admin--管理員，mc_rabbitmq--mcollective鏈接用）
~~~
[root@linuxmaster1poc ~]# rabbitmqctl list_users
Listing users ...
guest [administrator]
...done.
[root@linuxmaster1poc ~]# rabbitmqctl delete_user guest
Deleting user "guest" ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl add_user mc_rabbitmq 123.com
Creating user "mc_rabbitmq" ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl add_user admin password=123.com
Creating user "admin" ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl add_user web_admin 123.com
Creating user "web_admin" ...
...done.
~~~
設置用戶的角色
~~~
[root@linuxmaster1poc ~]# rabbitmqctl set_user_tags admin administrator
Setting tags for user "admin" to [administrator] ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl set_user_tags web_admin monitoring
Setting tags for user "web_admin" to [monitoring] ...
...done.
~~~
創建虛擬主機組
~~~
[root@linuxmaster1poc ~]# rabbitmqctl add_vhost /mcollective
Creating vhost "/mcollective" ...
...done.
~~~
設置用戶訪問虛擬主機組的權限
~~~
[root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p "/mcollective" mc_rabbitmq ".*" ".*" ".*"
Setting permissions for user "mc_rabbitmq" in vhost "/mcollective" ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p "/mcollective" admin ".*" ".*" ".*"
Setting permissions for user "admin" in vhost "/mcollective" ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p "/mcollective" web_admin ".*" ".*" ".*"
Setting permissions for user "web_admin" in vhost "/mcollective" ...
...done.
~~~
重啟rabbitmq-server服務
~~~
[root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server restart
Restarting rabbitmq-server: SUCCESS
rabbitmq-server.
~~~
查看用戶以及角色是否創建成功
~~~
[root@linuxmaster1poc ~]# rabbitmqctl list_users
Listing users ...
admin [administrator]
mc_rabbitmq []
web_admin [monitoring]
...done.
~~~
查看虛擬主機組“/mcollective”中所有用戶的權限
~~~
[root@linuxmaster1poc ~]# rabbitmqctl list_permissions -p "/mcollective"
Listing permissions in vhost "/mcollective" ...
admin .* .* .*
mc_rabbitmq .* .* .*
web_admin .* .* .*
...done.
[root@linuxmaster1poc ~]#
~~~
### 4、登錄[http://192.168.100.120:15672/設置虛擬主機“/mcollective”的exchanges](http://192.168.100.120:15672/設置虛擬主機“/mcollective”的exchanges)
默認配置
~~~
[root@linuxmaster1poc ~]# rabbitmqctl list_exchanges -p "/mcollective"
Listing exchanges ...
direct
amq.direct direct
amq.fanout fanout
amq.headers headers
amq.match headers
amq.rabbitmq.trace topic
amq.topic topic
...done.
~~~
設置后更新配置
~~~
[root@linuxmaster1poc ~]# rabbitmqctl list_exchanges -p "/mcollective"
Listing exchanges ...
direct
amq.direct direct
amq.fanout fanout
amq.headers headers
amq.match headers
amq.rabbitmq.trace topic
amq.topic topic
mcollective_broadcast topic
mcollective_directed direct
...done.
~~~
**備注：**可參考官網設置 [https://www.rabbitmq.com/man/rabbitmqctl.1.man.html](https://www.rabbitmq.com/man/rabbitmqctl.1.man.html)
# 二、 配置MCollective:
### 1. 配置mcollective client端
~~~
[root@linuxmaster1poc testing]# cat /etc/mcollective/client.cfg
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
logger_type = console
#loglevel = debug
loglevel = warn
# Plugins
securityprovider = psk
plugin.psk = a36cd839414370e10fd281b8a38a4f48
direct_addressing = 1
connector = rabbitmq
plugin.rabbitmq.vhost = /mcollective #虛擬主機
plugin.rabbitmq.pool.size = 2 #設置地址池里有兩個mq
plugin.rabbitmq.initial_reconnect_delay = 0.01
plugin.rabbitmq.max_reconnect_delay = 30.0 #重連時間
plugin.rabbitmq.use_exponential_back_off = true
plugin.rabbitmq.back_off_multiplier = 2
plugin.rabbitmq.max_reconnect_attempts = 0
plugin.rabbitmq.randomize = false
plugin.rabbitmq.timeout = -1
plugin.rabbitmq.pool.1.host = 192.168.100.120
plugin.rabbitmq.pool.1.port = 61613
plugin.rabbitmq.pool.1.user = mc_rabbitmq
plugin.rabbitmq.pool.1.password = 123.com
plugin.rabbitmq.pool.1.ssl = false
plugin.rabbitmq.pool.2.host = 192.168.100.121
plugin.rabbitmq.pool.2.port = 61613
plugin.rabbitmq.pool.2.user = mc_rabbitmq
plugin.rabbitmq.pool.2.password = 123.com
plugin.rabbitmq.pool.2.ssl = false
# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml
~~~
### 2. 配置mcollective server端
~~~
[root@linux57poc tmp]# cat /etc/mcollective/server.cfg
# --Global--
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
logfile = /var/log/puppet/mcollective.log
loglevel = info
daemonize = 1
# --rabbitmq Plugins--
securityprovider = psk
plugin.psk = a36cd839414370e10fd281b8a38a4f48
direct_addressing = 1
connector = rabbitmq
plugin.rabbitmq.vhost = /mcollective
plugin.rabbitmq.pool.size = 2
plugin.rabbitmq.initial_reconnect_delay = 0.01
plugin.rabbitmq.max_reconnect_delay = 30.0
plugin.rabbitmq.use_exponential_back_off = true
plugin.rabbitmq.back_off_multiplier = 2
plugin.rabbitmq.max_reconnect_attempts = 0
plugin.rabbitmq.randomize = false
plugin.rabbitmq.timeout = -1
plugin.rabbitmq.pool.1.host = 192.168.100.120
plugin.rabbitmq.pool.1.port = 61613
plugin.rabbitmq.pool.1.user = mc_rabbitmq
plugin.rabbitmq.pool.1.password = 123.com
plugin.rabbitmq.pool.1.ssl = false
plugin.rabbitmq.pool.2.host = 192.168.100.121
plugin.rabbitmq.pool.2.port = 61613
plugin.rabbitmq.pool.2.user = mc_rabbitmq
plugin.rabbitmq.pool.2.password = 123.com
plugin.rabbitmq.pool.2.ssl = false
# --Puppet provider specific options--
plugin.service.provider = puppet
plugin.service.puppet.hasstatus = true
plugin.service.puppet.hasrestart = true
plugin.puppet.command = puppet agent
plugin.puppet.splay = true
plugin.puppet.splaylimit = 30
plugin.puppet.config = /etc/puppet/puppet.conf
# --Facts--
factsource = yaml
##factsource = facter
plugin.yaml = /etc/mcollective/facts.yaml
~~~
# 三、高可用測試
**特別注意：** 節點mcollective的server.cfg中pool是有優先級的，默認數字小的生效，這點需要注意，也就是說當所有節點都連接在MQ2上的時候，啟動MQ1，mco命令是無法使用的，因為它在運行的時候連接的是MQ1，而所有節點都連接在MQ2上。
### 1. 停止MQ1，查看切換狀態
**1.1 先看當前的節點連接狀態**
~~~
[root@linuxmaster1poc ~]# mco ping #查看連接的節點
linux57poc time=69.46 ms
linux58poc time=70.05 ms
linux64poc time=70.59 ms
---- ping statistics ----
3 replies max: 70.59 min: 69.46 avg: 70.03
[root@linuxmaster1poc ~]# mco shell "lsof -i:61613" #查看所有節點監聽的端口情況，可以看到目前都連接在linuxmaster1poc上。
Do you really want to send this command unfiltered? (y/n): y
Discovering hosts using the mc method for 2 second(s) .... 3
Host: linux64poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 36625 root 6u IPv4 27771 0t0 TCP linux64poc:40493->linuxmaster1poc:61613 (ESTABLISHED)
Host: linux58poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 11060 root 6u IPv4 34046 0t0 TCP linux58poc:36295->linuxmaster1poc:61613 (ESTABLISHED)
Host: linux57poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby 18076 root 6u IPv4 1351365 TCP linux57poc:24698->linuxmaster1poc:61613 (ESTABLISHED)
[root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server stop
Stopping rabbitmq-server: rabbitmq-server.
~~~
**1.2 再次運行mco查看切換狀態**
~~~
[root@linuxmaster1poc ~]# mco ping
linux58poc time=73.54 ms
linux64poc time=74.61 ms
linux57poc time=75.39 ms
---- ping statistics ----
3 replies max: 75.39 min: 73.54 avg: 74.51
[root@linuxmaster1poc ~]# mco shell "lsof -i:61613"
Do you really want to send this command unfiltered? (y/n): y
Discovering hosts using the mc method for 2 second(s) .... 3
Host: linux58poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 11060 root 6u IPv4 34046 0t0 TCP linux58poc:36295->linuxmaster1poc:61613 (CLOSE_WAIT)
ruby 11060 root 9u IPv4 34137 0t0 TCP linux58poc:47200->linuxmaster2poc:61613 (ESTABLISHED)
Host: linux64poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 36625 root 6u IPv4 27771 0t0 TCP linux64poc:40493->linuxmaster1poc:61613 (CLOSE_WAIT)
ruby 36625 root 8u IPv4 27877 0t0 TCP linux64poc:37472->linuxmaster2poc:61613 (ESTABLISHED)
Host: linux57poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby 18076 root 9u IPv4 1351484 TCP linux57poc:9309->linuxmaster2poc:61613 (ESTABLISHED)
~~~
通過日志查看
~~~
[root@linuxmaster1poc ~]# mco shell "lsof -i:61613"
Do you really want to send this command unfiltered? (y/n): y
Discovering hosts using the mc method for 2 second(s) .... 3
Host: linux58poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 11428 root 6u IPv4 34283 0t0 TCP linux58poc:36300->linuxmaster1poc:61613 (CLOSE_WAIT)
ruby 11428 root 8u IPv4 34338 0t0 TCP linux58poc:47205->linuxmaster2poc:61613 (ESTABLISHED)
Host: linux57poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby 18447 root 6u IPv4 1351559 TCP linux57poc:59343->linuxmaster1poc:61613 (CLOSE_WAIT)
ruby 18447 root 8u IPv4 1351622 TCP linux57poc:29757->linuxmaster2poc:61613 (ESTABLISHED)
Host: linux64poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 37054 root 4u IPv4 28036 0t0 TCP linux64poc:37476->linuxmaster2poc:61613 (ESTABLISHED)
ruby 37054 root 6u IPv4 27990 0t0 TCP linux64poc:40497->linuxmaster1poc:61613 (CLOSE_WAIT)
~~~
**總結：**可以看到之前的連接已經變成CLOSE_WAIT，新的連接被建立
### 2. 停止MQ2，啟動MQ1 查看切換狀態
~~~
[root@linuxmaster2poc rabbitmq]# /etc/init.d/rabbitmq-server stop
Stopping rabbitmq-server: rabbitmq-server.
[root@linux57poc service]# lsof -i:61613
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby 18447 root 6u IPv4 1351559 TCP linux57poc:59343->linuxmaster1poc:61613 (CLOSE_WAIT)
ruby 18447 root 8u IPv4 1351622 TCP linux57poc:29757->linuxmaster2poc:61613 (CLOSE_WAIT)
[root@linux58poc ~]# lsof -i:61613
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 11428 root 6u IPv4 34283 0t0 TCP linux58poc:36300->linuxmaster1poc:61613 (CLOSE_WAIT)
ruby 11428 root 8u IPv4 34338 0t0 TCP linux58poc:47205->linuxmaster2poc:61613 (CLOSE_WAIT)
[root@linux64poc ~]# lsof -i:61613
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 37054 root 4u IPv4 28036 0t0 TCP linux64poc:37476->linuxmaster2poc:61613 (CLOSE_WAIT)
ruby 37054 root 6u IPv4 27990 0t0 TCP linux64poc:40497->linuxmaster1poc:61613 (CLOSE_WAIT)
[root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server start
Starting rabbitmq-server: SUCCESS
rabbitmq-server.
~~~
根據 plugin.rabbitmq.max_reconnect_delay = 30.0，需要過最多30秒，mcollective服務端會重新建立連接請求
~~~
[root@linuxmaster1poc ~]# tailf /var/log/rabbitmq/rabbit\@linuxmaster1poc.log
=INFO REPORT==== 24-Dec-2013::11:00:45 ===
accepting STOMP connection <0.332.0> (192.168.100.126:36316 -> 192.168.100.120:61613)
=INFO REPORT==== 24-Dec-2013::11:00:45 ===
accepting STOMP connection <0.348.0> (192.168.100.125:18945 -> 192.168.100.120:61613)
=INFO REPORT==== 24-Dec-2013::11:00:45 ===
accepting STOMP connection <0.382.0> (192.168.100.127:40513 -> 192.168.100.120:61613)
[root@linuxmaster1poc ~]# mco ping
linux58poc time=70.60 ms
linux57poc time=71.32 ms
linux64poc time=111.56 ms
---- ping statistics ----
3 replies max: 111.56 min: 70.60 avg: 84.49
[root@linuxmaster1poc ~]# mco shell "lsof -i:61613"
Do you really want to send this command unfiltered? (y/n): y
Discovering hosts using the mc method for 2 second(s) .... 3
Host: linux58poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 11428 root 6u IPv4 34283 0t0 TCP linux58poc:36300->linuxmaster1poc:61613 (CLOSE_WAIT)
ruby 11428 root 8u IPv4 34338 0t0 TCP linux58poc:47205->linuxmaster2poc:61613 (CLOSE_WAIT)
ruby 11428 root 10u IPv4 34444 0t0 TCP linux58poc:36316->linuxmaster1poc:61613 (ESTABLISHED)
Host: linux57poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby 18447 root 10u IPv4 1351723 TCP linux57poc:18945->linuxmaster1poc:61613 (ESTABLISHED)
Host: linux64poc
Statuscode: 0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 37054 root 4u IPv4 28036 0t0 TCP linux64poc:37476->linuxmaster2poc:61613 (CLOSE_WAIT)
ruby 37054 root 6u IPv4 27990 0t0 TCP linux64poc:40497->linuxmaster1poc:61613 (CLOSE_WAIT)
ruby 37054 root 9u IPv4 28206 0t0 TCP linux64poc:40513->linuxmaster1poc:61613 (ESTABLISHED)
~~~
- 序
- 第一章：Puppet基礎篇
- 編寫此系列文檔的目的
- 如何學習和使用Puppet
- 安裝Puppet前期的準備工作
- 安裝、配置并使用Puppet
- 如何建立master和agent之間的認證關系
- Puppet更新方式的選型
- 編寫第一個完整測試模塊puppet
- 編寫第二個完整測試模塊yum
- Puppetmaster多環境配置
- 自定義fact實現的四種方式介紹
- 第二章：Puppet擴展篇
- 自定義fact結合ENC(hirea)的應用實踐
- 如何使用虛擬資源解決puppet沖突問題
- 如何擴展master的SSL傳輸性能（apache）
- 如何擴展master的SSL傳輸性能（nginx）
- 通過多進程增強master的負載均衡能力（nginx+mongrel）
- 通過橫向擴展puppetmaster增加架構的靈活性
- puppet代碼與版本控制系統的結合
- Puppet dashboard的部署及測試
- 第三章：MCollective架構篇
- MCollecitve架構的引入
- MCollective+MQ架構的部署
- Puppet插件的部署及測試
- MCollective各種插件的部署及測試
- MCollective安全性設計
- MQ的安全性設計
- 多MQ下MCollective高可用部署
- 第四章：Foreman架構的引入
- Foreman作為自動化運維工具為什么會如此強大
- 安裝前環境準備
- 安裝Foreman1.5架構（all-in-one）
- 安裝Foreman1.6架構（foreman與puppetmaster分離）
- 安裝Foreman1.7架構（源碼，僅測試使用）
- 整合puppetmaster
- Foreman結合mcollective完成push動作
- Foreman結合puppetssh完成push動作
- Foreman的ENC環境與fact環境的對比
- hostgroup如何轉換為本地的fact
- 智能變量與puppet模塊參數化類的結合
- Foreman報告系統的使用
- Foreman-proxy如何做負載均衡
- Foreman上如何展現代碼及文件內容
- Foreman如何和虛擬化管理軟件結合
- 如何借助Foreman完成自動化部署操作系統（一）
- 如何借助Foreman完成自動化部署操作系統（二）
- Foreman CLI（Hammer）工具的使用
- Foreman目前的不足之處