mongodb3.6集群搭建:分片+副本集 準備環境： 10.248.56.192 MongoDB1 10.248.56.195 MongoDB2 10.248.56.209 MongoDB3 1、準備yum源 在MongoDB1 上安裝httpd yum -y install httpd 啟動httpd服務： systemctl start httpd 將以下包上傳至MongoDB1 主機 mongodb-org-3.6.4-1.el7.x86_64.rpm mongodb-org-mongos-3.6.4-1.el7.x86_64.rpm mongodb-org-server-3.6.4-1.el7.x86_64.rpm mongodb-org-shell-3.6.4-1.el7.x86_64.rpm mongodb-org-tools-3.6.4-1.el7.x86_64.rpm 將以上包移至httpd的根目錄： cp -r mogo/ /var/www/html 制作yum源： 安裝createrepo yum -y install createrepo cd /var/www/html createrepo mogo/ 在三臺主機編寫repo文件： [root@10-248-56-195 ~]# cat /etc/yum.repos.d/mogo.repo [maridb] name=mogo baseurl=http://10.248.56.192/mogo gpgcheck=0 enabled=1 開始部署： 三臺主機都執行： yum -y install mongodb-org 1、路徑規劃并創建 分別在每臺機器建立conf、mongos、config、shard1、shard2、shard3目錄，因為mongos不存儲數據，只需要建立日志文件目錄即可。 配置文件路徑： mkdir -p /etc/mongod/conf.d pid文件路徑 mkdir -p /var/run/mongodb 數據存儲路徑 #config server數據存儲路徑 mkdir -p /data1/mongoconfig/data #shard server數據存儲路徑 mkdir -p /data1/mongo/shard1/data mkdir -p /data1/mongo/shard2/data mkdir -p /data1/mongo/shard3/data chown -R mongod:mongod /var/lib/mongo chown -R mongod:mongod /data1/mongo/ chown -R mongod:mongod /data1/mongoconfig/ 日志文件路徑 /var/log/mongodb 2、config server配置服務器 mongodb3.4以后要求配置服務器也創建副本集，不然集群搭建不成功。 添加配置文件 vi /etc/mongod/conf.d/config.conf ## 配置文件內容 systemLog: destination: file logAppend: true path: /var/log/mongodb/configsvr.log storage: dbPath: /data1/mongoconfig/data journal: enabled: true processManagement: fork: true pidFilePath: /var/run/mongodb/configsvr.pid timeZoneInfo: /usr/share/zoneinfo net: port: 21000 bindIp: 0.0.0.0 maxIncomingConnections: 20000 replication: replSetName: csReplSet sharding: clusterRole: configsvr 創建systemctl unit文件vim /usr/lib/systemd/system/mongod-configsvr.service [Unit] Description=Mongodb Config Server After=network.target Documentation=https://docs.mongodb.org/manual [Service] User=mongod Group=mongod Environment="OPTIONS=-f /etc/mongod/conf.d/config.conf" ExecStart=/usr/bin/mongod $OPTIONS ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb PermissionsStartOnly=true PIDFile=/var/run/mongodb/configsvr.pid Type=forking #file size LimitFSIZE=infinity #cpu time LimitCPU=infinity #virtual memory size LimitAS=infinity #open files LimitNOFILE=64000 #processes/threads LimitNPROC=64000 #locked memory LimitMEMLOCK=infinity #total threads (user+kernel) TasksMax=infinity TasksAccounting=false #Recommended limits for for mongod as specified in #http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings [Install] WantedBy=multi-user.target 啟動三臺服務器的config server systemctl enable mongod-configsvr systemctl start mongod-configsvr 登錄任意一臺配置服務器，初始化配置副本集 #連接 mongo 127.0.0.1:21000 #config變量 config = { _id : "csReplSet", members : [ {_id : 1, host : "10.248.56.192:21000" }, {_id : 2, host : "10.248.56.195:21000" }, {_id : 3, host : "10.248.56.209:21000" } ] } #初始化副本集 rs.initiate(config) 其中，"_id" : " csReplSet "應與配置文件中配置的 replicaction.replSetName 一致，"members" 中的 "host" 為三個節點的 ip 和 port 3、配置分片副本集(三臺機器) 1) 設置第一個分片副本集 添加配置文件 vi /etc/mongod/conf.d/shard1.conf ##配置文件內容 systemLog: destination: file logAppend: true path: /var/log/mongodb/shard1.log storage: dbPath: /data1/mongo/shard1/data journal: enabled: true processManagement: fork: true pidFilePath: /var/run/mongodb/shard1.pid timeZoneInfo: /usr/share/zoneinfo net: port: 27001 bindIp: 0.0.0.0 maxIncomingConnections: 20000 replication: replSetName: shard1 sharding: clusterRole: shardsvr 創建systemctl unit文件 vim /usr/lib/systemd/system/mongod-shard1.service [Unit] Description=Mongodb Shard1 Server After=network.target Documentation=https://docs.mongodb.org/manual [Service] User=mongod Group=mongod Environment="OPTIONS=-f /etc/mongod/conf.d/shard1.conf" ExecStart=/usr/bin/mongod $OPTIONS ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb PermissionsStartOnly=true PIDFile=/var/run/mongodb/shard1.pid Type=forking #file size LimitFSIZE=infinity #cpu time LimitCPU=infinity #virtual memory size LimitAS=infinity #open files LimitNOFILE=64000 #processes/threads LimitNPROC=64000 #locked memory LimitMEMLOCK=infinity #total threads (user+kernel) TasksMax=infinity TasksAccounting=false #Recommended limits for for mongod as specified in #http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings [Install] WantedBy=multi-user.target 啟動三臺服務器的shard1 server systemctl enable mongod-shard1 systemctl start mongod-shard1 登陸任意一臺非仲裁節點服務器，初始化副本集 mongo 127.0.0.1:27001 #使用admin數據庫 use admin #定義副本集配置，第三個節點的 "arbiterOnly":true 代表其為仲裁節點。 config = { _id : "shard1", members : [ {_id : 1, host : "10.248.56.192:27001" , priority:2}, {_id : 2, host : "10.248.56.195:27001" , priority:1}, {_id : 3, host : "10.248.56.209:27001" , arbiterOnly: true } ] } #初始化副本集配置 rs.initiate(config); 2) 設置第二個分片副本集 添加配置文件 vi /etc/mongod/conf.d/shard2.conf ##配置文件內容 systemLog: destination: file logAppend: true path: /var/log/mongodb/shard2.log storage: dbPath: /data1/mongo/shard2/data journal: enabled: true processManagement: fork: true pidFilePath: /var/run/mongodb/shard2.pid timeZoneInfo: /usr/share/zoneinfo net: port: 27002 bindIp: 0.0.0.0 maxIncomingConnections: 20000 replication: replSetName: shard2 sharding: clusterRole: shardsvr 創建systemctl unit文件vim /usr/lib/systemd/system/mongod-shard2.service [Unit] Description=Mongodb Shard2 Server After=network.target Documentation=https://docs.mongodb.org/manual [Service] User=mongod Group=mongod Environment="OPTIONS=-f /etc/mongod/conf.d/shard2.conf" ExecStart=/usr/bin/mongod $OPTIONS ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb PermissionsStartOnly=true PIDFile=/var/run/mongodb/shard2.pid Type=forking #file size LimitFSIZE=infinity #cpu time LimitCPU=infinity #virtual memory size LimitAS=infinity #open files LimitNOFILE=64000 #processes/threads LimitNPROC=64000 #locked memory LimitMEMLOCK=infinity #total threads (user+kernel) TasksMax=infinity TasksAccounting=false #Recommended limits for for mongod as specified in #http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings [Install] WantedBy=multi-user.target 啟動三臺服務器的shard2 server systemctl enable mongod-shard2 systemctl start mongod-shard2 登陸任意一臺非仲裁節點服務器，初始化副本集 mongo 127.0.0.1:27002 #使用admin數據庫 use admin #定義副本集配置，第一個節點的 "arbiterOnly":true 代表其為仲裁節點。 config = { _id : "shard2", members : [ {_id : 1, host : "10.248.56.192:27002" , arbiterOnly: true }, {_id : 2, host : "10.248.56.195:27002" , priority:2}, {_id : 3, host : "10.248.56.209:27002" , priority:1} ] } #初始化副本集配置 rs.initiate(config); 3) 設置第三個分片副本集 添加配置文件 vi /etc/mongod/conf.d/shard3.conf ##配置文件內容 systemLog: destination: file logAppend: true path: /var/log/mongodb/shard3.log storage: dbPath: /data1/mongo/shard3/data journal: enabled: true processManagement: fork: true pidFilePath: /var/run/mongodb/shard3.pid timeZoneInfo: /usr/share/zoneinfo net: port: 27003 bindIp: 0.0.0.0 maxIncomingConnections: 20000 replication: replSetName: shard3 sharding: clusterRole: shardsvr 創建systemctl unit文件 vim /usr/lib/systemd/system/mongod-shard3.service [Unit] Description=Mongodb Shard3 Server After=network.target Documentation=https://docs.mongodb.org/manual [Service] User=mongod Group=mongod Environment="OPTIONS=-f /etc/mongod/conf.d/shard3.conf" ExecStart=/usr/bin/mongod $OPTIONS ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb PermissionsStartOnly=true PIDFile=/var/run/mongodb/shard3.pid Type=forking #file size LimitFSIZE=infinity #cpu time LimitCPU=infinity #virtual memory size LimitAS=infinity #open files LimitNOFILE=64000 #processes/threads LimitNPROC=64000 #locked memory LimitMEMLOCK=infinity #total threads (user+kernel) TasksMax=infinity TasksAccounting=false #Recommended limits for for mongod as specified in #http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings [Install] WantedBy=multi-user.target 啟動三臺服務器的shard3 server systemctl enable mongod-shard3 systemctl start mongod-shard3 登陸任意一臺非仲裁節點服務器，初始化副本集 mongo 127.0.0.1:27003 #使用admin數據庫 use admin #定義副本集配置，第二個節點的 "arbiterOnly":true 代表其為仲裁節點。 config = { _id : "shard3", members : [ {_id : 1, host : "10.248.56.192:27003" , priority:1}, {_id : 2, host : "10.248.56.195:27003" , arbiterOnly: true}, {_id : 3, host : "10.248.56.209:27003" , priority:2} ] } #初始化副本集配置 rs.initiate(config); 4、配置路由服務器 mongos 先啟動配置服務器和分片服務器,后啟動路由實例啟動路由實例:（三臺機器） vi /etc/mongod/conf.d/mongos.conf ##配置文件內容 systemLog: destination: file logAppend: true path: /var/log/mongodb/mongos.log processManagement: fork: true pidFilePath: /var/run/mongodb/mongos.pid timeZoneInfo: /usr/share/zoneinfo net: port: 20000 bindIp: 0.0.0.0 maxIncomingConnections: 20000 sharding: configDB: csReplSet/10.248.56.192:21000, 10.248.56.195:21000, 10.248.56.209:21000 #注意監聽的配置服務器,只能有1個或者3個 csReplSet為配置服務器的副本集名字 configDB: csReplSet/10.248.56.192:21000, 10.248.56.195:21000, 10.248.56.209:21000 創建systemctl unit文件 vim /usr/lib/systemd/system/mongod-mongos.service [Unit] Description=Mongodb Mongos Server After=network.target mongod-shard1.service mongod-shard2.service mongod-shard3.service Documentation=https://docs.mongodb.org/manual [Service] User=mongod Group=mongod Environment="OPTIONS=-f /etc/mongod/conf.d/mongos.conf" ExecStart=/usr/bin/mongos $OPTIONS ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb PermissionsStartOnly=true PIDFile=/var/run/mongodb/mongos.pid Type=forking #file size LimitFSIZE=infinity #cpu time LimitCPU=infinity #virtual memory size LimitAS=infinity #open files LimitNOFILE=64000 #processes/threads LimitNPROC=64000 #locked memory LimitMEMLOCK=infinity #total threads (user+kernel) TasksMax=infinity TasksAccounting=false #Recommended limits for for mongod as specified in #http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings [Install] WantedBy=multi-user.target 啟動三臺服務器的mongos server systemctl enable mongod-mongos systemctl start mongod-mongos 5、啟用分片 目前搭建了mongodb配置服務器、路由服務器，各個分片服務器，不過應用程序連接到mongos路由服務器并不能使用分片機制，還需要在程序里設置分片配置，讓分片生效。 登陸任意一臺mongos mongo 127.0.0.1:20000 #使用admin數據庫 use admin #串聯路由服務器與分配副本集 sh.addShard("shard1/10.248.56.192:27001,10.248.56.195:27001,10.248.56.209:27001") sh.addShard("shard2/10.248.56.192:27002,10.248.56.195:27002,10.248.56.209:27002") sh.addShard("shard3/10.248.56.192:27003,10.248.56.195:27003,10.248.56.209:27003") #查看集群狀態 sh.status() mongos> sh.status() --- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : 6, "clusterId" : ObjectId("5a86b6255d128f35cb22de20") } shards: { "_id" : "shard1", "host" : "shard1/10.248.56.192:27001,10.248.56.195:27001", "state" : 1 } { "_id" : "shard2", "host" : "shard2/10.248.56.195:27002,10.248.56.209:27002", "state" : 1 } { "_id" : "shard3", "host" : "shard3/10.248.56.192:27003,10.248.56.209:27003", "state" : 1 } active mongoses: "3.6.2" : 3 autosplit: Currently enabled: yes balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: No recent migrations databases: { "_id" : "config", "primary" : "config", "partitioned" : true } 6、測試 目前配置服務、路由服務、分片服務、副本集服務都已經串聯起來了，但我們的目的是希望插入數據，數據能夠自動分片。連接在mongos上，準備讓指定的數據庫、指定的集合分片生效。 mongo 127.0.0.1:20000 #設置分片chunk大小 use config db.settings.save({ "_id" : "chunksize", "value" : 1 }) 設置1M是為了測試，否則要插入大量數據才能分片。 #指定test分片生效 sh.enableSharding("test") #指定數據庫里需要分片的集合和片鍵 use test db.users.createIndex({user_id : 1}) use admin sh.shardCollection("test.users", {user_id: 1}) 我們設置testdb的 table1 表需要分片，根據 user_id 自動分片到 shard1 ，shard2，shard3 上面去。要這樣設置是因為不是所有mongodb 的數據庫和表 都需要分片！ 測試分片配置結果 mongo 127.0.0.1:20000 use test; for (var i = 1; i <=1000000; i++){ db.users.save({user_id: i, username: "user"+i}); } #查看分片情況如下，部分無關信息省掉了 sh.status() --- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : 6, "clusterId" : ObjectId("5a8919101e5f8edd5feddf36") } shards: { "_id" : "shard1", "host" : "shard1/192.168.56.101:27001,192.168.56.102:27001", "state" : 1 } { "_id" : "shard2", "host" : "shard2/192.168.56.102:27002,192.168.56.103:27002", "state" : 1 } { "_id" : "shard3", "host" : "shard3/192.168.56.101:27003,192.168.56.103:27003", "state" : 1 } active mongoses: "3.6.2" : 3 autosplit: Currently enabled: yes balancer: Currently enabled: yes Currently running: no Collections with active migrations: test.users started at Sun Feb 18 2018 21:14:30 GMT+0800 (CST) Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: 43 : Success 1 : Failed with error 'aborted', from shard1 to shard3 139 : Failed with error 'aborted', from shard1 to shard2 databases: … { "_id" : "test", "primary" : "shard3", "partitioned" : true } test.users shard key: { "user_id" : 1 } unique: false balancing: true chunks: shard1 34 shard2 31 shard3 31 too many chunks to print, use verbose if you want to force print 可以看到數據分到3個分片，各自分片chunk 數為： shard1 : 34，shard2 : 31，shard3 : 31。已經成功了！ 后期運維 啟動 mongodb的啟動順序是，先啟動配置服務器，在啟動分片，最后啟動mongos. systemctl start mongod-configsvr systemctl start mongod-shard1 systemctl start mongod-shard2 systemctl start mongod-shard3 systemctl start mongod-mongos 關閉: systemctl stop mongod-mongos systemctl stop mongod-shard3 systemctl stop mongod-shard2 systemctl stop mongod-shard1 systemctl stop mongod-configsvr