本次分片集群部署采用 1 个前端路由 (mongos),3 个分片 (shard),每个分片 (shard)都有 3 个副本 (replSet)。另外配置服务 (configsvr)也采用了 3 个副本。总启动容器数为 13 个。
前提
安装好 docker 和 docker-compose 的环境,具体安装教程查看链接
下载镜像
docker pull mongo:4.0.10
创建所需的文件和目录
mkdir -p /root/softdata/mongo # 工作目录cd /root/softdata/mongotouch docker-compose.yaml # docker-compose 启动文件touch docker-compose-auth.yaml # 启用用户授权的 docker-compose 文件openssl rand -base64 753 > keyfile # 创建授权秘钥文件chmod 400 keyfile # 修改权限
编写 docker-compose 文件
启动文件
version: '3'networks:mongo:services:# 配置服务副本 1mongo-config1:image: mongo:4.0.10restart: alwayscontainer_name: mongo-config1 # 容器名称working_dir: /datacommand: mongod --configsvr --replSet replConfig --directoryperdb --smallfilesports:- 27001:27019volumes: # 挂载的卷 [本机路径下的目录或文件]:[容器中所映射到的地址]- /root/softdata/mongo/data/config/config1/db/:/data/db/- /root/softdata/mongo/data/config/config1/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 配置服务副本 2mongo-config2:image: mongo:4.0.10restart: alwayscontainer_name: mongo-config2working_dir: /data/command: mongod --configsvr --replSet replConfig --directoryperdb --smallfilesports:- 27002:27019volumes:- /root/softdata/mongo/data/config/config2/db/:/data/db/- /root/softdata/mongo/data/config/config2/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 配置服务副本 3mongo-config3:image: mongo:4.0.10restart: alwayscontainer_name: mongo-config3working_dir: /data/command: mongod --configsvr --replSet replConfig --directoryperdb --smallfilesports:- 27003:27019volumes:- /root/softdata/mongo/data/config/config3/db/:/data/db/- /root/softdata/mongo/data/config/config3/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 1 的副本 1mongo-shard1-rs1:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard1-rs1working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard1ports:- 27011:27018volumes:- /root/softdata/mongo/data/shard1/shard1-rs1/db/:/data/db/- /root/softdata/mongo/data/shard1/shard1-rs1/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 1 的副本 2mongo-shard1-rs2:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard1-rs2working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard1ports:- 27012:27018volumes:- /root/softdata/mongo/data/shard1/shard1-rs2/db/:/data/db/- /root/softdata/mongo/data/shard1/shard1-rs2/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 1 的副本 3mongo-shard1-rs3:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard1-rs3working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard1ports:- 27013:27018volumes:- /root/softdata/mongo/data/shard1/shard1-rs3/db/:/data/db/- /root/softdata/mongo/data/shard1/shard1-rs3/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 2 的副本 1mongo-shard2-rs1:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard2-rs1working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard2ports:- 27021:27018volumes:- /root/softdata/mongo/data/shard2/shard2-rs1/db/:/data/db/- /root/softdata/mongo/data/shard2/shard2-rs1/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 2 的副本 2mongo-shard2-rs2:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard2-rs2working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard2ports:- 27022:27018volumes:- /root/softdata/mongo/data/shard2/shard2-rs2/db/:/data/db/- /root/softdata/mongo/data/shard2/shard2-rs2/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 2 的副本 3mongo-shard2-rs3:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard2-rs3working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard2ports:- 27023:27018volumes:- /root/softdata/mongo/data/shard2/shard2-rs3/db/:/data/db/- /root/softdata/mongo/data/shard2/shard2-rs3/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 3 的副本 1mongo-shard3-rs1:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard3-rs1working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard3ports:- 27031:27018volumes:- /root/softdata/mongo/data/shard3/shard3-rs1/db/:/data/db/- /root/softdata/mongo/data/shard3/shard3-rs1/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 3 的副本 2mongo-shard3-rs2:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard3-rs2working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard3ports:- 27032:27018volumes:- /root/softdata/mongo/data/shard3/shard3-rs2/db/:/data/db/- /root/softdata/mongo/data/shard3/shard3-rs2/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 3 的副本 3mongo-shard3-rs3:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard3-rs3working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard3ports:- 27033:27018volumes:- /root/softdata/mongo/data/shard3/shard3-rs3/db/:/data/db/- /root/softdata/mongo/data/shard3/shard3-rs3/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 前端路由 mongosmongos:image: mongo:4.0.10restart: alwayscontainer_name: mongosworking_dir: /datacommand: mongos --configdb replConfig/mongo-config1:27019,mongo-config2:27019,mongo-config3:27019 --bind_ip 0.0.0.0 --port 27017ports:- 27017:27017volumes:- /root/softdata/mongo/data/mongos/db/:/data/db/- /root/softdata/mongo/data/mongos/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfiledepends_on:- mongo-config1- mongo-config2- mongo-config3networks:- mongo
用户授权启动文件
version: '3'networks:mongo:services:# 配置服务副本 1mongo-config1:image: mongo:4.0.10restart: alwayscontainer_name: mongo-config1 # 容器名称working_dir: /datacommand: mongod --configsvr --replSet replConfig --directoryperdb --smallfiles --keyFile=/data/keyfileports:- 27001:27019volumes: # 挂载的卷 [本机路径下的目录或文件]:[容器中所映射到的地址]- /root/softdata/mongo/data/config/config1/db/:/data/db/- /root/softdata/mongo/data/config/config1/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 配置服务副本 2mongo-config2:image: mongo:4.0.10restart: alwayscontainer_name: mongo-config2working_dir: /data/command: mongod --configsvr --replSet replConfig --directoryperdb --smallfiles --keyFile=/data/keyfileports:- 27002:27019volumes:- /root/softdata/mongo/data/config/config2/db/:/data/db/- /root/softdata/mongo/data/config/config2/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 配置服务副本 3mongo-config3:image: mongo:4.0.10restart: alwayscontainer_name: mongo-config3working_dir: /data/command: mongod --configsvr --replSet replConfig --directoryperdb --smallfiles --keyFile=/data/keyfileports:- 27003:27019volumes:- /root/softdata/mongo/data/config/config3/db/:/data/db/- /root/softdata/mongo/data/config/config3/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 1 的副本 1mongo-shard1-rs1:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard1-rs1working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard1 --keyFile=/data/keyfileports:- 27011:27018volumes:- /root/softdata/mongo/data/shard1/shard1-rs1/db/:/data/db/- /root/softdata/mongo/data/shard1/shard1-rs1/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 1 的副本 2mongo-shard1-rs2:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard1-rs2working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard1 --keyFile=/data/keyfileports:- 27012:27018volumes:- /root/softdata/mongo/data/shard1/shard1-rs2/db/:/data/db/- /root/softdata/mongo/data/shard1/shard1-rs2/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 1 的副本 3mongo-shard1-rs3:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard1-rs3working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard1 --keyFile=/data/keyfileports:- 27013:27018volumes:- /root/softdata/mongo/data/shard1/shard1-rs3/db/:/data/db/- /root/softdata/mongo/data/shard1/shard1-rs3/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 2 的副本 1mongo-shard2-rs1:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard2-rs1working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard2 --keyFile=/data/keyfileports:- 27021:27018volumes:- /root/softdata/mongo/data/shard2/shard2-rs1/db/:/data/db/- /root/softdata/mongo/data/shard2/shard2-rs1/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 2 的副本 2mongo-shard2-rs2:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard2-rs2working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard2 --keyFile=/data/keyfileports:- 27022:27018volumes:- /root/softdata/mongo/data/shard2/shard2-rs2/db/:/data/db/- /root/softdata/mongo/data/shard2/shard2-rs2/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 2 的副本 3mongo-shard2-rs3:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard2-rs3working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard2 --keyFile=/data/keyfileports:- 27023:27018volumes:- /root/softdata/mongo/data/shard2/shard2-rs3/db/:/data/db/- /root/softdata/mongo/data/shard2/shard2-rs3/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 3 的副本 1mongo-shard3-rs1:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard3-rs1working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard3 --keyFile=/data/keyfileports:- 27031:27018volumes:- /root/softdata/mongo/data/shard3/shard3-rs1/db/:/data/db/- /root/softdata/mongo/data/shard3/shard3-rs1/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 3 的副本 2mongo-shard3-rs2:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard3-rs2working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard3 --keyFile=/data/keyfileports:- 27032:27018volumes:- /root/softdata/mongo/data/shard3/shard3-rs2/db/:/data/db/- /root/softdata/mongo/data/shard3/shard3-rs2/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 分片 3 的副本 3mongo-shard3-rs3:image: mongo:4.0.10restart: alwayscontainer_name: mongo-shard3-rs3working_dir: /datacommand: mongod --shardsvr --directoryperdb --replSet shard3 --keyFile=/data/keyfileports:- 27033:27018volumes:- /root/softdata/mongo/data/shard3/shard3-rs3/db/:/data/db/- /root/softdata/mongo/data/shard3/shard3-rs3/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfilenetworks:- mongo# 前端路由 mongosmongos:image: mongo:4.0.10restart: alwayscontainer_name: mongosworking_dir: /datacommand: mongos --configdb replConfig/mongo-config1:27019,mongo-config2:27019,mongo-config3:27019 --bind_ip 0.0.0.0 --port 27017 --keyFile=/data/keyfileports:- 27017:27017volumes:- /root/softdata/mongo/data/mongos/db/:/data/db/- /root/softdata/mongo/data/mongos/configdb/:/data/configdb/- /root/softdata/mongo/data/keyfile:/data/keyfiledepends_on:- mongo-config1- mongo-config2- mongo-config3networks:- mongo
启动容器
cd /root/softdata/mongodocker-compose -f docker-compose.yaml up -d
配置副本集和分片
配置 configsrv 的副本集
# 任意进入一个 config 的副本集容器docker exec -it mongo-config1 bash# 登陆数据库mongo --port 27019#初始化配置副本集> rs.initiate( {_id : "replConfig",configsvr: true,members : [{_id : 0, host : "mongo-config1:27019" },{_id : 1, host : "mongo-config2:27019" },{_id : 2, host : "mongo-config3:27019" }]})> rs.status(); # 查看当前 configsvr 副本集状态
配置 shard1 的副本集
# 任意进入一个 shard1 的副本集容器docker exec -it mongo-shard1-rs1 bash# 登陆数据库mongo --port 27018#初始化配置副本集> rs.initiate({_id : "shard1",members: [{ _id : 0, host : "mongo-shard1-rs1:27018" },{ _id : 1, host : "mongo-shard1-rs2:27018" },{ _id : 2, host : "mongo-shard1-rs3:27018" }]})shard1:PRIMARY> rs.status(); # 查看当前 shard1 副本集状态
配置 shard2 的副本集
# 任意进入一个 shard2 的副本集容器docker exec -it mongo-shard2-rs1 bash# 登陆数据库mongo --port 27018#初始化配置副本集> rs.initiate({_id : "shard2",members: [{ _id : 0, host : "mongo-shard2-rs1:27018" },{ _id : 1, host : "mongo-shard2-rs2:27018" },{ _id : 2, host : "mongo-shard2-rs3:27018" }]})shard2:PRIMARY> rs.status(); # 查看当前 shard2 副本集状态
配置 shard3 的副本集
# 任意进入一个 shard3 的副本集容器docker exec -it mongo-shard3-rs1 bash# 登陆数据库mongo --port 27018#初始化配置副本集> rs.initiate({_id : "shard3",members: [{ _id : 0, host : "mongo-shard3-rs1:27018" },{ _id : 1, host : "mongo-shard3-rs2:27018" },{ _id : 2, host : "mongo-shard3-rs3:27018" }]})shard3:PRIMARY> rs.status(); # 查看当前 shard3 副本集状态
配置分片
# 进入前端路由 mongos 容器docker exec -it mongos bash# 登陆数据库mongo# 设置分片配置mongos> sh.addShard("shard1/mongo-shard1-rs1:27018,mongo-shard1-rs2:27018,mongo-shard1-rs3:27018")mongos> sh.addShard("shard2/mongo-shard2-rs1:27018,mongo-shard2-rs2:27018,mongo-shard2-rs3:27018")mongos> sh.addShard("shard3/mongo-shard3-rs1:27018,mongo-shard3-rs2:27018,mongo-shard3-rs3:27018")mongos> sh.status() # 查看集群状态
创建相关用户
创建 root管理员账户
# 进入前端路由 mongos 容器docker exec -it mongos bash# 登陆数据库mongomongos> use admin # 使用 admin 数据库mongos> db.createUser({user: "admin",pwd: "123456",roles: [{ role: "root", db: "admin" }]}) # root所有权限mongos> use config # 使用 config 数据库mongos> db.createUser({user: "admin",pwd: "123456",roles: [{ role: "root", db: "admin" }]}) # root所有权限mongos> db.auth("admin","123456") # 登陆认证,返回1为验证成功
创建集群管理员账户
# 接着上一步mongos> use adminmongos> db.createUser({user:"clusterManager",pwd:"123456",roles:[{role:"clusterAdmin", db: "admin"},{role:"readWriteAnyDatabase", db: "admin"},{role:"dbOwner", db: "local"},{role:"userAdminAnyDatabase", db: "admin"},]})
创建 shard1 分片管理员账户
# 进入 shard1 的副本集的主节点容器docker exec -it mongo-shard1-rs1 bash# 登陆数据库mongo --port 27018shard1:PRIMARY> use adminshard1:PRIMARY> db.createUser({user:"shardManager",pwd:"123456",roles:[{role:"clusterAdmin", db: "admin"},{role:"readWriteAnyDatabase", db: "admin"},{role:"dbOwner", db: "local"},{role:"userAdminAnyDatabase", db: "admin"},]})
创建 shard2 分片管理员账户
# 进入 shard2 的副本集的主节点容器docker exec -it mongo-shard2-rs1 bash# 登陆数据库mongo --port 27018shard2:PRIMARY> use adminshard2:PRIMARY> db.createUser({user:"shardManager",pwd:"123456",roles:[{role:"clusterAdmin", db: "admin"},{role:"readWriteAnyDatabase", db: "admin"},{role:"dbOwner", db: "local"},{role:"userAdminAnyDatabase", db: "admin"},]})
创建 shard3 分片管理员账户
# 进入 shard3 的副本集的主节点容器docker exec -it mongo-shard3-rs1 bash# 登陆数据库mongo --port 27018shard3:PRIMARY> use adminshard3:PRIMARY> db.createUser({user:"shardManager",pwd:"123456",roles:[{role:"clusterAdmin", db: "admin"},{role:"readWriteAnyDatabase", db: "admin"},{role:"dbOwner", db: "local"},{role:"userAdminAnyDatabase", db: "admin"},]})
注意:在分片上创建管理员账户一定要在主节点上,也就是需要在PRIMARY上创建,而不能在SECONDARY上,否则无法创建;这里 3 个分片创建的管理员账户相同,是为了使用 mongo-connector 等共计做数据同步需要。
启用用户登录授权
# 使用 docker-compose-auth.yaml 文件重启所有容器docker-compose -f docker-compose-auth.yaml restart
验证授权
# 进入前端路由 mongos 容器docker exec -it mongos bash# 登陆数据库mongomongos> use admin # 使用 admin 数据库mongos> sh.status() # 现在无法查看分片状态,提示需要登录mongos> db.auth("admin","123456") # 登陆认证,返回1为验证成功mongos> sh.status() # 再次执行,可以查看分片状态
插入数据验证分片集群
创建 testdb 数据库实例,并启用分片
# 创建 testdb 数据库实例mongos> use testdbmongos> sh.enableSharding("testdb") # 启用分片# 创建集合 book,为其执行分片初始化mongos> use testdbmongos> db.createCollection("book")mongos> db.device.ensureIndex({createTime:1})mongos> sh.shardCollection("appdb.book", {bookId:"hashed"}, false, { numInitialChunks: 4})
往book集合写入 50000条记录,观察chunks的分布情况!
mongos> use testdbmongos> var cnt = 0;mongos> for(var i=0; i<100; i++){var dl = [];for(var j=0; j<500; j++){dl.push({"bookId" : "BBK-" + i + "-" + j,"type" : "Revision","version" : "IricSoneVB"+j,"title" : "这是一本书哦","subCount" : 10,"location" : "北京市朝阳区望京SOHO","author" : {"name" : "dmego","email" : "dmeago@gmail.com","gender" : "female"},"createTime" : new Date()});}cnt += dl.length;db.book.insertMany(dl);print("insert ", cnt);}
执行db.book.getShardDistribution() 查看分片情况
执行 db.book.stats() 查看 book 集合状态
