[关闭]
@zero1036 2017-03-29T00:41:39.000000Z 字数 6438 阅读 2431

mongodb自动分片

Mongodb

操作步骤

1、config配置库启动：mongodb --dbpath=D:\Data\dbs\shard\config --port 2222
2、mongos指定config库启动：mongos --port 3333 --configdb=morton:2222
3、mongod分片库1启动：mongod --dbpath=D:\Data\dbs\shard\s1 --port 4444
4、mongod分片库2启动：mongod --dbpath=D:\Data\dbs\shard\s2 --port 5555
5、连接mongos，指定分片addshard：mongo morton:3333/admin
db.runCommand({"addshard":"morton:4444",allowLocal:true})
==> { "shardAdded" : "shard0000", "ok" : 1 }
db.runCommand({"addshard":"morton:5555",allowLocal:true})
==> { "shardAdded" : "shard0001", "ok" : 1 }

6、当前Mongos已集群，但未指定分片键，mongos仍然未知如何切分数据
7、开启分片功能：db.runCommand({"enablesharding":"test"})
8、指定集合中分片的片键：db.runCommand({"shardCollection":"test.ActivityResult","key":{"ActivityId":1}})

9、查看分片状态：db.printShardingStatus()；为了查看结果，导入ActivityResult

注意：
1、定长集合不能分片，否则报错：`{ "ok" : 0, "errmsg" : "can't shard capped collection" }
2、必须为分片键Key建立索引，否则报错：please create an index that starts with the shard key before sharding

  1. db.runCommand({"shardCollection":"test.ActivityResult","key":{"ActivityId":1}})
  2. {
  3.     "proposedKey" : {
  4.         "ActivityId" : 1
  5.     },
  6.     "curIndexes" : [
  7.         {
  8.             "v" : 1,
  9.             "key" : {
  10.                 "_id" : 1
  11.             },
  12.             "name" : "_id_",
  13.             "ns" : "test.ActivityResult"
  14.         }
  15.     ],
  16.     "ok" : 0,
  17.     "errmsg" : "please create an index that starts with the shard key before sharding."
  18. }

3、建立分片键Key后，再插入数据，才会对数据分片；

性能测试

插入性能

  1. for(var i =1; i <= 100000; i++){
  2.     db.getCollection('ActivityResultShard').insert({
  3.     "Aid" : "56aacfd4d4307006200d0bac",
  4.     "Cid" : 111,
  5.     "At" : "56aacfd4d4307006200d0bac",
  6.     "Time" : ISODate("2017-03-28T09:35:05.779Z"),
  7.     "EvtN" : null,
  8.     "Evt" : null,
  9.     "As" : 0,
  10.     "Mis" : {
  11.         "Mid" : "e3196ec3-b624-4b11-a5e7-df7c4400ba75",
  12.         "Mt" : "asdf",
  13.         "Ms" : 1,
  14.         "Filter" : [ 
  15.             {
  16.                 "Fs" : true,
  17.                 "Exp" : "Evt.Amount>0",
  18.                 "Pro" : "GetInfo"
  19.             }
  20.         ],
  21.         "Rew" : [ 
  22.             {
  23.                 "Rid" : "123",
  24.                 "Ra" : "",
  25.                 "Rs" : 0
  26.             }
  27.         ]
  28.     }
  29. });
  30. }

硬件对比

硬件 客户端 基数 样本数 常驻内存 索引 耗时 tps
1.72 8G VMware 64位Linux Robomongo 0 10万 295m _id，大小：24m 638s 157
1.72 8G VMware 64位Linux mongo shell 0 10万 _id，大小：24m 116s 862
1.72 8G VMware 64位Linux C# Client 单线程 0 10万 _id，大小：24m 268s 373
8G i7 SSD 64位win7 mongo shell 0 10万 _id，大小：24m 27s 3703
8G i5 SSD 64位win7 mongo shell 0 10万 _id，大小：24m 54s 1852

索引对比
测试硬件：1.72 8G VMware 64位Linux
操作客户端：mongo shell （单线程）

基数 样本数 索引 索引大小 耗时 tps
0 10万 _id 24m 116s 862
0 10万 _id、Aid、Cid、Mis.Mid 24m 54s 1852

附录

config配置库启动日志：

  1. 2017-02-21T09:03:32.563+0800 [initandlisten] MongoDB starting : pid=6980 port=2222 dbpath=D:\Data\dbs\shard\config 64-bit host=PC-linzc
  2. 2017-02-21T09:03:32.563+0800 [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
  3. 2017-02-21T09:03:32.563+0800 [initandlisten] db version v2.6.5
  4. 2017-02-21T09:03:32.563+0800 [initandlisten] git version: e99d4fcb4279c0279796f237aa92fe3b64560bf6
  5. 2017-02-21T09:03:32.563+0800 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
  6. 2017-02-21T09:03:32.563+0800 [initandlisten] allocator: system
  7. 2017-02-21T09:03:32.563+0800 [initandlisten] options: { net: { port: 2222 }, storage: { dbPath: "D:\Data\dbs\shard\config" } }
  8. 2017-02-21T09:03:32.565+0800 [initandlisten] journal dir=D:\Data\dbs\shard\config\journal
  9. 2017-02-21T09:03:32.565+0800 [initandlisten] recover : no journal files present, no recovery needed
  10. 2017-02-21T09:03:32.573+0800 [initandlisten] allocating new ns file D:\Data\dbs\shard\config\local.ns, filling with zeroes...
  11. 2017-02-21T09:03:32.623+0800 [FileAllocator] allocating new datafile D:\Data\dbs\shard\config\local.0, filling with zeroes...
  12. 2017-02-21T09:03:32.623+0800 [FileAllocator] creating directory D:\Data\dbs\shard\config\_tmp
  13. 2017-02-21T09:03:32.720+0800 [FileAllocator] done allocating datafile D:\Data\dbs\shard\config\local.0, size: 64MB,  took 0.095 secs
  14. 2017-02-21T09:03:32.720+0800 [initandlisten] build index on: local.startup_log properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
  15. 2017-02-21T09:03:32.721+0800 [initandlisten]     added index to empty collection
  16. 2017-02-21T09:03:32.721+0800 [initandlisten] command local.$cmd command: create { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 numYields:0  reslen:37 147ms
  17. 2017-02-21T09:03:32.721+0800 [initandlisten] waiting for connections on port 2222

mongos指定config库启动日志：

  1. 2017-02-21T09:09:28.933+0800 warning: running with 1 config server should be done only for testing purposes and is not recommended for production
  2. 2017-02-21T09:09:28.940+0800 [mongosMain] MongoS version 2.6.5 starting: pid=4808 port=3333 64-bit host=PC-linzc (--help for usage)
  3. 2017-02-21T09:09:28.940+0800 [mongosMain] db version v2.6.5
  4. 2017-02-21T09:09:28.940+0800 [mongosMain] git version: e99d4fcb4279c0279796f237aa92fe3b64560bf6
  5. 2017-02-21T09:09:28.940+0800 [mongosMain] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
  6. 2017-02-21T09:09:28.940+0800 [mongosMain] allocator: system
  7. 2017-02-21T09:09:28.940+0800 [mongosMain] options: { net: { port: 3333 }, sharding: { configDB: "morton:2222" } }
  8. 2017-02-21T09:09:28.966+0800 [LockPinger] creating distributed lock ping thread for morton:2222 and process PC-linzc:3333:1487639368:41 (sleeping for 30000ms)
  9. 2017-02-21T09:09:29.120+0800 [LockPinger] cluster morton:2222 pinged successfully at Tue Feb 21 09:09:28 2017 by distributed lock pinger 'morton:2222/PC-linzc:3333:1487639368:41', sleeping for 30000ms
  10. 2017-02-21T09:09:29.121+0800 [mongosMain] distributed lock 'configUpgrade/PC-linzc:3333:1487639368:41' acquired, ts : 58ab9348fb4eb1b0b3428c03
  11. 2017-02-21T09:09:29.124+0800 [mongosMain] starting upgrade of config server from v0 to v5
  12. 2017-02-21T09:09:29.124+0800 [mongosMain] starting next upgrade step from v0 to v5
  13. 2017-02-21T09:09:29.124+0800 [mongosMain] about to log new metadata event: { _id: "PC-linzc-2017-02-21T01:09:29-58ab9349fb4eb1b0b3428c04", server: "PC-linzc", clientAddr: "N/A", time: new Date(1487639369124), what: "starting upgrade of config database", ns: "config.version", details: { from: 0, to: 5 } }
  14. 2017-02-21T09:09:29.126+0800 [mongosMain] creating WriteBackListener for: morton:2222 serverID: 000000000000000000000000
  15. 2017-02-21T09:09:29.289+0800 [mongosMain] writing initial config version at v5
  16. 2017-02-21T09:09:29.303+0800 [mongosMain] about to log new metadata event: { _id: "PC-linzc-2017-02-21T01:09:29-58ab9349fb4eb1b0b3428c06", server: "PC-linzc", clientAddr: "N/A", time: new Date(1487639369303), what: "finished upgrade of config database", ns: "config.version", details: { from: 0, to: 5 } }
  17. 2017-02-21T09:09:29.315+0800 [mongosMain] upgrade of config server to v5 successful
  18. 2017-02-21T09:09:29.317+0800 [mongosMain] distributed lock 'configUpgrade/PC-linzc:3333:1487639368:41' unlocked. 
  19. 2017-02-21T09:09:29.406+0800 [mongosMain] scoped connection to morton:2222 not being returned to the pool
  20. 2017-02-21T09:09:29.407+0800 [Balancer] about to contact config servers and shards
  21. 2017-02-21T09:09:29.407+0800 [Balancer] config servers and shards contacted successfully
  22. 2017-02-21T09:09:29.408+0800 [Balancer] balancer id: PC-linzc:3333 started at Feb 21 09:09:29
  23. 2017-02-21T09:09:29.409+0800 [mongosMain] waiting for connections on port 3333
添加新批注
在作者公开此批注前，只有你和作者可见。
回复批注