[置顶] MongoDB 分布式操作——分片操作

MongoDB 分布式操作——分片操作

描述：

像其它分布式数据库一样，MongoDB同样支持分布式操作，且MongoDB将分布式已经集成到数据库中，其分布式体系如下图所示：

所谓的片，其实就是一个单独的MongoDB进程，它可以运行在不同的节点上，也可以运行在同一节点但监听不同的端口。同一个集合的数据可能在每个片中都存储的有数据，但它们在逻辑上整合成一个整体。分片的依据往往是根据表中的某一列或者是集合中的某一组属性。且分片是系统自动的，但是分片的依据需要用户自己指定，分片的依据称之为“片键”。常见的片键选择为“随机片键”或者是“递增片键”。如果使用递增片键，数据的分布则会按照一定的顺序来写入数据，即当一个节点写满才会写入到下一个片键。而如果使用随机片键，数据则可能在任何一个片中都有数据。

对于分布式中片的管理（片与片之间的通信），MongoDB中通过mongos进程来进行管理，客户端对数据的操作，都由mongos进程来处理。对于节点、分片的管理，MongoDB由配置服务器来管理，配置服务器其实也是一个单独MongoDB进程。

那么，一般在什么时候使用分片呢？在单个节点的磁盘不足时会分片，另外，当单个mongod不能满足写数据的性能要求时，可以使用分片；当单台服务器内存太小，将大量的数据放到内存中速度会很慢，为了提高性能，我们可以使用分片操作。

以下是对于MongoDB分片操作的实验：

操作环境：

Win8 64位操作系统，虚拟机CentOS 32位操作系统。MongoDB 2.4.5

实验步骤：

（1）启动配置服务器（放置一些元数据）：

[root@h3 dbs]# mongod -dbpath /var/lib/mongo/dbs/config --nojournal --port 20000

Mon Aug 12 13:36:25.999 [initandlisten] MongoDB starting : pid=22951 port=20000   dbpath=/var/lib/mongo/dbs/config 32-bit host=h3

Mon Aug 12 13:36:26.001 [initandlisten]

Mon Aug 12 13:36:26.002 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary  .

Mon Aug 12 13:36:26.003 [initandlisten] **       32 bit builds are limited to le  ss than 2GB of data (or less with --journal).

Mon Aug 12 13:36:26.004 [initandlisten] **       Note that journaling defaults t  o off for 32 bit and is currently off.

Mon Aug 12 13:36:26.005 [initandlisten] **       See http://dochub.mongodb.org/c  ore/32bit

Mon Aug 12 13:36:26.005 [initandlisten]

Mon Aug 12 13:36:26.006 [initandlisten] db version v2.4.5

Mon Aug 12 13:36:26.007 [initandlisten] git version: a2ddc68ba7c9cee17bfe69ed840  383ec3506602b

Mon Aug 12 13:36:26.007 [initandlisten] build info: Linux bs-linux32.10gen.cc 2.  6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49

Mon Aug 12 13:36:26.008 [initandlisten] allocator: system

Mon Aug 12 13:36:26.009 [initandlisten] options: { dbpath: "/var/lib/mongo/dbs/c  onfig", nojournal: true, port: 20000 }

Mon Aug 12 13:36:26.071 [FileAllocator] allocating new datafile /var/lib/mongo/d  bs/config/local.ns, filling with zeroes...

Mon Aug 12 13:36:26.072 [FileAllocator] creating directory /var/lib/mongo/dbs/co  nfig/_tmp

Mon Aug 12 13:36:26.582 [FileAllocator] done allocating datafile /var/lib/mongo/  dbs/config/local.ns, size: 16MB,  took 0.504 secs

Mon Aug 12 13:36:26.585 [FileAllocator] allocating new datafile /var/lib/mongo/d  bs/config/local.0, filling with zeroes...

Mon Aug 12 13:36:27.725 [FileAllocator] done allocating datafile /var/lib/mongo/  dbs/config/local.0, size: 16MB,  took 1.138 secs

Mon Aug 12 13:36:27.757 [initandlisten] command local.$cmd command: { create: "s  tartup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:37   1714ms

Mon Aug 12 13:36:27.764 [websvr] admin web console waiting for connections on po  rt 21000

Mon Aug 12 13:36:27.771 [initandlisten] waiting for connections on port 20000

（2）启动mongos：

[root@h3 ~]# mongos --port 30000  --configdb localhost:20000

Mon Aug 12 13:40:13.042 warning: running with 1 config server should be done only for testing purposes and is not recommended for production

Mon Aug 12 13:40:13.075 [mongosMain] MongoS version 2.4.5 starting: pid=22968 port=30000 32-bit host=h3 (--help for usage)

Mon Aug 12 13:40:13.076 [mongosMain] git version: a2ddc68ba7c9cee17bfe69ed840383ec3506602b

Mon Aug 12 13:40:13.076 [mongosMain] build info: Linux bs-linux32.10gen.cc 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49

Mon Aug 12 13:40:13.076 [mongosMain] options: { configdb: "localhost:20000", port: 30000 }

Mon Aug 12 13:40:13.250 [LockPinger] creating distributed lock ping thread for localhost:20000 and process h3:30000:1376286013:1804289383 (sleeping for 30000ms)

Mon Aug 12 13:40:17.105 [mongosMain] distributed lock 'configUpgrade/h3:30000:1376286013:1804289383' acquired, ts : 52087541ff5991dd77979f2c

Mon Aug 12 13:40:17.126 [mongosMain] starting upgrade of config server from v0 to v4

Mon Aug 12 13:40:17.128 [mongosMain] starting next upgrade step from v0 to v4

Mon Aug 12 13:40:17.130 [mongosMain] about to log new metadata event: { _id: "h3-2013-08-12T05:40:17-52087541ff5991dd77979f2d", server: "h3", clientAddr: "N/A", time: new Date(1376286017130), what: "starting upgrade of config database", ns: "config.version", details: { from: 0, to: 4 } }

Mon Aug 12 13:40:17.138 [mongosMain] writing initial config version at v4

Mon Aug 12 13:40:17.232 [mongosMain] about to log new metadata event: { _id: "h3-2013-08-12T05:40:17-52087541ff5991dd77979f2f", server: "h3", clientAddr: "N/A", time: new Date(1376286017232), what: "finished upgrade of config database", ns: "config.version", details: { from: 0, to: 4 } }

Mon Aug 12 13:40:17.237 [mongosMain] upgrade of config server to v4 successful

Mon Aug 12 13:40:17.244 [mongosMain] distributed lock 'configUpgrade/h3:30000:1376286013:1804289383' unlocked.

Mon Aug 12 13:40:17.474 [websvr] admin web console waiting for connections on port 31000

Mon Aug 12 13:40:17.476 [Balancer] about to contact config servers and shards

Mon Aug 12 13:40:17.492 [Balancer] config servers and shards contacted successfully

Mon Aug 12 13:40:17.494 [Balancer] balancer id: h3:30000 started at Aug 12 13:40:17

Mon Aug 12 13:40:17.527 [mongosMain] waiting for connections on port 30000

（3）添加分片1：

[root@h3 ~]# cd /var/lib/mongo/dbs/

[root@h3 dbs]# mkdir shard1

[root@h3 dbs]# mongod --dbpath /var/lib/mongo/dbs/shard1  --port 10000 --nojournal

Mon Aug 12 13:43:10.560 [initandlisten] MongoDB starting : pid=23016 port=10000

dbpath=/var/lib/mongo/dbs/shard1 32-bit host=h3

Mon Aug 12 13:43:10.562 [initandlisten]

Mon Aug 12 13:43:10.562 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.

Mon Aug 12 13:43:10.563 [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).

Mon Aug 12 13:43:10.564 [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.

Mon Aug 12 13:43:10.565 [initandlisten] **       See http://dochub.mongodb.org/core/32bit

Mon Aug 12 13:43:10.566 [initandlisten]

Mon Aug 12 13:43:10.566 [initandlisten] db version v2.4.5

Mon Aug 12 13:43:10.567 [initandlisten] git version: a2ddc68ba7c9cee17bfe69ed840383ec3506602b

Mon Aug 12 13:43:10.568 [initandlisten] build info: Linux bs-linux32.10gen.cc 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49

Mon Aug 12 13:43:10.568 [initandlisten] allocator: system

Mon Aug 12 13:43:10.569 [initandlisten] options: { dbpath: "/var/lib/mongo/dbs/shard1", nojournal: true, port: 10000 }

Mon Aug 12 13:43:10.626 [FileAllocator] allocating new datafile /var/lib/mongo/dbs/shard1/local.ns, filling with zeroes...

Mon Aug 12 13:43:10.628 [FileAllocator] creating directory /var/lib/mongo/dbs/shard1/_tmp

Mon Aug 12 13:43:11.547 [FileAllocator] done allocating datafile /var/lib/mongo/dbs/shard1/local.ns, size: 16MB,  took 0.907 secs

Mon Aug 12 13:43:11.549 [FileAllocator] allocating new datafile /var/lib/mongo/dbs/shard1/local.0, filling with zeroes...

Mon Aug 12 13:43:13.339 [FileAllocator] done allocating datafile /var/lib/mongo/dbs/shard1/local.0, size: 16MB,  took 1.787 secs

Mon Aug 12 13:43:13.427 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:372801ms

Mon Aug 12 13:43:13.472 [initandlisten] waiting for connections on port 10000

Mon Aug 12 13:43:13.497 [websvr] admin web console waiting for connections on port 11000

（4）添加分片2：

[root@h3 dbs]# cd /var/lib/mongo/dbs/

[root@h3 dbs]# mkdir shard2

[root@h3 dbs]# mongod --dbpath /var/lib/mongo/dbs/shard2 --port 10001 --nojournal

Mon Aug 12 13:48:25.549 [initandlisten] MongoDB starting : pid=23129 port=10001 dbpath=/var/lib/mongo/dbs/shard2 32-bit host=h3

Mon Aug 12 13:48:25.550 [initandlisten]

Mon Aug 12 13:48:25.550 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.

Mon Aug 12 13:48:25.550 [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).

Mon Aug 12 13:48:25.550 [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.

Mon Aug 12 13:48:25.550 [initandlisten] **       See http://dochub.mongodb.org/core/32bit

Mon Aug 12 13:48:25.551 [initandlisten]

Mon Aug 12 13:48:25.551 [initandlisten] db version v2.4.5

Mon Aug 12 13:48:25.552 [initandlisten] git version: a2ddc68ba7c9cee17bfe69ed840383ec3506602b

Mon Aug 12 13:48:25.552 [initandlisten] build info: Linux bs-linux32.10gen.cc 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49

Mon Aug 12 13:48:25.552 [initandlisten] allocator: system

Mon Aug 12 13:48:25.552 [initandlisten] options: { dbpath: "/var/lib/mongo/dbs/shard2", nojournal: true, port: 10001 }

Mon Aug 12 13:48:25.570 [FileAllocator] allocating new datafile /var/lib/mongo/dbs/shard2/local.ns, filling with zeroes...

Mon Aug 12 13:48:25.572 [FileAllocator] creating directory /var/lib/mongo/dbs/shard2/_tmp

Mon Aug 12 13:48:26.956 [FileAllocator] done allocating datafile /var/lib/mongo/dbs/shard2/local.ns, size: 16MB,  took 1.378 secs

Mon Aug 12 13:48:26.957 [FileAllocator] allocating new datafile /var/lib/mongo/dbs/shard2/local.0, filling with zeroes...

Mon Aug 12 13:48:28.023 [FileAllocator] done allocating datafile /var/lib/mongo/dbs/shard2/local.0, size: 16MB,  took 1.065 secs

Mon Aug 12 13:48:28.038 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:37 2469ms

Mon Aug 12 13:48:28.041 [websvr] admin web console waiting for connections on port 11001

Mon Aug 12 13:48:28.042 [initandlisten] waiting for connections on port 10001

（5）添加分片3：

[root@h3 ~]# cd /var/lib/mongo/dbs/

[root@h3 dbs]# mkdir shard3

[root@h3 dbs]# mongod --dbpath /var/lib/mongo/dbs/shard3 --port 10002 --nojournal

Mon Aug 12 13:50:43.735 [initandlisten] MongoDB starting : pid=23142 port=10002 dbpath=/var/lib/mongo/dbs/shard3 32-bit host=h3

Mon Aug 12 13:50:43.737 [initandlisten]

Mon Aug 12 13:50:43.737 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.

Mon Aug 12 13:50:43.738 [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).

Mon Aug 12 13:50:43.739 [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.

Mon Aug 12 13:50:43.739 [initandlisten] **       See http://dochub.mongodb.org/core/32bit

Mon Aug 12 13:50:43.740 [initandlisten]

Mon Aug 12 13:50:43.740 [initandlisten] db version v2.4.5

Mon Aug 12 13:50:43.740 [initandlisten] git version: a2ddc68ba7c9cee17bfe69ed840383ec3506602b

Mon Aug 12 13:50:43.740 [initandlisten] build info: Linux bs-linux32.10gen.cc 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49

Mon Aug 12 13:50:43.740 [initandlisten] allocator: system

Mon Aug 12 13:50:43.741 [initandlisten] options: { dbpath: "/var/lib/mongo/dbs/shard3", nojournal: true, port: 10002 }

Mon Aug 12 13:50:43.756 [FileAllocator] allocating new datafile /var/lib/mongo/dbs/shard3/local.ns, filling with zeroes...

Mon Aug 12 13:50:43.756 [FileAllocator] creating directory /var/lib/mongo/dbs/shard3/_tmp

Mon Aug 12 13:50:45.079 [FileAllocator] done allocating datafile /var/lib/mongo/dbs/shard3/local.ns, size: 16MB,  took 1.32 secs

Mon Aug 12 13:50:45.099 [FileAllocator] allocating new datafile /var/lib/mongo/dbs/shard3/local.0, filling with zeroes...

Mon Aug 12 13:50:45.948 [FileAllocator] done allocating datafile /var/lib/mongo/dbs/shard3/local.0, size: 16MB,  took 0.848 secs

Mon Aug 12 13:50:45.957 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:37 2202ms

Mon Aug 12 13:50:45.960 [initandlisten] waiting for connections on port 10002

Mon Aug 12 13:50:45.961 [websvr] admin web console waiting for connections on port 11002

（6）添加刚刚启动的三个片：

[root@h3 dbs]# mongo localhost:30000/admin

MongoDB shell version: 2.4.5

connecting to: localhost:30000/admin

mongos> db.runCommand({addshard : "localhost:10000", allowLocal : true});

{ "shardAdded" : "shard0000", "ok" : 1 }

mongos> db.runCommand({addshard : "localhost:10001", allowLocal : true});

{ "shardAdded" : "shard0001", "ok" : 1 }

mongos> db.runCommand({addshard : "localhost:10002", allowLocal : true});

{ "shardAdded" : "shard0002", "ok" : 1 }

mongos>

（7）打开数据库foo的分片功能：

 mongos> db.runCommand({"enablesharding" : "foo"});

{ "ok" : 1 }

mongos>

（8）打开数据库foo中数据集合bar的分片功能：

mongos> db.runCommand({"shardcollection" : "foo.bar", "key" : {"_id" : 1}});

{ "collectionsharded" : "foo.bar", "ok" : 1 }

mongos>

（9）查看所有的分片：

mongos> use config

switched to db config

mongos> db.shards.find();

{ "_id" : "shard0000", "host" : "localhost:10000" }

{ "_id" : "shard0001", "host" : "localhost:10001" }

{ "_id" : "shard0002", "host" : "localhost:10002" }

mongos>

（10）查看databases集合：

mongos> db.databases.find();

{ "_id" : "admin", "partitioned" : false, "primary" : "config" }

{ "_id" : "foo", "partitioned" : true, "primary" : "shard0000" }

mongos>

（11）查看chucks集合：

mongos> db.chunks.find();

{ "_id" : "foo.bar-_id_MinKey", "lastmod" : Timestamp(1, 0), "lastmodEpoch" : ObjectId("52087d0a1072e2912d4242ea"), "ns" : "foo.bar", "min" : { "_id" : { "$minKey" : 1 } }, "max" : { "_id" : { "$maxKey" : 1 } }, "shard" : "shard0000" }

mongos>

（12）获得概要信息：

mongos> db.printShardingStatus();

--- Sharding Status ---

  sharding version: {

        "_id" : 1,

        "version" : 3,

        "minCompatibleVersion" : 3,

        "currentVersion" : 4,

        "clusterId" : ObjectId("52087bf41072e2912d4242b8")

}

  shards:

        {  "_id" : "shard0000",  "host" : "localhost:10000" }

        {  "_id" : "shard0001",  "host" : "localhost:10001" }

        {  "_id" : "shard0002",  "host" : "localhost:10002" }

  databases:

        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

        {  "_id" : "foo",  "partitioned" : true,  "primary" : "shard0000" }

                foo.bar

                        shard key: { "_id" : 1 }

                        chunks:

                                shard0000       1

                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)

mongos>

（13）移除分片：

mongos> use admin

switched to db admin

mongos> db.runCommand({"removeshard" : "localhost:10000"});

{

        "msg" : "draining started successfully",

        "state" : "started",

        "shard" : "shard0000",

        "note" : "you need to drop or movePrimary these databases",

        "dbsToMove" : [

                "foo"

        ],

        "ok" : 1

}

mongos>

总结：对MongoDB的分片操作执行如下顺序：

（1）启动配置服务器；

（2）启动mongos进程；

（3）添加mongo实例（片）；

（4）对数据库或者数据集使用分片操作。

[置顶] MongoDB 分布式操作——分片操作的更多相关文章

[置顶] SPL讲解(4)--Criteria操作篇
概念以前一篇文章中,描述了实体Entity的操作,很明显,仅仅实体的操作是远远不够的.如:我们经常会根据查询条件从数据库中获取记录集并绑定到DataGrid上,会根据条件进行批量的Update和De ...
[置顶] c&num;对于文件的操作
在system.iO:命名空间中提供了两种遍历枚举文件夹的类,Directory 和DirectoryInfo, 他们都有一个方法GetDirectories():来便利枚举文件夹, /// < ...
[置顶] Android中使用sqlite3操作SQLite
SQLite库包含一个名字叫做sqlite3的命令行,它可以让用户手工输入并执行面向SQLite数据库的SQL命令.本文档提供一个样使用sqlite3的简要说明. 一.创建数据库: 1.将sqlit ...
vue 通过自定义指令实现置顶操作；
项目需求:要求当前项目每个页面滑到超出一屏的距离时,出现 backTop 按钮,点击则回到最顶端:俗称置顶操作: 因为涉及到的页面较多,每个页面都加肯定显得重复累赘,最终想到了 Vue 的自定义指令 ...
jQuery操作table数据上移、下移和置顶
jQuery 操作table中的tr换行的步骤如下: 1.获取当前tr var $tr = $(this).parents("tr"); 2.移动tr //上移 $tr.prev( ...
Mongodb 分片操作介绍
为什么需要分片操作?由于数据量太大,使得CPU,内存,磁盘I/O等压力过大.当MongoDB存储海量的数据时,一台机器可能不足以存储数据,也可能不足以提供可接受的读写吞吐量.这时,我们就可以通过在多台 ...
006PHP文件处理—— 目录操作删除目录删除置顶类型文件
<?php /** * 目录操作删除目录删除置顶类型文件 */ //echo rmdir('61') or die('目录删除失败'); //删除一个目录中有其他文件的内容的方法: //第1 ...
表格中上移下移置顶的js操作
<script> $(function(){ //上移 var $up = $(".up") $up.click(function() { var $tr = ...
Qt 窗口操作函数（置顶、全屏，最大化最小化按钮设置等）
一.窗口置顶与取消置顶 void MainWindow::on_windowTopButton_clicked() { if (m_flags == NULL) { m_flags = windo ...

随机推荐

Android自定义控件（二）
这一篇主要来讲一下自定义控件中的自定义viewgroup,我们以项目中最常用的下拉刷新和加载更多组件为例简单介绍一下自定义viewgroup时应该怎么做. 分析:下拉刷新和加载更多的原理和步骤自定 ...
Oracle 查看某表被哪些表外键引用
分2步骤: 1. select * from user_constraints e where e.table_name='xxx'; 2. select a.constraint_type,a.ta ...
苹果开发者账号注册&amp&semi;真机调试
苹果公司开发者账号注册流程详解 http://www.cnblogs.com/xilinch/p/4037164.html 真机调试教程 http://segmentfault.com/a/11900 ...
maven（一）初步搭建，项目结构
一.环境准备 java环境, jdk 1.5 以上 MyEclipse集成工具,我装的是8.5 版本二.快速安装及配置 1.下载maven: http://maven.apache.org/docs ...
看完final的感受
今天没课,(其实是有体育课的,去打了一会球就跑路了...)就在宿舍看world final ; 我去,老毛子真是好厉害,看的我目瞪口呆,哈喇子直流; 上交的大神好厉害,本来还以为上交要夺冠的,最后罚时 ...
2016 Multi-University Training Contest 5&amp&semi;6 总结
第五场和第六场多校都打得很糟糕. 能做到不以物喜不以己悲是假的,这对队伍的情绪也可以算上是比较大的打击. 很多时候我们发现了问题,但是依旧没有采取有效的方法去解决它,甚至也没有尝试去改变.这是一件相当 ...
Idea构建Maven项目教程
Idea构建Maven项目,以及其中遇到的坑,及解决方案步骤一:配置Idea的maven插件步骤二:创建maven项目 file-->new-->project 下一步,设置group ...
odoo开发笔记 -- 日常开发注意点小节
onchange depends区别视图字段增加readonly属性
开启&commat;EnableRedisHttpSession
sessionId=569662ce-c6d5-42a9-a94b-c9df826df716 1800秒后失效 sessionId=23913542-9b5f-4699-8a87-1023b57f5f ...
css兼容性记录
* , ie6,ie7可以识别: _和- , ie6可以识别: !important ,表示高优先级,ie7及以上,firefox都支持,ie6认识带!important的样式属性, ...