Ceph+nfs部署及使用

部署环境

五台装有CentOS 7.3的主机，不建议使用最新版的，源经常出现超时，个人推荐使用7.3,ip分别为192.168.1.56,192.168.1.182、192.168.1.183、192.168.1.184、192.168.1.185

升级内核

使用uname -r 查看内核版本，官方建议使用最新内核版本

[[email protected] ~]# uname -r

3.10.0-514.el7.x86_64

建立本地软件仓库

[[email protected] ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

安装最新内核repo包

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

yum安装最新的内核包

[[email protected] ~]# yum --enablerepo=elrepo-kernel install kernel-ml -y

修改启动配置文件，启动项设置0

[[email protected] ~]#sed -i 's/GRUB_DEFAULT=saved/GRUB_DEFAULT=0/g' /etc/default/grub

重新加载内核配置文件

[[email protected] ~]grub2-mkconfig -o /boot/grub2/grub.cfg

重启后再次输入uname -r

[[email protected] ~]# uname -r

4.13.12-1.el7.elrepo.x86_64

1、每台主机有三个磁盘(虚拟机磁盘要大于100G),详细信息如下：

[[email protected] ~]# cat /etc/redhat-release

CentOS Linux release 7.3.1611 (Core)

[[email protected] ~]# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sdd 8:48 0 100G 0 disk

sdb 8:16 0 100G 0 disk

sr0 11:0 1 4.1G 0 rom

fd0 2:0 1 4K 0 disk

sdc 8:32 0 100G 0 disk

sda 8:0 0 20G 0 disk

└─sda1 8:1 0 20G 0 part /

[[email protected] ~]# cat /etc/hosts

192.168.1.56 admin-node

192.168.1.182 node1

192.168.1.183 node2

192.168.1.184 node3

192.168.1.185 node4

2、集群配置如下：

主机	IP	环境
Node-admin	192.168.1.56	1 * ceph-depoly 1 * mon 3 osd
Node1	192.168.1.182	1 * mon 3 osd
Node2	192.168.1.183	1 * mon 3 osd
Node3	192.168.1.184	1 * mon 3 osd nfs服务器
Node4	192.168.1.185	1 * mon 3 osd

3、环境清理

如果之前部署失败了，不必删除ceph客户端，或者重新搭建虚拟机，只需要在每个节点上执行如下指令即可将环境清理至刚安装完ceph客户端时的状态！强烈建议在旧集群上搭建之前清理干净环境，否则会发生各种异常情况。

ps aux|grep ceph |awk '{print $2}'|xargs kill -9

ps -ef|grep ceph

#确保此时所有ceph进程都已经关闭！！！如果没有关闭，多执行几次。

umount /var/lib/ceph/osd/*

rm -rf /var/lib/ceph/osd/*

rm -rf /var/lib/ceph/mon/*

rm -rf /var/lib/ceph/mds/*

rm -rf /var/lib/ceph/bootstrap-mds/*

rm -rf /var/lib/ceph/bootstrap-osd/*

rm -rf /var/lib/ceph/bootstrap-rgw/*

rm -rf /var/lib/ceph/tmp/*

rm -rf /etc/ceph/*

rm -rf /var/run/ceph/*

或者使用命令卸载ceph服务和清除配置

ceph-deploy purge 节点名

4、推荐使用阿里云源

需要在每个主机上执行以下指令

yum clean all

rm -rf /etc/yum.repos.d/*.repo

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

sed -i '/aliyuncs/d' /etc/yum.repos.d/CentOS-Base.repo

sed -i '/aliyuncs/d' /etc/yum.repos.d/epel.repo

sed -i 's/$releasever/7/g' /etc/yum.repos.d/CentOS-Base.repo

5、官方源需要收费，增加ceph源，使用163ceph源

vim /etc/yum.repos.d/ceph.repo

添加以下内容：

[ceph]

name=ceph

baseurl=http://mirrors.163.com/ceph/rpm-jewel/el7/x86_64/

gpgcheck=0

[ceph-noarch]

name=cephnoarch

baseurl=http://mirrors.163.com/ceph/rpm-jewel/el7/noarch/

gpgcheck=0

6、安装ceph客户端：

yum makecache

yum install ceph ceph-radosgw rdate -y

关闭selinux&firewalld

sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

setenforce 0

systemctl stop firewalld

systemctl disable firewalld

7、同步各个节点时间：

yum -y install rdate

rdate -s time-a.nist.gov

echo rdate -s time-a.nist.gov >> /etc/rc.d/rc.local

chmod +x /etc/rc.d/rc.local

8、开始部署

在部署节点(admin-node)安装ceph-deploy，下文的部署节点统一指admin-node:

[[email protected] ~]# yum -y install ceph-deploy

[[email protected] ~]# ceph-deploy --version

1.5.34

[[email protected] ~]# ceph -v

ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)

9、免密码登陆部署

输入ssh-******命令，一直按回车

[[email protected] ~]# ssh-******

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

b4:d4:a9:fa:9e:47:43:8c:2e:72:6d:c7:cd:56:6e:c9 [email protected]

The key's randomart image is:

+--[ RSA 2048]----+

| |

| . . |

| ooo |

| o.oo . |

| oSo o + . |

| . o.+ = + E |

| o.o o o . |

| . .. |

| .+. |

+-----------------+

[[email protected] ~]# ssh-copy-id node1

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

[email protected]'s password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'node1'"

and check to make sure that only the key(s) you wanted were added.

测试

[[email protected] ~]# ssh node1

Last login: Mon Nov 13 00:48:11 2017 from 192.168.1.188

[[email protected] ~]#

出现node1 则为成功

其他的节点也是以上操作

10、在部署节点创建部署目录并开始部署：

[[email protected] ~]# mkdir cluster

[[email protected] ~]# cd cluster/

[[email protected] cluster]# ceph-deploy new admin-node node1 node2

此时，目录内容如下：

[[email protected] cluster]# ls

ceph.conf ceph-deploy-ceph.log ceph.mon.keyring

11、根据自己的IP配置向ceph.conf中添加public_network，并稍微增大mon之间时差允许范围(默认为0.05s，现改为2s)：

[[email protected] cluster]# echo public_network=192.168.1.0/24 >> ceph.conf

[[email protected] cluster]# echo mon_clock_drift_allowed = 2 >> ceph.conf

[[email protected] cluster]# cat ceph.conf

12、开始部署monitor:

[[email protected] cluster]# ceph-deploy mon create-initial

[global]

fsid = d40f3021-0b2b-4c88-9958-76e276f1de39

mon_initial_members = admin-node, node1, node2, node3, node4

mon_host = 192.168.1.56,192.168.1.182,192.168.1.183,192.168.1.184,192.168.1.185

auth_cluster_required = cephx

auth_service_required = cephx

auth_client_required = cephx

public_network=192.168.1.0/24

mon_clock_drift_allowed=2

[[email protected] cluster]# ls

ceph.bootstrap-mds.keyring ceph.bootstrap-rgw.keyring ceph.conf ceph.mon.keyring

ceph.bootstrap-osd.keyring ceph.client.admin.keyring ceph-deploy-ceph.log

13、查看集群状态：

[[email protected] cluster]# ceph -s

cluster d40f3021-0b2b-4c88-9958-76e276f1de39

health HEALTH_ERR

clock skew detected on mon.node2

no osds

Monitor clock skew detected

monmap e2: 5 mons at {admin-node=192.168.1.56:6789/0,node1=192.168.1.182:6789/0,node2=192.168.1.183:6789/0,node3=192.168.1.184:6789/0,node4=192.168.1.185:6789/0}

election epoch 8, quorum 0,1,2,3,4 admin-node,node1,node2,node3,node4

osdmap e1: 0 osds: 0 up, 0 in

flags sortbitwise,require_jewel_osds

pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects

0 kB used, 0 kB / 0 kB avail

64 creating

14、开始部署OSD:

调试admin-node node1 node2 node3 node4 新增的磁盘

ceph-deploy --overwrite-conf osd prepare admin-node:/dev/sdb admin-node:/dev/sdc admin-node:/dev/sdd node1:/dev/sdb node1:/dev/sdc node1:/dev/sdd node2:/dev/sdb node2:/dev/sdc node2:/dev/sdd node3:/dev/sdb node3:/dev/sdc node3:/dev/sdd node4:/dev/sdb node4:/dev/sdc node4:/dev/sdd --zap-disk

**新增的磁盘

ceph-deploy --overwrite-conf osd activate admin-node:/dev/sdb1 admin-node:/dev/sdc1 admin-node:/dev/sdd1 node1:/dev/sdb1 node1:/dev/sdc1 node1:/dev/sdd1 node2:/dev/sdb1 node2:/dev/sdc1 node2:/dev/sdd1 node3:/dev/sdb1 node3:/dev/sdc1 node3:/dev/sdd1 node4:/dev/sdb1 node4:/dev/sdc1 node4:/dev/sdd1

15、我在部署的时候出了个小问题，有一个OSD没成功(待所有OSD部署完毕后，再重新部署问题OSD即可解决)，如果不出意外的话，集群状态应该如下：

[[email protected] cluster]# ceph -s

cluster d40f3021-0b2b-4c88-9958-76e276f1de39

health HEALTH_ERR

clock skew detected on mon.node2

1 pgs are stuck inactive for more than 300 seconds

3 pgs peering

1 pgs stuck inactive

1 pgs stuck unclean

too few PGs per OSD (12 < min 30)

Monitor clock skew detected

monmap e2: 5 mons at {admin-node=192.168.1.56:6789/0,node1=192.168.1.182:6789/0,node2=192.168.1.183:6789/0,node3=192.168.1.184:6789/0,node4=192.168.1.185:6789/0}

election epoch 8, quorum 0,1,2,3,4 admin-node,node1,node2,node3,node4

osdmap e85: 15 osds: 15 up, 15 in

flags sortbitwise,require_jewel_osds

pgmap v249: 64 pgs, 1 pools, 0 bytes data, 0 objects

418 MB used, 1139 GB / 1139 GB avail

61 active+clean

3 remapped+peering

16、去除这个WARN，只需要增加rbd池的PG就好：

[[email protected] cluster]# ceph osd pool set rbd pg_num 128

set pool 0 pg_num to 128

[[email protected] cluster]# ceph osd pool set rbd pgp_num 128

set pool 0 pgp_num to 128

[[email protected] cluster]# ceph -s

cluster 0248817a-b758-4d6b-a217-11248b098e10

health HEALTH_ERR

19 pgs are stuck inactive for more than 300 seconds

12 pgs peering

19 pgs stuck inactive

monmap e1: 3 mons at {admin-node=192.168.57.222:6789/0,node1=192.168.57.223:6789/0,node2=192.168.57.224:6789/0}

election epoch 22, quorum 0,1,2 admin-node,node1,node2

osdmap e49: 9 osds: 9 up, 9 in

flags sortbitwise

pgmap v96: 128 pgs, 1 pools, 0 bytes data, 0 objects

308 MB used, 18377 GB / 18378 GB avail

103 active+clean

12 peering

9 creating

4 activating

[[email protected] cluster]# ceph -s

cluster d40f3021-0b2b-4c88-9958-76e276f1de39

health HEALTH_OK

monmap e2: 5 mons at {admin-node=192.168.1.56:6789/0,node1=192.168.1.182:6789/0,node2=192.168.1.183:6789/0,node3=192.168.1.184:6789/0,node4=192.168.1.185:6789/0}

election epoch 8, quorum 0,1,2,3,4 admin-node,node1,node2,node3,node4

osdmap e101: 15 osds: 12 up, 12 in

flags sortbitwise,require_jewel_osds

pgmap v318: 256 pgs, 1 pools, 0 bytes data, 0 objects

436 MB used, 1139 GB / 1139 GB avail

256 active+clean至此，集群部署完毕。

17、config推送

请不要使用直接修改某个节点的/etc/ceph/ceph.conf文件的方式，而是去部署节点(此处为admin-node:/root/cluster/ceph.conf)目录下修改。因为节点到几十个的时候，不可能一个个去修改的，采用推送的方式快捷安全！

修改完毕后，执行如下指令，将conf文件推送至各个节点：

[[email protected] cluster]# ceph-deploy --overwrite-conf config push admin-node node1 node2

此时，需要重启各个节点的monitor服务，见下一节。

18、mon&osd启动方式

#monitor start/stop/restart

#admin-node为各个monitor所在节点的主机名。

systemctl start [email protected]

systemctl restart [email protected]

systemctl stop [email protected]

#OSD start/stop/restart

#0为该节点的OSD的id，可以通过`ceph osd tree`查看

systemctl start/stop/restart [email protected]

[[email protected] cluster]# ceph osd tree

ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY

-1 1.39046 root default

-2 0.27809 host admin-node

0 0.09270 osd.0 up 1.00000 1.00000

1 0.09270 osd.1 up 1.00000 1.00000

2 0.09270 osd.2 up 1.00000 1.00000

-3 0.27809 host node1

3 0.09270 osd.3 up 1.00000 1.00000

4 0.09270 osd.4 up 1.00000 1.00000

5 0.09270 osd.5 up 1.00000 1.00000

-4 0.27809 host node2

6 0.09270 osd.6 down 0 1.00000

7 0.09270 osd.7 down 0 1.00000

8 0.09270 osd.8 down 0 1.00000

-5 0.27809 host node3

9 0.09270 osd.9 up 1.00000 1.00000

10 0.09270 osd.10 up 1.00000 1.00000

11 0.09270 osd.11 up 1.00000 1.00000

-6 0.27809 host node4

12 0.09270 osd.12 up 1.00000 1.00000

13 0.09270 osd.13 up 1.00000 1.00000

14 0.09270 osd.14 up 1.00000 1.00000

二、部署ceph 块设备

创建新的主机

创建主机名为node-client的虚拟机,在admin-node主机中输入ceph-deploy install node-client

[[email protected] cluster]# ceph-deploy install node-client

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf

[ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy install node-client

……..

将node-client主机名和IP添加至集群/etc/hosts文件中，保证各个主机之间能够正常通讯

添加client端到集群内

[[email protected] cluster]# ceph-deploy admin node-client

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf

[ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy admin node-client

直到出现write cluster configuration to /etc/ceph/{cluster}.conf则配置成功

在node-client端配置块设备

[[email protected] ~]# rbd create foo --size 4096 -m node1 -k /etc/ceph/ceph.client.admin.keyring

没有任何返回表示创建成功

[[email protected] ~]# rbd map foo --name client.admin -m node1 -k /etc/ceph/ceph.client.admin.keyring

/dev/rbd0

格式化和挂载

[[email protected] ~]# mkfs.ext4 -m0 /dev/rbd0

mke2fs 1.42.9 (28-Dec-2013)

Discarding device blocks: done

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=1024 blocks, Stripe width=1024 blocks

262144 inodes, 1048576 blocks

[[email protected] ~]# mkdir /mnt/ceph-block-device

[[email protected] ~]# mount /dev/rbd0 /mnt/ceph-block-device

[[email protected] ~]# cd /mnt/ceph-block-device

[[email protected] ceph-block-device]# chmod 777 /mnt/ceph-block-device

测试

传输313M的文件到/mnt/ceph-block-device文件夹中，查看本地挂载和ceph集群使用情况

Ceph+nfs部署及使用

部署CephFS

创建MDS元数据服务器

[[email protected] cluster]# ceph-deploy mds create node2 node3

创建存储池

[[email protected] cluster]# ceph osd pool create cephfs_data 8

pool 'cephfs_data' created

[[email protected] cluster]# ceph osd pool create cephfs_metadata 10

pool 'cephfs_metadata' created

使用fs new命令enable文件系统

[[email protected] cluster]# ceph fs new YYcephfs cephfs_metadata cephfs_data

new fs with metadata pool 2 and data pool 1

使用mds来查看状态

[[email protected] cluster]# ceph mds stat

e6: 1/1/1 up {0=node3=up:active}, 1 up:standby

查看系统内核是否支持rbd

[[email protected] ~]# mod

modinfo modprobe modutil

[[email protected] ~]# modprobe rbd

[[email protected] ~]# lsmod |grep rbd

rbd 86016 0

libceph 315392 2 ceph,rbd

挂载

创建/mnt/mycephfs目录

[[email protected] ~]# mkdir /mnt/mycephfs

[[email protected] ~]# mount -t ceph 192.168.1.56:6789:/ /mnt/mycephfs

mount error 22 = Invalid argument

出现mount error 22报错，需要在后面加上挂载IP的认证

[[email protected] cluster]# cat ceph.client.admin.keyring

[client.admin]

key = AQBEUQlaw5IsAhAAONP2y999yTcbhVdPZoSOUw==

在node3节点上重新挂载

[[email protected] ~]# mount -t ceph 192.168.1.56:6789:/ /mnt/mycephfs -o name=admin,secret=AQBEUQlaw5IsAhAAONP2y999yTcbhVdPZoSOUw==

Ceph+nfs部署及使用

测试

Ceph+nfs部署及使用

传入313mb文件后

Ceph+nfs部署及使用

三、部署nfs

1、安装nfs服务器

[[email protected] mycephfs]# yum install nfs-utils -y

2、将/mnt/mycephfs策略写入/etc/exports

[[email protected] mycephfs]#echo “/mnt/mycephfs *(rw,async,no_root_squash,no_subtree_check)” >> /etc/exports

3、执行exportfs -a生效

[[email protected] mycephfs]#exportfs -a

4、nfs、rpcbind服务设置自启

[[email protected] mycephfs]#service rpcbind restart

[[email protected] mycephfs]#service nfs restart

[[email protected] mycephfs]#chkconfig nfs on

[[email protected] mycephfs]#chkconfig rpcbind on

秒客网

Ceph+nfs部署及使用

相关文章