017 Ceph的集群管理_3

时间:2022-10-17 21:44:27

一、验证OSD

1.1 osd状态

       运行状态有:up,in,out,down

        正常状态的OSD为up且in

       当OSD故障时,守护进程offline,在5分钟内,集群仍会将其标记为up和in,这是为了防止网络抖动

       如果5分钟内仍未恢复,则会标记为down和out。此时该OSD上的PG开始迁移。这个5分钟的时间间隔可以通过mon_osd_down_out_interval配置项修改

       当故障的OSD重新上线以后,会触发新的数据再平衡

       当集群有noout标志位时,则osd下线不会导致数据重平衡

       OSD每隔6s会互相验证状态。并每隔120s向mon报告一次状态。

容量状态:nearfull,full

1.2 常用指令

[root@ceph2 ~]# ceph osd stat

9 osds: 9 up, 9 in; 32 remapped pgs    # 显示OSD状态

[root@ceph2 ~]# ceph osd df

ID CLASS WEIGHT  REWEIGHT SIZE   USE   AVAIL  %USE VAR  PGS    # 报告osd使用量
 0   hdd 0.01500  1.00000 15348M  112M 15236M 0.74 0.58  48 
 2   hdd 0.01500  1.00000 15348M  112M 15236M 0.73 0.57  40 
 1   hdd 0.01500  1.00000 15348M  114M 15234M 0.75 0.58  39 
 3   hdd 0.01500  1.00000 15348M  269M 15079M 1.76 1.37 256 
 6   hdd 0.01500  1.00000 15348M  208M 15140M 1.36 1.07 248 
 5   hdd 0.01500  1.00000 15348M  228M 15120M 1.49 1.17 238 
 8   hdd 0.01500  1.00000 15348M  245M 15103M 1.60 1.25 266 
 4   hdd 0.01500  1.00000 15348M  253M 15095M 1.65 1.29 245 
 7   hdd 0.01500  1.00000 15348M  218M 15130M 1.42 1.11 228 
 3   hdd 0.01500  1.00000 15348M  269M 15079M 1.76 1.37 256 
 6   hdd 0.01500  1.00000 15348M  208M 15140M 1.36 1.07 248 
 5   hdd 0.01500  1.00000 15348M  228M 15120M 1.49 1.17 238 
 8   hdd 0.01500  1.00000 15348M  245M 15103M 1.60 1.25 266 
 4   hdd 0.01500  1.00000 15348M  253M 15095M 1.65 1.29 245 
 7   hdd 0.01500  1.00000 15348M  218M 15130M 1.42 1.11 228 
                    TOTAL   134G 1765M   133G 1.28          
MIN/MAX VAR: 0.57/1.37  STDDEV: 0.36

 [root@ceph2 ~]# ceph osd find osd.0

{             
    "osd": 0,                      
    "ip": "172.25.250.11:6800/185671",             #查找指定osd位置
    "crush_location": {
        "host": "ceph2-ssd",
        "root": "ssd-root"
    }
}

1.3 osd的心跳参数

        osd_heartbeat_interval # osd之间传递心跳的间隔时间

        osd_heartbeat_grace # 一个osd多久没心跳,就会被集群认为它down了

        mon_osd_min_down_reporters # 确定一个osd状态为down的最少报告来源osd数

        mon_osd_min_down_reports # 一个OSD必须重复报告一个osd状态为down的次数

        mon_osd_down_out_interval # 当osd停止响应多长时间,将其标记为down和out

        mon_osd_report_timeout # monitor宣布失败osd为down前的等待时间

        osd_mon_report_interval_min # 一个新的osd加入集群时,等待多长时间,开始向monitor报告osd_mon_report_interval_max # monitor允许osd报告的最大间隔,超时就认为它down了

        osd_mon_heartbeat_interval # osd向monitor报告心跳的时间

二、管理OSD容量

        当集群容量达到mon_osd_nearfull_ratio的值时,集群会进入HEALTH_WARN状态。这是为了在达到full_ratio之前,提醒添加OSD。默认设置为0.85,即85%

        当集群容量达到mon_osd_full_ratio的值时,集群将停止写入,但允许读取。集群会进入到HEALTH_ERR状态。默认为0.95,即95%。这是为了防止当一个或多个OSD故障时仍留有余地能重平衡数据

2.1 设置

[root@ceph2 ~]# ceph osd set-nearfull-ratio 0.75

osd set-nearfull-ratio 0.75

[root@ceph2 ~]# ceph osd set-full-ratio 0.85

osd set-full-ratio 0.85

[root@ceph2 ~]# ceph osd dump

crush_version 43

full_ratio 0.85

backfillfull_ratio 0.9

nearfull_ratio 0.75

[root@ceph2 ~]# ceph daemon osd.0 config show|grep full_ratio

"mon_osd_backfillfull_ratio": "0.900000",
"mon_osd_full_ratio": "0.950000",
"mon_osd_nearfull_ratio": "0.850000",
"osd_failsafe_full_ratio": "0.970000",
"osd_pool_default_cache_target_full_ratio": "0.800000",

[root@ceph2 ~]# ceph daemon osd.0 config show|grep full_ratio

"mon_osd_backfillfull_ratio": "0.900000",
"mon_osd_full_ratio": "0.950000",
"mon_osd_nearfull_ratio": "0.850000",
"osd_failsafe_full_ratio": "0.970000",
"osd_pool_default_cache_target_full_ratio": "0.800000",

[root@ceph2 ~]# ceph tell osd.* injectargs --mon_osd_full_ratio 0.85

[root@ceph2 ~]# ceph daemon osd.0 config show|grep full_ratio

"mon_osd_backfillfull_ratio": "0.900000",
"mon_osd_full_ratio": "0.850000",
"mon_osd_nearfull_ratio": "0.850000",
"osd_failsafe_full_ratio": "0.970000",
"osd_pool_default_cache_target_full_ratio": "0.800000",

三、集群状态full的问题

3.1 设置集群状态为full

[root@ceph2 ~]# ceph osd set full

full is set

[root@ceph2 ~]# ceph -s

 cluster:
    id:     35a91e48-8244-4e96-a7ee-980ab989d20d
    health: HEALTH_WARN
            full flag(s) set
 
  services:
    mon:        3 daemons, quorum ceph2,ceph3,ceph4
    mgr:        ceph4(active), standbys: ceph2, ceph3
    mds:        cephfs-1/1/1 up  {0=ceph2=up:active}, 1 up:standby
    osd:        9 osds: 9 up, 9 in; 32 remapped pgs
                flags full
    rbd-mirror: 1 daemon active
 
  data:
    pools:   14 pools, 536 pgs
    objects: 220 objects, 240 MB
    usage:   1768 MB used, 133 GB / 134 GB avail
    pgs:     508 active+clean
             28  active+clean+remapped           #pg有问题
 
  io:
    client:   2558 B/s rd, 0 B/s wr, 2 op/s rd, 0 op/s wr

3.2 取消full状态

[root@ceph2 ~]# ceph osd unset full

full is unset

[root@ceph2 ~]# ceph -s

cluster:
id: 35a91e48-8244-4e96-a7ee-980ab989d20d
health: HEALTH_ERR
full ratio(s) out of order
Reduced data availability: 32 pgs inactive, 32 pgs peering, 32 pgs stale
Degraded data redundancy: 32 pgs unclean          #PG也有问题

services:
mon: 3 daemons, quorum ceph2,ceph3,ceph4
mgr: ceph4(active), standbys: ceph2, ceph3
mds: cephfs-1/1/1 up {0=ceph2=up:active}, 1 up:standby
osd: 9 osds: 9 up, 9 in
rbd-mirror: 1 daemon active

data:
pools: 14 pools, 536 pgs
objects: 221 objects, 240 MB
usage: 1780 MB used, 133 GB / 134 GB avail
pgs: 5.970% pgs not active
504 active+clean
32 stale+peering

io:
client: 4911 B/s rd, 0 B/s wr, 5 op/s rd, 0 op/s wr

查看,去的定是一个存储池ssdpool的问题

3.3 删除ssdpool

[root@ceph2 ~]# ceph osd pool delete ssdpool ssdpool --yes-i-really-really-mean-it

[root@ceph2 ~]# ceph -s

  cluster:
    id:     35a91e48-8244-4e96-a7ee-980ab989d20d
    health: HEALTH_ERR
            full ratio(s) out of order
 
  services:
    mon:        3 daemons, quorum ceph2,ceph3,ceph4
    mgr:        ceph4(active), standbys: ceph2, ceph3
    mds:        cephfs-1/1/1 up  {0=ceph2=up:active}, 1 up:standby
    osd:        9 osds: 9 up, 9 in
    rbd-mirror: 1 daemon active
 
  data:
    pools:   13 pools, 504 pgs
    objects: 221 objects, 241 MB
    usage:   1772 MB used, 133 GB / 134 GB avail
    pgs:     504 active+clean
 
  io:
    client:   341 B/s rd, 0 op/s rd, 0 op/s wr

[root@ceph2 ~]# ceph osd unset full

[root@ceph2 ceph]# ceph -s

cluster:
    id:     35a91e48-8244-4e96-a7ee-980ab989d20d
    health: HEALTH_ERR
            full ratio(s) out of order   #依然不起作用
 
  services:
    mon:        3 daemons, quorum ceph2,ceph3,ceph4
    mgr:        ceph4(active), standbys: ceph2, ceph3
    mds:        cephfs-1/1/1 up  {0=ceph2=up:active}, 1 up:standby
    osd:        9 osds: 9 up, 9 in
    rbd-mirror: 1 daemon active
 
  data:
    pools:   13 pools, 504 pgs
    objects: 221 objects, 241 MB
    usage:   1773 MB used, 133 GB / 134 GB avail
    pgs:     504 active+clean
 
  io:
    client:   2046 B/s rd, 0 B/s wr, 2 op/s rd, 0 op/s wr

[root@ceph2 ceph]# ceph health detail

HEALTH_ERR full ratio(s) out of order
OSD_OUT_OF_ORDER_FULL full ratio(s) out of order
full_ratio (0.85) < backfillfull_ratio (0.9), increased    #发现是在前面配置full_ratio导致小于backfillfull_ratio

3.4 重设full_ratio

[root@ceph2 ceph]# ceph osd set-full-ratio 0.95

osd set-full-ratio 0.95

[root@ceph2 ceph]# ceph osd set-nearfull-ratio 0.9

osd set-nearfull-ratio 0.9

[root@ceph2 ceph]# ceph osd dump

epoch 325
fsid 35a91e48-8244-4e96-a7ee-980ab989d20d
created 2019-03-16 12:39:22.552356
modified 2019-03-28 10:54:42.035882
flags sortbitwise,recovery_deletes,purged_snapdirs
crush_version 46
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.9
require_min_compat_client jewel
min_compat_client jewel
require_osd_release luminous
pool 1 'testpool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 190 flags hashpspool stripe_width 0 application rbd
    snap 1 'testpool-snap-20190316' 2019-03-16 22:27:34.150433
    snap 2 'testpool-snap-2' 2019-03-16 22:31:15.430823
pool 5 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 191 flags hashpspool stripe_width 0 application rbd
    removed_snaps [1~13]
pool 6 'rbdmirror' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 192 flags hashpspool stripe_width 0 application rbd
    removed_snaps [1~7]
pool 7 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 176 flags hashpspool stripe_width 0 application rgw
pool 8 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 178 flags hashpspool stripe_width 0 application rgw
pool 9 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 180 flags hashpspool stripe_width 0 application rgw
pool 10 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 182 flags hashpspool stripe_width 0 application rgw
pool 11 'xiantao.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 194 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 12 'xiantao.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 196 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 13 'xiantao.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 198 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 14 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 214 flags hashpspool stripe_width 0 application cephfs
pool 15 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 214 flags hashpspool stripe_width 0 application cephfs
pool 16 'test' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 32 pgp_num 32 last_change 280 flags hashpspool stripe_width 0 application rbd
max_osd 9
osd.0 up   in  weight 1 up_from 314 up_thru 315 down_at 313 last_clean_interval [308,312) 172.25.250.11:6808/1141125 172.25.250.11:6809/1141125 172.25.250.11:6810/1141125 172.25.250.11:6811/1141125 exists,up 745dce53-1c63-4c50-b434-d441038dafe4
osd.1 up   in  weight 1 up_from 315 up_thru 315 down_at 313 last_clean_interval [310,312) 172.25.250.13:6805/592704 172.25.250.13:6806/592704 172.25.250.13:6807/592704 172.25.250.13:6808/592704 exists,up a7562276-6dfd-4803-b248-a7cbdb64ebec
osd.2 up   in  weight 1 up_from 314 up_thru 315 down_at 313 last_clean_interval [308,312) 172.25.250.12:6800/94300 172.25.250.12:6801/94300 172.25.250.12:6802/94300 172.25.250.12:6803/94300 exists,up bbef1a00-3a31-48a0-a065-3a16b9edc3b1
osd.3 up   in  weight 1 up_from 315 up_thru 315 down_at 314 last_clean_interval [308,312) 172.25.250.11:6800/1140952 172.25.250.11:6801/1140952 172.25.250.11:6802/1140952 172.25.250.11:6803/1140952 exists,up e934a4fb-7125-4e85-895c-f66cc5534ceb
osd.4 up   in  weight 1 up_from 315 up_thru 315 down_at 313 last_clean_interval [310,312) 172.25.250.13:6809/592702 172.25.250.13:6810/592702 172.25.250.13:6811/592702 172.25.250.13:6812/592702 exists,up e2c33bb3-02d2-4cce-85e8-25c419351673
osd.5 up   in  weight 1 up_from 314 up_thru 315 down_at 313 last_clean_interval [308,312) 172.25.250.12:6804/94301 172.25.250.12:6805/94301 172.25.250.12:6806/94301 172.25.250.12:6807/94301 exists,up d299e33c-0c24-4cd9-a37a-a6fcd420a529
osd.6 up   in  weight 1 up_from 315 up_thru 315 down_at 314 last_clean_interval [308,312) 172.25.250.11:6804/1140955 172.25.250.11:6805/1140955 172.25.250.11:6806/1140955 172.25.250.11:6807/1140955 exists,up debe7f4e-656b-48e2-a0b2-bdd8613afcc4
osd.7 up   in  weight 1 up_from 314 up_thru 315 down_at 313 last_clean_interval [309,312) 172.25.250.13:6801/592699 172.25.250.13:6802/592699 172.25.250.13:6803/592699 172.25.250.13:6804/592699 exists,up 8c403679-7530-48d0-812b-72050ad43aae
osd.8 up   in  weight 1 up_from 315 up_thru 315 down_at 313 last_clean_interval [310,312) 172.25.250.12:6808/94302 172.25.250.12:6810/94302 172.25.250.12:6811/94302 172.25.250.12:6812/94302 exists,up bb73edf8-ca97-40c3-a727-d5fde1a9d1d9

3.5 再次尝试

[root@ceph2 ceph]# ceph osd unset full

full is unset

[root@ceph2 ceph]# ceph -s

 cluster:
    id:     35a91e48-8244-4e96-a7ee-980ab989d20d
    health: HEALTH_OK     #成功
 
  services:
    mon:        3 daemons, quorum ceph2,ceph3,ceph4
    mgr:        ceph4(active), standbys: ceph2, ceph3
    mds:        cephfs-1/1/1 up  {0=ceph2=up:active}, 1 up:standby
    osd:        9 osds: 9 up, 9 in
    rbd-mirror: 1 daemon active
 
  data:
    pools:   13 pools, 504 pgs
    objects: 221 objects, 241 MB
    usage:   1773 MB used, 133 GB / 134 GB avail
    pgs:     504 active+clean
 
  io:
    client:   0 B/s wr, 0 op/s rd, 0 op/s wr

四、手动控制PG的Primary OSD

        可以通过手动修改osd的权重以提升 特定OSD被选为PG Primary OSD的概率,避免将速度慢的磁盘用作primary osd

4.1 查看osd.4为主的pg

[root@ceph2 ceph]# ceph pg dump|grep 'active+clean'|egrep "\[4,"

dumped all                          #查看OSD.4位主的PG
1.7e          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.280490        0'0    311:517 [4,0,8]          4 [4,0,8]              4        0'0 2019-03-27 13:28:30.900982             0'0 2019-03-24 06:16:20.594466 
1.7b          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.256673        0'0    311:523 [4,6,5]          4 [4,6,5]              4        0'0 2019-03-28 02:46:27.659275             0'0 2019-03-23 09:10:34.438462 
15.77         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.282033        0'0    311:162 [4,5,0]          4 [4,5,0]              4        0'0 2019-03-28 04:25:28.324399             0'0 2019-03-26 17:10:19.390530 
1.77          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:34:03.733420        0'0    312:528 [4,0,5]          4 [4,0,5]              4        0'0 2019-03-28 08:34:03.733386             0'0 2019-03-27 08:26:21.579623 
15.7a         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.257051        0'0    311:158 [4,2,3]          4 [4,2,3]              4        0'0 2019-03-28 03:27:22.186467             0'0 2019-03-26 17:10:19.390530 
15.7c         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.273391        0'0    311:144 [4,0,8]          4 [4,0,8]              4        0'0 2019-03-27 17:59:38.124535             0'0 2019-03-26 17:10:19.390530 
1.72          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.276870        0'0    311:528 [4,8,0]          4 [4,8,0]              4        0'0 2019-03-28 06:36:06.125767             0'0 2019-03-24 13:59:12.569691 
15.7f         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.258669        0'0    311:149 [4,8,0]          4 [4,8,0]              4        0'0 2019-03-27 21:48:22.082918             0'0 2019-03-27 21:48:22.082918 
15.69         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.258736        0'0    311:150 [4,0,8]          4 [4,0,8]              4        0'0 2019-03-28 00:07:06.805003             0'0 2019-03-28 00:07:06.805003 
1.67          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.275098        0'0    311:517 [4,0,8]          4 [4,0,8]              4        0'0 2019-03-27 21:08:41.166673             0'0 2019-03-24 06:16:29.598240 
14.22         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.257257        0'0    311:149 [4,5,6]          4 [4,5,6]              4        0'0 2019-03-27 20:32:16.816439             0'0 2019-03-26 17:09:56.246887 
14.29         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.252788        0'0    311:151 [4,5,6]          4 [4,5,6]              4        0'0 2019-03-27 21:55:42.189434             0'0 2019-03-26 17:09:56.246887 
5.21          2                  0        0         0       0  4210688  139      139  active+clean 2019-03-28 08:02:25.257694    189'139    311:730 [4,6,2]          4 [4,6,2]              4    189'139 2019-03-27 19:02:33.483252         189'139 2019-03-25 08:42:13.970938 
14.2a         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.256911        0'0    311:150 [4,6,5]          4 [4,6,5]              4        0'0 2019-03-27 18:09:45.512728             0'0 2019-03-26 17:09:56.246887 
14.2b         0                  0        0         0       0        0    1        1  active+clean 2019-03-28 08:02:25.258316      214'1    311:162 [4,6,2]          4 [4,6,2]              4      214'1 2019-03-27 23:48:05.092971             0'0 2019-03-26 17:09:56.246887 
14.2d         1                  0        0         0       0       46    1        1  active+clean 2019-03-28 08:02:25.282383      214'1    311:171 [4,3,2]          4 [4,3,2]              4      214'1 2019-03-28 03:14:08.690676             0'0 2019-03-26 17:09:56.246887 
15.2c         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.258195        0'0    311:157 [4,3,5]          4 [4,3,5]              4        0'0 2019-03-28 02:03:17.819746             0'0 2019-03-28 02:03:17.819746 
6.1a          1                  0        0         0       0       19    2        2  active+clean 2019-03-28 08:02:25.281807      161'2    311:267 [4,2,6]          4 [4,2,6]              4      161'2 2019-03-27 22:42:45.639905           161'2 2019-03-26 12:51:51.614941 
5.18          4                  0        0         0       0    49168   98       98  active+clean 2019-03-28 08:02:25.258482     172'98    311:621 [4,8,3]          4 [4,8,3]              4     172'98 2019-03-27 21:27:03.723920          172'98 2019-03-27 21:27:03.723920 
15.14         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.252656        0'0    311:148 [4,6,5]          4 [4,6,5]              4        0'0 2019-03-27 19:56:18.466744             0'0 2019-03-26 17:10:19.390530 
15.17         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.256549        0'0    311:164 [4,5,0]          4 [4,5,0]              4        0'0 2019-03-27 23:58:46.490357             0'0 2019-03-26 17:10:19.390530 
1.18          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.277674        0'0    311:507 [4,6,8]          4 [4,6,8]              4        0'0 2019-03-28 01:14:47.944309             0'0 2019-03-26 18:31:14.774358 
5.1c          2                  0        0         0       0       16  250      250  active+clean 2019-03-28 08:02:25.257857    183'250  311:19066 [4,2,6]          4 [4,2,6]              4    183'250 2019-03-28 05:42:09.856046         183'250 2019-03-25 23:36:49.652800 
15.19         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.257506        0'0    311:164 [4,2,3]          4 [4,2,3]              4        0'0 2019-03-28 00:39:31.020637             0'0 2019-03-26 17:10:19.390530 
16.7          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.282212        0'0     311:40 [4,3,2]          4 [4,3,2]              4        0'0 2019-03-28 01:11:12.974900             0'0 2019-03-26 21:40:00.073686 
6.e           0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.258109        0'0    311:251 [4,6,2]          4 [4,6,2]              4        0'0 2019-03-27 06:36:11.963158             0'0 2019-03-27 06:36:11.963158 
13.5          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.257437        0'0    311:168 [4,0,2]          4 [4,0,2]              4        0'0 2019-03-27 19:52:21.320611             0'0 2019-03-26 13:31:34.012304 
16.19         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.257560        0'0     311:42 [4,2,6]          4 [4,2,6]              4        0'0 2019-03-28 04:21:53.015903             0'0 2019-03-26 21:40:00.073686 
7.1           3                  0        0         0       0     1813   14       14  active+clean 2019-03-28 08:02:25.257994     192'14    311:303 [4,2,3]          4 [4,2,3]              4     192'14 2019-03-27 12:08:04.858102          192'14 2019-03-27 12:08:04.858102 
14.9          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.252723        0'0    311:163 [4,3,5]          4 [4,3,5]              4        0'0 2019-03-28 04:45:30.060857             0'0 2019-03-28 04:45:30.060857 
5.1           3                  0        0         0       0  8404992  119      119  active+clean 2019-03-28 08:02:25.258586    189'119    311:635 [4,3,8]          4 [4,3,8]              4    189'119 2019-03-28 01:01:39.725401         189'119 2019-03-25 09:40:24.623173 
13.6          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.257198        0'0    311:157 [4,5,0]          4 [4,5,0]              4        0'0 2019-03-27 15:49:19.196870             0'0 2019-03-26 13:31:34.012304 
5.f           5                  0        0         0       0    86016  128      128  active+clean 2019-03-28 08:02:25.258053    183'128   311:1179 [4,2,3]          4 [4,2,3]              4    183'128 2019-03-27 12:15:30.134353         183'128 2019-03-22 12:21:02.832942 
16.1d         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.257306        0'0     311:42 [4,0,2]          4 [4,0,2]              4        0'0 2019-03-28 01:15:37.043172             0'0 2019-03-26 21:40:00.073686 
12.0          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.258535        0'0    311:140 [4,6,8]          4 [4,6,8]              4        0'0 2019-03-27 15:42:11.927266             0'0 2019-03-26 13:31:31.916623 
16.1f         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.258248        0'0     311:41 [4,0,5]          4 [4,0,5]              4        0'0 2019-03-28 08:01:48.349363             0'0 2019-03-28 08:01:48.349363 
9.6           0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.257612        0'0    311:211 [4,2,3]          4 [4,2,3]              4        0'0 2019-03-27 23:02:31.386965             0'0 2019-03-27 23:02:31.386965 
1.f           0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.279868        0'0    311:503 [4,3,8]          4 [4,3,8]              4        0'0 2019-03-28 07:41:02.022670             0'0 2019-03-24 07:50:30.260358 
1.10          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.257936        0'0    311:538 [4,2,0]          4 [4,2,0]              4        0'0 2019-03-28 01:43:31.429879             0'0 2019-03-23 06:36:38.178339 
1.12          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.256725        0'0    311:527 [4,3,5]          4 [4,3,5]              4        0'0 2019-03-28 04:49:49.213043             0'0 2019-03-25 17:35:25.833155 
16.2          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.278599        0'0     311:31 [4,6,8]          4 [4,6,8]              4        0'0 2019-03-28 07:32:10.065419             0'0 2019-03-26 21:40:00.073686 
15.1d         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.252838        0'0    311:155 [4,5,3]          4 [4,5,3]              4        0'0 2019-03-28 00:50:04.416619             0'0 2019-03-26 17:10:19.390530 
5.2a          0                  0        0         0       0        0  107      107  active+clean 2019-03-28 08:02:25.281096    172'107    311:621 [4,6,8]          4 [4,6,8]              4    172'107 2019-03-27 23:39:40.781443         172'107 2019-03-25 17:35:38.835798 
5.2b          7                  0        0         0       0 16826368  225      225  active+clean 2019-03-28 08:02:25.257363    189'225   311:2419 [4,0,5]          4 [4,0,5]              4    189'225 2019-03-27 10:24:42.972494         189'225 2019-03-25 04:13:33.567532 
1.31          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.256401        0'0    311:514 [4,5,6]          4 [4,5,6]              4        0'0 2019-03-27 20:39:23.076113             0'0 2019-03-25 10:06:22.224727 
5.31          1                  0        0         0       0  4194304  113      113  active+clean 2019-03-28 08:02:25.282326    189'113    311:661 [4,2,3]          4 [4,2,3]              4    189'113 2019-03-27 23:35:50.633871         189'113 2019-03-25 10:27:03.837772 
14.37         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.282270        0'0    311:153 [4,5,0]          4 [4,5,0]              4        0'0 2019-03-27 20:36:25.969312             0'0 2019-03-26 17:09:56.246887 
15.34         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.258369        0'0    311:132 [4,8,3]          4 [4,8,3]              4        0'0 2019-03-27 23:30:49.442053             0'0 2019-03-26 17:10:19.390530 
1.43          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.279242        0'0    311:501 [4,6,8]          4 [4,6,8]              4        0'0 2019-03-27 21:59:51.254952             0'0 2019-03-26 13:16:37.312462 
1.48          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.281910        0'0    311:534 [4,0,5]          4 [4,0,5]              4        0'0 2019-03-27 23:47:00.053793             0'0 2019-03-24 04:51:10.218424 
15.45         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.258421        0'0    311:155 [4,0,8]          4 [4,0,8]              4        0'0 2019-03-28 01:39:15.366349             0'0 2019-03-26 17:10:19.390530 
1.4e          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.252906        0'0    311:519 [4,5,3]          4 [4,5,3]              4        0'0 2019-03-27 20:50:17.495390             0'0 2019-03-21 01:02:41.709506 
1.51          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.281974        0'0    311:530 [4,6,2]          4 [4,6,2]              4        0'0 2019-03-28 07:23:04.730515             0'0 2019-03-26 00:23:54.419333 
15.5a         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.257140        0'0    311:158 [4,0,2]          4 [4,0,2]              4        0'0 2019-03-28 00:12:17.000955             0'0 2019-03-26 17:10:19.390530 
1.56          0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.256961        0'0    311:521 [4,5,3]          4 [4,5,3]              4        0'0 2019-03-27 16:24:10.512235             0'0 2019-03-27 16:24:10.512235 
15.50         0                  0        0         0       0        0    0        0  active+clean 2019-03-28 08:02:25.252599        0'0    311:154 [4,5,3]          4 [4,5,3]              4        0'0 2019-03-28 00:25:01.475477             0'0 2019-03-26 17:10:19.390530 

统计

[root@ceph2 ceph]# ceph pg dump|grep 'active+clean'|egrep "\[4,"|wc -l

dumped all
56

4.2 权重设为0

[root@ceph2 ceph]# ceph osd primary-affinity osd.4 0

Error EPERM: you must enable 'mon osd allow primary affinity = true' on the mons before you can adjust primary-affinity.  note that older clients will no longer be able to communicate with the cluster.

[root@ceph2 ceph]# ceph daemon /var/run/ceph/ceph-mon.$(hostname -s).asok config show|grep primary

   "mon_osd_allow_primary_affinity": "false",
    "mon_osd_allow_primary_temp": "false",

4.3 修改配置文件

[root@ceph1 ~]# vim  /etc/ceph/ceph.conf

[global]
fsid = 35a91e48-8244-4e96-a7ee-980ab989d20d
mon initial members = ceph2,ceph3,ceph4
mon host = 172.25.250.11,172.25.250.12,172.25.250.13
public network = 172.25.250.0/24
cluster network = 172.25.250.0/24
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
[osd]
osd mkfs type = xfs
osd mkfs options xfs = -f -i size=2048
osd mount options xfs = noatime,largeio,inode64,swalloc
osd journal size = 5120

[mon]
mon_allow_pool_delete = true
mon_osd_allow_primary_affinity = true

[root@ceph1 ~]# ansible all -m copy -a 'src=/etc/ceph/ceph.conf dest=/etc/ceph/ceph.conf owner=ceph group=ceph mode=0644'

[root@ceph1 ~]# ansible mons -m shell -a ' systemctl restart ceph-mon.target'

[root@ceph1 ~]# ansible mons -m shell -a ' systemctl restart ceph-osd.target'

[root@ceph2 ceph]# ceph daemon /var/run/ceph/ceph-mon.$(hostname -s).asok config show|grep primary

017 Ceph的集群管理_3

没有生效

4.5 使用命令行修改

[root@ceph2 ceph]# ceph tell mon.\* injectargs '--mon_osd_allow_primary_affinity=true'

mon.ceph2: injectargs:mon_osd_allow_primary_affinity = 'true' (not observed, change may require restart) 
mon.ceph3: injectargs:mon_osd_allow_primary_affinity = 'true' (not observed, change may require restart) 
mon.ceph4: injectargs:mon_osd_allow_primary_affinity = 'true' (not observed, change may require restart) 

[root@ceph2 ceph]# ceph daemon /var/run/ceph/ceph-mon.$(hostname -s).asok config show|grep primary

017 Ceph的集群管理_3

[root@ceph2 ceph]# ceph pg dump|grep 'active+clean'|egrep "\[4,"|wc -l

dumped all
56

4.6 修改权重

[root@ceph2 ceph]#  ceph pg dump|grep 'active+clean'|egrep "\[4,"|wc -l
dumped all
56
[root@ceph2 ceph]# ceph osd primary-affinity osd.4 0
set osd.4 primary-affinity to 0 (802)
[root@ceph2 ceph]#  ceph pg dump|grep 'active+clean'|egrep "\[4,"|wc -l
dumped all
56
[root@ceph2 ceph]#  ceph pg dump|grep 'active+clean'|egrep "\[4,"|wc -l
dumped all
56
[root@ceph2 ceph]#  ceph pg dump|grep 'active+clean'|egrep "\[4,"|wc -l
dumped all
0
[root@ceph2 ceph]# ceph osd primary-affinity osd.4 0.5
set osd.4 primary-affinity to 0.5 (8327682)
[root@ceph2 ceph]#  ceph pg dump|grep 'active+clean'|egrep "\[4,"|wc -l
dumped all
26
[root@ceph2 ceph]#  ceph pg dump|grep 'active+clean'|egrep "\[4,"|wc -l
dumped all
26
[root@ceph2 ceph]#  ceph pg dump|grep 'active+clean'|egrep "\[4,"|wc -l
dumped all
26

博主声明:本文的内容来源主要来自誉天教育晏威老师,由本人实验完成操作验证,需要的博友请联系誉天教育(http://www.yutianedu.com/),获得官方同意或者晏老师(https://www.cnblogs.com/breezey/)本人同意即可转载,谢谢!