一、环境和准备条件
两台虚拟机：Ubuntu 14.04.41LTS，ip分别是192.168.110.151（hostname是master）和192.168.110.152（hostname是dockertest4）
Ceph采用当前Ubuntu 14.04源中最新的Ceph LTS版本：JEWEL10.2.3。
Kubernetes版本为上次安装时的1.4版本。
二、Ceph安装原理
Ceph分布式存储集群由若干组件组成，包括：Ceph Monitor、Ceph OSD和Ceph MDS，其中如果你仅使用对象存储和块
存储时，MDS不是必须的（本次我们也不需要安装MDS），仅当你要用到Cephfs时，MDS才是需要安装的。
Ceph的安装模型与k8s有些类似，也是通过一个deploy node远程操作其他Node以create、prepare和activate各个Node上
的Ceph组件，官方手册中给出的示意图如下：
三、在151机器上安装ceph-deploy
1.配置apt-get源
#wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
OK
# echo deb https://download.ceph.com/debian-jewel/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
2.更新apt-get
apt-get update
3.安装ceph-deploy
apt-get install ceph-deploy
注意：ceph-deploy只需要在admin/deploy node上安装即可。
四、151和152两台机器配置无密码登录
1. 将这一账号命名为cephd，我们需要在每个ceph node上(包括admin node/deploy node)都建立一个cephd用户，并加
入到sudo组中
以下命令分别在151和152两台虚拟机上执行：
useradd -d /home/cephd -m cephd
passwd cephd
添加sudo权限：
echo "cephd ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephd
sudo chmod 0440 /etc/sudoers.d/cephd
2.在admin node(deploy node)上，登入cephd账号，创建该账号下deploy node到其他各个Node的ssh免密登录设
置，密码留空：
在deploy node上执行：
$ ssh-keygen
3.秘钥拷贝
将deploy node的公钥copy到自己和152上去：
$ ssh-copy-id cephd@master
$ ssh-copy-id cephd@dockertest4
4.最后，在Deploy node上创建并编辑~/.ssh/config，这是Ceph官方doc推荐的步骤，这样做的目的是可以避免每次
执行ceph-deploy时都要去指定 –username {username}参数。
//~/.ssh/config
Host master
Hostname master
User cephd
Host dockertest4
Hostname dockertest4
User cephd
五、如果之前安装过ceph，可以先执行如下命令以获得一个干净的环境：（在151上执行如下命令）
ceph-deploy purge master dockertest4
ceph-deploy forgetkeys
ceph-deploy purgedata master dockertest4
六、创建工作目录
在151上，建立cephinstall目录，然后进入cephinstall目录执行相关步骤。
七、安装ceph步骤
1.首先来创建一个ceph cluster，这个环节需要通过执行ceph-deploy new {initial-monitor-node(s)}命令。按照上面的
安装设计，我们的ceph monitor node就是master，因此我们执行下面命令来创建一个名为ceph的ceph cluster：执行
如下命令
sudo ceph-deploy new master
2.new命令执行完后，ceph-deploy会在当前目录下创建一些辅助文件：
# ls
ceph.conf ceph-deploy-ceph.log ceph.mon.keyring
$ cat ceph.conf
[global]
fsid = f5166c78-e3b6-4fef-b9e7-1ecf7382fd93
mon_initial_members = master
mon_host = 192.168.110.151
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
3.由于我们仅有两个OSD节点，因此我们在进一步安装之前，需要先对ceph.conf文件做一些配置调整：修改配置以
进行后续安装：
在[global]标签下，添加下面一行：
osd pool default size = 2
4.ceph.conf保存退出。接下来，我们执行下面命令在node1和node2上安装ceph运行所需的各个binary包：
#sudo ceph-deploy install master dockertest4
5.初始化ceph monitor nod
有了ceph启动的各个程序后，我们首先来初始化ceph cluster的monitor node。在deploy node的工作目录cephinstall下，
执行：
# sudo ceph-deploy mon create-initial
6.在master(monitor node)上，我们看到ceph-mon已经运行起来了：
cephd@master:~/cephinstall$ ps -ef | grep ceph
root 5814 4834 0 05:19 pts/14 00:00:00 su - cephd
cephd 5815 5814 0 05:19 pts/14 00:00:09 -su
ceph 41046 1 0 16:58 ? 00:00:03 /usr/bin/ceph-mon --cluster=ceph -i master -f --setuser ceph --setgroup ceph
ceph 41123 1 0 16:58 ? 00:00:04 /usr/bin/ceph-osd --cluster=ceph -i 0 -f --setuser ceph --setgroup ceph
cephd 41396 5815 0 17:30 pts/14 00:00:00 ps -ef
cephd 41397 5815 0 17:30 pts/14 00:00:00 grep --color=auto ceph
注意：这一点很重要，如果ceph进程不存在，那么在后续的prepare和activate步骤中，也会提示No data was
received after 300 seconds, disconnecting...，虽然是个警告错误，但是也是的ceph安装失败。
7.prepare ceph OSD node
至此，ceph-mon组件程序已经成功启动了，剩下的只有OSD这一关了。启动OSD node分为两步：prepare和
activate。OSD node是真正存储数据的节点，我们需要为ceph-osd提供独立存储空间，一般是一个独立的disk。但我们
环境不具备这个条件，于是在本地盘上创建了个目录，提供给OSD。
在deploy node上执行：
ssh master
sudo mkdir /var/local/osd0
exit
ssh dockertest4
sudo mkdir /var/local/osd1
exit
8.接下来，我们就可以执行prepare操作了，prepare操作会在上述的两个osd0和osd1目录下创建一些后续activate激
活以及osd运行时所需要的文件：
ceph-deploy osd prepare master:/var/local/osd0 dockertest4:/var/local/osd1
9.激活ceph OSD node
ceph-deploy osd activate master:/var/local/osd0 dockertest4:/var/local/osd1
注意：一般情况下着这一部会报错，错误信息类似如下：
[master][WARNIN] 2016-12-16 14:25:40.325075 7fd1aa73f800 -1 ** ERROR: error creating empty object store in
/var/local/osd0: (13) Permission denied
出现该问题的原因是：
osd0被root拥有，以ceph用户启动的ceph-osd程序自然没有权限在/var/local/osd0目录下创建文件并写入数据了。这个
问题在ceph官方issue中有很多人提出来，也给出了临时修正方法：
将osd0和osd1的权限赋予ceph:ceph：
master：
sudo chown -R ceph:ceph /var/local/osd0
dockertest4：
sudo chown -R ceph:ceph /var/local/osd1
10.重新激活ceph OSD node
ceph-deploy osd activate master:/var/local/osd0 dockertest4:/var/local/osd1
11.接下来，查看一下ceph集群中的OSD节点状态：
$ ceph osd tree
cephd@master:~/cephinstall$ ceph osd tree
2016-12-16 16:55:43.674267 7f326989d700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
2016-12-16 16:55:43.674669 7f326989d700 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2016-12-16 16:55:43.674922 7f326989d700 0 librados: client.admin initialization error (2) No such file or directory
很明显，这是存在问题的。本实验采用临时解决方案，跳过认证，方法是：分别修改151和152/etc/ceph/ceph.conf文
件，修改后的文件内容如下：
[global]
osd_pool_default_size = 2
osd max object name len = 256
osd max object namespace len = 64
fsid = 48c8e252-2a4a-4af7-ba53-f93f744d3c6e
mon_initial_members = master
mon_host = 192.168.110.151
auth_cluster_required = none
auth_service_required = none
auth_client_required = none
注意：（1）cephx改成了none
（2）并在原文件基础之上增加了：
osd max object name len = 256
osd max object namespace len = 64
增加这两行的原因是：当前虚拟机的文件系统类型是ext4，详细请参考官网说明，http://docs.ceph.com/docs/jewel/rados/configuration/filesystem-recommendations/
12.接下来，查看一下ceph集群中的OSD节点状态：
cephd@master:~/cephinstall$ ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.05557 root default
-2 0.02779 host master
0 0.02779 osd.0 up 1.00000 1.00000
-3 0.02779 host dockertest4
1 0.02779 osd.1 up 1.00000 1.00000
cephd@master:~/cephinstall$ ceph -s
cluster 48c8e252-2a4a-4af7-ba53-f93f744d3c6e
health HEALTH_OK
monmap e1: 1 mons at {master=192.168.110.151:6789/0}
election epoch 4, quorum 0 master
osdmap e10: 2 osds: 2 up, 2 in
flags sortbitwise,require_jewel_osds
pgmap v871: 64 pgs, 1 pools, 0 bytes data, 0 objects
28434 MB used, 26765 MB / 58202 MB avail
64 active+clean
13.目前epch已经正常启动
八、ceph与k8s集成测试
1.测试案例1（参考kubernetes\examples\volumes\rbd下的例子）
（1）rbd-with-secret.json对应本实验如下
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"name": "rbd2"
},
"spec": {
"containers": [
{
"name": "rbd-rw",
"image": "kubernetes/pause",
"volumeMounts": [
{
"mountPath": "/mnt/rbd",
"name": "rbdpd"
}
]
}
],
"volumes": [
{
"name": "rbdpd",
"rbd": {
"monitors": [
"192.168.110.151:6789"
],
"pool": "rbd",
"image": "foo",
"user": "admin",
"secretRef": {
"name": "ceph-secret"
},
"fsType": "ext4",
"readOnly": true
}
}
]
}
}
name：volume名字，这个没什么可说的，顾名思义即可。
rbd.monitors：前面提到过ceph集群的monitor组件，这里填写monitor组件的通信信息，集群里有几个monitor就填几
个，本实验中只有一个151；
rbd.pool：Ceph中的pool记号，它用来给ceph中存储的对象进行逻辑分区用的。默认的pool是”rbd”；
rbd.image：Ceph磁盘块设备映像文件；
rbd.user：ceph client访问ceph storage cluster所使用的用户名。ceph有自己的一套user管理系统，user的写法通常
是TYPE.ID，比如client.admin（是不是想到对应的文件：ceph.client.admin.keyring）。client是一种type，而admin则
是user。一般来说，Type基本都是client。
secret.Ref：引用的k8s secret对象名称。
（2）创建image
# rbd create foo -s 1024
（3）查看image
# rbd list
foo
在rbd pool中(在上述命令中未指定pool name，默认image建立在rbd pool中)创建一个大小为1024Mi的ceph image
foo，rbd list命令的输出告诉我们foo image创建成功。
（4）将foo image映射到内核
root@master:~# rbd map foo
注意：初次执行映射操作的时候会出现如下错误：
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail" or so.
rbd: map failed: (6) No such device or address
错误原因：Ubuntu 14.04.1内核仅支持其中的layering feature
解决办法：a）# rbd feature disable foo exclusive-lock, object-map, fast-diff, deep-flatten
b）在/etc/ceph/ceph.conf配置文件中加：rbd_default_features = 1 #仅是layering对应的bit码所对应的整数值，然后重启ceph
（5）将foo image映射到内核
# rbd map foo
/dev/rbd1
（6）格式化image foo
# mkfs.ext4 /dev/rbd0
（7）创建ceph-secret这个k8s secret对象
# ceph auth get-key client.admin
AQBiKBxYuPXiJRAAsupnTBsURoWzb0k00oM3iQ==
# echo "AQBiKBxYuPXiJRAAsupnTBsURoWzb0k00oM3iQ=="|base64
QVFBZnExTll2NkRUQWhBQWZQQWNteXMzTjlQZHhDei9SOHBQamc9PQo=
（8）创建ceph-secret.yaml
//ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret
data:
key: QVFBZnExTll2NkRUQWhBQWZQQWNteXMzTjlQZHhDei9SOHBQamc9PQo=
（9）创建pod（根据创建ceph-secret.yaml文件）
# kubectl create -f rbd-with-secret.json
pod "rbd2" created
（10）Inspect 容器
root@dockertest5:/home/docker/xu/k8s/server/bin# docker inspect 3dfa4a410643
[
{
"Id": "3dfa4a410643721a595f8d0bebe6c84af11ca87f957f387b11802c991bdbbc20",
"Created": "2016-12-17T01:19:43.126748592Z",
"Path": "/pause",
"Args": [],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 4875,
"ExitCode": 0,
"Error": "",
"StartedAt": "2016-12-17T01:19:43.478492336Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:f9d5de0795395db6c50cb1ac82ebed1bd8eb3eefcebb1aa724e01239594e937b",
"ResolvConfPath": "/var/lib/docker/containers/2e8914eda03d3e1f2a1cc73a505b446cc31632d2eac5a96c8fd3c6db60808a14/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/2e8914eda03d3e1f2a1cc73a505b446cc31632d2eac5a96c8fd3c6db60808a14/hostname",
"HostsPath": "/var/lib/kubelet/pods/ceb15fbf-c3f6-11e6-a39e-000c29ec0bcd/etc-hosts",
"LogPath": "/var/lib/docker/containers/3dfa4a410643721a595f8d0bebe6c84af11ca87f957f387b11802c991bdbbc20/3dfa4a410643721a595f8d0bebe6c84af11ca87f957f387b11802c991bdbbc20-json.log",
"Name": "/k8s_rbd-rw.4ab8fe40_rbd2_default_ceb15fbf-c3f6-11e6-a39e-000c29ec0bcd_4826a0bb",
"RestartCount": 0,
"Driver": "aufs",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/var/lib/kubelet/pods/ceb15fbf-c3f6-11e6-a39e-000c29ec0bcd/volumes/kubernetes.io~rbd/rbdpd:/mnt/rbd",
"/var/lib/kubelet/pods/ceb15fbf-c3f6-11e6-a39e-000c29ec0bcd/etc-hosts:/etc/hosts",
"/var/lib/kubelet/pods/ceb15fbf-c3f6-11e6-a39e-000c29ec0bcd/containers/rbd-rw/4826a0bb:/dev/termination-log"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "container:2e8914eda03d3e1f2a1cc73a505b446cc31632d2eac5a96c8fd3c6db60808a14",
"PortBindings": null,
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "container:2e8914eda03d3e1f2a1cc73a505b446cc31632d2eac5a96c8fd3c6db60808a14",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 1000,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined"
],
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 2,
"Memory": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": -1,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Name": "aufs",
"Data": null
},
"Mounts": [
{
"Source": "/var/lib/kubelet/pods/ceb15fbf-c3f6-11e6-a39e-000c29ec0bcd/volumes/kubernetes.io~rbd/rbdpd",
"Destination": "/mnt/rbd",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Source": "/var/lib/kubelet/pods/ceb15fbf-c3f6-11e6-a39e-000c29ec0bcd/etc-hosts",
"Destination": "/etc/hosts",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Source": "/var/lib/kubelet/pods/ceb15fbf-c3f6-11e6-a39e-000c29ec0bcd/containers/rbd-rw/4826a0bb",
"Destination": "/dev/termination-log",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "rbd2",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"KUBERNETES_SERVICE_HOST=192.168.110.1",
"KUBERNETES_SERVICE_PORT=443",
"KUBERNETES_SERVICE_PORT_HTTPS=443",
"KUBERNETES_PORT=tcp://192.168.110.1:443",
"KUBERNETES_PORT_443_TCP=tcp://192.168.110.1:443",
"KUBERNETES_PORT_443_TCP_PROTO=tcp",
"KUBERNETES_PORT_443_TCP_PORT=443",
"KUBERNETES_PORT_443_TCP_ADDR=192.168.110.1",
"HOME=/",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "kubernetes/pause",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/pause"
],
"OnBuild": null,
"Labels": {
"io.kubernetes.container.hash": "4ab8fe40",
"io.kubernetes.container.name": "rbd-rw",
"io.kubernetes.container.restartCount": "0",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.pod.name": "rbd2",
"io.kubernetes.pod.namespace": "default",
"io.kubernetes.pod.terminationGracePeriod": "30",
"io.kubernetes.pod.uid": "ceb15fbf-c3f6-11e6-a39e-000c29ec0bcd"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": null,
"SandboxKey": "",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": null
}
}
]
1.测试案例2
Kubernetes volume与Ceph RBD的结合，但是k8s volume还不能完全满足实际生产过程对持久化存储的需求，因为k8s
volume的lifetime和pod的生命周期相同，一旦pod被delete，那么volume中的数据就不复存在了。于是k8s又推出了
Persistent Volume(PV)和Persistent Volume Claim(PVC)组合，故名思意：即便挂载其的pod被delete了，PV依旧存在，PV
上的数据依旧存在。
（1）创建ceph的disk image
# rbd create ceph-image -s 128
（2）创建PV（ceph-pv.yaml，文件内容如下）
apiVersion: v1
kind: PersistentVolume
metadata:
name: ceph-pv
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
rbd:
monitors:
- 192.168.110.151:6789
pool: rbd
image: ceph-image
user: admin
secretRef:
name: ceph-secret
fsType: ext4
readOnly: false
persistentVolumeReclaimPolicy: Recycle
（3）创建PVC（ceph-pvc.yaml，文件内容如下）
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: ceph-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
说明：PV和PVC是如何联系在一起的呢，请看官网解释
https://docs.openshift.com/enterprise/3.1/install_config/storage_examples/ceph_example.html
（4）创建pod（ceph-es.yaml,文件内容如下）
apiVersion: v1
kind: Pod
metadata:
name: ceph-pod1
spec:
containers:
- name: es
image: 192.168.110.151:5000/elasticsearch:1.7.1
env:
- name: "CLUSTER_NAME"
value: "myesdb"
- name: NODE_MASTER
value: "true"
- name: NODE_DATA
value: "true"
- name: HTTP_ENABLE
value: "true"
ports:
- containerPort: 9200
name: http
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
volumeMounts:
- name: ceph-vol1
mountPath: /data
readOnly: false
volumes:
- name: ceph-vol1
persistentVolumeClaim:
claimName: ceph-claim
（5）查看pod运行状态
root@master:/home/docker/xu/ceph# kubectl get pod
NAME READY STATUS RESTARTS AGE
ceph-pod1 1/1 Running 0 41m
rbd2 1/1 Running 0 14m
注意：a）在这两个试验中一定要格式化image
b） ceph-secret在两个试验中是一样的，所以在实验二中没再创建
c）如果不对image格式化，不创建ceph-secret，那么pod将一直处于CreatingContainer状态
（6）查看ceph-pod1（pog）对应容器的
root@dockertest4:/opt/k8s# docker inspect 8352cf93f053
[
{
"Id": "8352cf93f0535a57c317a337b41fe35492ccd0b27da0f264f901f03d27ae5f1c",
"Created": "2016-12-17T00:51:34.452430089Z",
"Path": "/docker-entrypoint.sh",
"Args": [
"elasticsearch"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 19912,
"ExitCode": 0,
"Error": "",
"StartedAt": "2016-12-17T00:51:35.456354338Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:42014923291599c0ad9c1da557285d1a4d3c096dd9fbbb346189101931507640",
"ResolvConfPath": "/var/lib/docker/containers/86e869e1d45a9d79646aa0b0e8c17821f77cf95ca82a505d1761d9af4761501c/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/86e869e1d45a9d79646aa0b0e8c17821f77cf95ca82a505d1761d9af4761501c/hostname",
"HostsPath": "/var/lib/kubelet/pods/ee7ac2ca-c3f2-11e6-a39e-000c29ec0bcd/etc-hosts",
"LogPath": "/var/lib/docker/containers/8352cf93f0535a57c317a337b41fe35492ccd0b27da0f264f901f03d27ae5f1c/8352cf93f0535a57c317a337b41fe35492ccd0b27da0f264f901f03d27ae5f1c-json.log",
"Name": "/k8s_ceph-busybox.ec05c2ae_ceph-pod1_default_ee7ac2ca-c3f2-11e6-a39e-000c29ec0bcd_ad714501",
"RestartCount": 0,
"Driver": "aufs",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/var/lib/kubelet/pods/ee7ac2ca-c3f2-11e6-a39e-000c29ec0bcd/volumes/kubernetes.io~rbd/ceph-pv:/data",
"/var/lib/kubelet/pods/ee7ac2ca-c3f2-11e6-a39e-000c29ec0bcd/etc-hosts:/etc/hosts",
"/var/lib/kubelet/pods/ee7ac2ca-c3f2-11e6-a39e-000c29ec0bcd/containers/ceph-busybox/ad714501:/dev/termination-log"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "container:86e869e1d45a9d79646aa0b0e8c17821f77cf95ca82a505d1761d9af4761501c",
"PortBindings": null,
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "container:86e869e1d45a9d79646aa0b0e8c17821f77cf95ca82a505d1761d9af4761501c",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 1000,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined"
],
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 2,
"Memory": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": -1,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Name": "aufs",
"Data": null
},
"Mounts": [
{
"Source": "/var/lib/kubelet/pods/ee7ac2ca-c3f2-11e6-a39e-000c29ec0bcd/volumes/kubernetes.io~rbd/ceph-pv",
"Destination": "/data",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Source": "/var/lib/kubelet/pods/ee7ac2ca-c3f2-11e6-a39e-000c29ec0bcd/etc-hosts",
"Destination": "/etc/hosts",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Source": "/var/lib/kubelet/pods/ee7ac2ca-c3f2-11e6-a39e-000c29ec0bcd/containers/ceph-busybox/ad714501",
"Destination": "/dev/termination-log",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Name": "3617289e7981f85040680427cbb963453e678ddb7211ca4d8a2446a290d91101",
"Source": "/var/lib/docker/volumes/3617289e7981f85040680427cbb963453e678ddb7211ca4d8a2446a290d91101/_data",
"Destination": "/usr/share/elasticsearch/data",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "ceph-pod1",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"9200/tcp": {},
"9300/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"CLUSTER_NAME=myesdb",
"NODE_MASTER=true",
"NODE_DATA=true",
"HTTP_ENABLE=true",
"KUBERNETES_SERVICE_PORT_HTTPS=443",
"KUBERNETES_PORT=tcp://192.168.110.1:443",
"KUBERNETES_PORT_443_TCP=tcp://192.168.110.1:443",
"KUBERNETES_PORT_443_TCP_PROTO=tcp",
"KUBERNETES_PORT_443_TCP_PORT=443",
"KUBERNETES_PORT_443_TCP_ADDR=192.168.110.1",
"KUBERNETES_SERVICE_HOST=192.168.110.1",
"KUBERNETES_SERVICE_PORT=443",
"PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"JAVA_VERSION=8u66",
"JAVA_DEBIAN_VERSION=8u66-b01-1~bpo8+1",
"CA_CERTIFICATES_JAVA_VERSION=20140324",
"ELASTICSEARCH_MAJOR=1.7",
"ELASTICSEARCH_VERSION=1.7.1"
],
"Cmd": [
"elasticsearch"
],
"Image": "192.168.110.151:5000/elasticsearch:1.7.1",
"Volumes": {
"/usr/share/elasticsearch/data": {}
},
"WorkingDir": "",
"Entrypoint": [
"/docker-entrypoint.sh"
],
"OnBuild": null,
"Labels": {
"io.kubernetes.container.hash": "ec05c2ae",
"io.kubernetes.container.name": "ceph-busybox",
"io.kubernetes.container.ports": "[{\"name\":\"http\",\"containerPort\":9200,\"protocol\":\"TCP\"},{\"name\":\"transport\",\"containerPort\":9300,\"protocol\":\"TCP\"}]",
"io.kubernetes.container.restartCount": "0",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.pod.name": "ceph-pod1",
"io.kubernetes.pod.namespace": "default",
"io.kubernetes.pod.terminationGracePeriod": "30",
"io.kubernetes.pod.uid": "ee7ac2ca-c3f2-11e6-a39e-000c29ec0bcd"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": null,
"SandboxKey": "",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": null
}
}
]
九、验证试验2是否起作用
在没有引入PV和pvc之前，如果pod备删除，对应的数据也应该一并丢失，所以可以通过下面的方法对pv和pvc进行验
证
1） 向ceph-pod1对应的容器的/data目录下写测试数据（通过docker cp向里面拷贝一个文件就OK）
2） 删除ceph-pod1
3）重新创建ceph-pod1，查看/data下的数据是否还存在，如果存在说明本实验就是成功了