SUSE CaaS Platform 4 - 使用 Ceph RBD 作为持久存储（动态）

时间：2023-03-08 22:35:34

存储类

  • 存储类（storage class）是kubernetes资源类型，它是由管理员为管理PV之便而按需创建的类别
  • 存储类好处是支持 PV 的动态创建，系统按PVC的需求标准动态创建适配的PV会为存储管理带来极大的灵活性。
  • PV的动态供给，其重点是在存储类的定义，其分类大概是对存储的性能进行分类的，如图1：金存储类、银存储类、铜存储类等。

SUSE CaaS Platform 4 - 使用 Ceph RBD 作为持久存储（动态）

图1 基于综合服务质量的存储系统分类

一、Provisioner（存储分配器）

Storage class 有一个分配器，用来决定使用哪个卷插件分配 PV。图2 中可以看到，目前 Ceph 只有 RBD 接口支持内部分配器。

SUSE CaaS Platform 4 - 使用 Ceph RBD 作为持久存储（动态）  图2 各存储插件对动态供给方式的支持状况

二、实验环境搭建 - 动态供给

SUSE CaaS Platform 4 - 使用 Ceph RBD 作为持久存储（动态）

图3 实验环境架构图

环境设置

1、搭建环境

  • 操作系统版本： SLES15 SP1，无需安装 swap
  • 内核版本：4.12.14-197.18-default
  • Kubernetes版本：v1.15.2
  • VMware Workstation 14

2、虚拟化环境搭建和系统安装参考：

安装部署

1、所有 CaaS Platform 节点安装

# zypper install ceph-common

复制 ceph.conf 到 worker 节点上

# scp admin:/etc/ceph/ceph.conf /etc/ceph/

2、创建池,并将应用名称与存储池关联

# ceph osd pool create caasp4-dynamic

# ceph osd pool application enable caasp4-dynamic rbd

3、创建 CaaSP4 client user

# cd /etc/ceph

# ceph auth get-or-create client.caasp4-dynamic mon 'allow r' \

    osd 'allow class-read object_prefix rbd_children, allow rwx pool=caasp4-dynamic' \

    -o ceph.client.dynamic.keyring

4、获取 client.admin 用户 key 信息，并生成基于 base64 编码 key

# ceph auth get client.admin

exported keyring for client.admin

[client.admin]

        key = AQA9w4VdAAAAABAAHZr5bVwkALYo6aLVryt7YA==

        caps mds = "allow *"

        caps mgr = "allow *"

        caps mon = "allow *"

        caps osd = "allow *
# echo AQA9w4VdAAAAABAAHZr5bVwkALYo6aLVryt7YA== | base64

QVFBOXc0VmRBQUFBQUJBQUhacjViVndrQUxZbzZhTFZyeXQ3WUE9PQo=

5、在Master节点上，为 client.admin，创建 secret 资源

# vim ceph-secret-admin.yaml

apiVersion: v1

kind: Secret

metadata:

  name: ceph-secret-admin

  namespace: kube-system

data:

  key: QVFBOXc0VmRBQUFBQUJBQUhacjViVndrQUxZbzZhTFZyeXQ3WUE9PQo=

type: kubernetes.io/rbd
# kubectl get secrets -n kube-system

NAME                                       TYPE                                 DATA   AGE

....

ceph-secret-admin                          kubernetes.io/rbd                           24s

....

6、获取 client.caasp4-dynamic 用户 key 信息，并生成基于 base64编码的key

# ceph auth get client.caasp4-dynamic

exported keyring for client.caasp4-dynamic

[client.caasp4-dynamic]

        key = AQA29ppdTDmzHhAAET2mSbvovrS67kspPlqmLA==

        caps mon = "allow r"

        caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=caasp4-dynamic"
# echo AQA29ppdTDmzHhAAET2mSbvovrS67kspPlqmLA== | base64

QVFBMjlwcGRURG16SGhBQUVUMm1TYnZvdnJTNjdrc3BQbHFtTEE9PQo=

7、在Master节点上，为 client.caasp4-dynamic 创建 secret

# vim ceph-secret-user.yaml

apiVersion: v1

kind: Secret

metadata:

  name: ceph-user-secret

  namespace: default

data:

  key: QVFBMjlwcGRURG16SGhBQUVUMm1TYnZvdnJTNjdrc3BQbHFtTEE9PQo=

type: kubernetes.io/rbd
# kubectl create -f ceph-secret-user.yaml

secret "ceph-user-secret" created
# kubectl get secrets

NAME                  TYPE                                  DATA   AGE

ceph-secret-test      Opaque                                      20h

ceph-user-secret      kubernetes.io/rbd                           4s

default-token-4hslq   kubernetes.io/service-account-token         24h

8、创建 storage class 存储类

# vim ceph-storageclass.yaml

apiVersion: storage.k8s.io/v1beta1

kind: StorageClass

metadata:

  name: dynamic

  annotations:

     storageclass.beta.kubernetes.io/is-default-class: "true"

provisioner: kubernetes.io/rbd

parameters:

  monitors: 192.168.2.40:,192.168.2.41:,192.168.2.42:

  adminId: admin

  adminSecretName: ceph-secret-admin

  adminSecretNamespace: kube-system

  pool: caasp4-dynamic

  userId: caasp4-dynamic

  userSecretName: ceph-user-secret
# kubectl create -f ceph-storageclass.yaml

storageclass "dynamic" created
# kubectl get storageclasses

NAME                PROVISIONER         AGE

dynamic (default)   kubernetes.io/rbd   10m

StorageClass SPEC

（1）provisioner(供给方)：即提供存储资源的存储系统，供给方名字都以“kubernetes.io”为前缀
（2）parameters（参数）：使用参数描述要关联到的存储卷，不同的provisioner有不同的参数

SUSE CaaS Platform 4 - 使用 Ceph RBD 作为持久存储（动态）

kubernetes 官方参数说明

9、创建 PVC

# vim ceph-pvc.yaml

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

  name: ceph-claim-dynamic

spec:

  accessModes:

    - ReadWriteOnce

  resources:

    requests:

      storage: 3Gi
# kubectl create -f ceph-pvc.yaml

persistentvolumeclaim "ceph-claim-dynamic" created
# kubectl get pvc -o wide

NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE

ceph-claim-dynamic   Bound    pvc-70b3b3ca---a898-c1f5524de008   3Gi        RWO            dynamic        10m   Filesystem

10、创建 Pod

vim ceph-pod-dynamic.yaml

apiVersion: v1

kind: Pod

metadata:

  name: ceph-pod1-dynamic

spec:

  containers:

  - name: ceph-busybox

    image: busybox

    command: ["sleep", ""]

    volumeMounts:

    - name: ceph-vol1-dynamic

      mountPath: /usr/share/busybox

      readOnly: false

  volumes:

  - name: ceph-vol1-dynamic

    persistentVolumeClaim:

      claimName: ceph-claim-dynamic
# kubectl create -f ceph-pod-dynamic.yaml

pod "ceph-pod1-dynamic" created

# kubectl get pods

NAME                READY   STATUS    RESTARTS   AGE

ceph-pod1-dynamic   /     Running             24m

# kubectl get pods -o wide

NAME                READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES

ceph-pod1-dynamic   /     Running             48m   10.244.2.194   worker02   <none>           <none>
# kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS   REASON   AGE

pvc-70b3b3ca-5267-4417-a898-c1f5524de008   3Gi        RWO            Delete           Bound    default/ceph-claim-dynamic   dynamic                 169m

11、storage6 分布式存储上，查看是否创建了镜像

admin:/etc/ceph # rbd ls -p caasp4-dynamic

kubernetes-dynamic-pvc-e6d98bbf-50e1--a9fc-867d1db810c8
admin:/etc/ceph # rbd info kubernetes-dynamic-pvc-e6d98bbf-50e1--a9fc-867d1db810c8 -p caasp4-dynamic

rbd image 'kubernetes-dynamic-pvc-e6d98bbf-50e1-4488-a9fc-867d1db810c8':

        size  GiB in  objects

        order  ( MiB objects)

        snapshot_count:

        id: cd53f75a03b15

        block_name_prefix: rbd_data.cd53f75a03b15

        format:

        features:

        op_features:

        flags:

        create_timestamp: Mon Oct   ::

        access_timestamp: Mon Oct   ::

        modify_timestamp: Mon Oct   ::

12、worker02节点上，查看RBD映射

# rbd showmapped

id pool           namespace image                                                       snap device

0  caasp4-dynamic           kubernetes-dynamic-pvc-e6d98bbf-50e1-4488-a9fc-867d1db810c8 -    /dev/rbd0

# df -h | grep dev/rbd0

/dev/rbd0                2.9G  9.0M  2.9G   1% /var/lib/kubelet/pods/e7c75785-4533-4fac-b4ab-368c75e16421/volumes/kubernetes.io~rbd/pvc-70b3b3ca-5267-4417-a898-c1f5524de008

排错

1、搭建的时候，发现创建PVC的时候 pending 状态。

# kubectl get pvc

NAME                 STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE

ceph-claim-dynamic   Pending                                      dynamic        44s

2、通过 event 事件查看，由于忘记创建 admin secret 导致，重新创建 admin secret 即可。

# kubectl get events

LAST SEEN   TYPE      REASON               OBJECT                                     MESSAGE

60s         Warning   ProvisioningFailed   persistentvolumeclaim/ceph-claim-dynamic   Failed to provision volume with StorageClass "dynamic": failed to get admin secret from ["kube-system"/"ceph-secret-admin"]: failed to get secret from ["kube-system"/"ceph-secret-admin"]: secrets "ceph-secret-admin" not found

35s         Warning   FailedScheduling     pod/ceph-pod1-dynamic                      pod has unbound immediate PersistentVolumeClaims (repeated 2 times)

SUSE CaaS Platform 4 - 使用 Ceph RBD 作为持久存储（动态）