如何快速搭建k8s的kubemark性能测试环境

时间:2022-10-19 12:01:02

Kubemark学习文档

搭建k8s集群

搭建细节参考:https://blog.51cto.com/u_12040959/5043212

什么是kubemark

kubemark是K8S官方给出的性能测试工具,能够利用相当小的资源,模拟出一个大规模K8S集群。其主要架构如图所示:需要一个外部 K8S集群(external cluster,需要带有worker节点)以及一套完整的kubemark master控制面(可以是单节点也可以是多节点),即另外一个 K8S集群(kubemark集群),但是该只有master节点,没有worker节点。我们需要在external cluster中部署运行hollow pod,这些pod会主动向kubemark集群中的master注册,成为kubemark集群中的 hollow node(虚拟节点)。然后我们就可以在kubemark集群中进行性能测试。虽然与真实集群的稍微有点误差,不过可以代表真实集群的数据。

如何快速搭建k8s的kubemark性能测试环境

总之需要两个集群,一个external cluster集群,需要工作节点,一个kubemark集群只需要master节点。

搭建过程

3.1拉取镜像

external cluster集群手动拉取kubemark镜像。

官网地址:https://hub.docker.com/ 搜索kubemark

找到合适的镜像 例:docker pull ss104301/kubemark

如何快速搭建k8s的kubemark性能测试环境

3.2创建secret

将kubemark k8s的config文件(任意kubemark k8s主节点的/root/.kube/config)拷贝到当前路径(/home/hdf)

kubectl create ns kubemark

kubectl create secret generic kubeconfig --type=Opaque --namespace=kubemark --from-file=kubelet.kubecnotallow=config --from-file=kubeproxy.kubecnotallow=config


3.3 创建ReplicationController

镜像都需要提前手动下载,以及替换成自己需要的镜像

apiVersion: v1

kind: ReplicationController

metadata:

name: hollow-node

namespace: kubemark

labels:

name: hollow-node

spec:

replicas: 3

selector:

name: hollow-node

template:

metadata:

labels:

name: hollow-node

spec:

initContainers:

- name: init-inotify-limit

image: busybox:1.32

imagePullPolicy: IfNotPresent

command: ['sysctl', '-w', 'fs.inotify.max_user_instances=1000']

securityContext:

privileged: true

volumes:

- name: kubeconfig-volume

secret:

secretName: kubeconfig

- name: logs-volume

hostPath:

path: /var/log

- name: no-serviceaccount-access-to-real-master

emptyDir: {}

containers:

- name: hollow-kubelet

image: staging-k8s.gcr.io/kubemark:v1.20.10

imagePullPolicy: IfNotPresent

ports:

- containerPort: 4194

- containerPort: 10250

- containerPort: 10255

env:

- name: NODE_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

command: [

"/kubemark",

"--morph=kubelet",

"--name=$(NODE_NAME)",

"--kubecnotallow=/kubeconfig/kubelet.kubeconfig",

"--log-file=/var/log/kubelet-$(NODE_NAME).log",

"--logtostderr=false"

]

volumeMounts:

- name: kubeconfig-volume

mountPath: /kubeconfig

readOnly: true

- name: logs-volume

mountPath: /var/log

resources:

requests:

cpu: 50m

memory: 100M

securityContext:

privileged: true

- name: hollow-proxy

image: staging-k8s.gcr.io/kubemark:v1.20.10

imagePullPolicy: IfNotPresent

env:

- name: NODE_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

command: [

"/kubemark",

"--morph=proxy",

"--name=$(NODE_NAME)",

"--kubecnotallow=/kubeconfig/kubeproxy.kubeconfig",

"--log-file=/var/log/kubeproxy-$(NODE_NAME).log",

"--logtostderr=false"

]

volumeMounts:

- name: kubeconfig-volume

mountPath: /kubeconfig

readOnly: true

- name: logs-volume

mountPath: /var/log

resources:

requests:

cpu: 50m

memory: 100M


3.4效果图

External集群

如何快速搭建k8s的kubemark性能测试环境

Kubemark集群

如何快速搭建k8s的kubemark性能测试环境

基于perf-test测试性能

4.1安装go基础环境

官网下载对于系统的golang安装包,然后解压到/usr/local目录即可使用


cd /usr/local

wget https://golang.google.cn/dl/go1.17.9.linux-amd64.tar.gz

tar -zxvf go1.17.9.linux-amd64.tar.g

vi /etc/profile

export PATH=$PATH:/usr/local/go/bin

source /etc/profile

4.2下载perf-tests源码

官网地址:https://github.com/kubernetes/perf-tests/

cd perf-tests/

go build -o clusterloader './cmd/'

# kube config for kubernetes api

KUBE_CONFIG=${HOME}/.kube/config


# Provider setting

# Supported provider for xiaomi: local, kubemark, lvm-local, lvm-kubemark

PROVIDER='kubemark'


# SSH config for metrics' collection

KUBE_SSH_KEY_PATH=$HOME/.ssh/id_rsa

MASTER_SSH_IP=10.142.43.51

MASTER_SSH_USER_NAME=root


# Clusterloader2 testing strategy config paths

# It supports setting up multiple test strategy. Each testing strategy is individual and serial.

TEST_CONFIG='configs/examples/density/config.yaml'


# Clusterloader2 testing override config paths

# It supports setting up multiple override config files. All of override config files will be applied to each testing strategy.

# OVERRIDE_CONFIG='testing/density/override/200-nodes.yaml'


# Log config

REPORT_DIR='./reports'

LOG_FILE='logs/tmp.log'


4.3运行clusterloader2

$CLUSTERLOADER_BIN --kubecnotallow=$KUBE_CONFIG \

--provider=$PROVIDER \

--masterip=$MASTER_SSH_IP --mastername=$MASTER_SSH_USER_NAME \

--testcnotallow=$TEST_CONFIG \

--report-dir=$REPORT_DIR \

--alsologtostderr 2>&1 | tee $LOG_FILE

# --testoverrides="${OVERRIDE_CONFIG:-}" \

4.4测试结果

整个性能测试过程以及部分测试结果会打印到stdout中,如果按本文所述方法运行clusterloader2,打印结果会保存一份至${LOG_FILE}。


调度吞吐量结果:

Jan 9 13:55:47.876: INFO: E2E startup time for 10500 pods: 28m40.333662404s

Jan 9 13:55:47.876: INFO: Throughput (pods/s) during cluster saturation phase: 6.1034675

^[[1mSTEP^[[0m: Printing Pod to Node allocation data

Jan 9 13:55:47.884: INFO: Density Pods: 10500 out of 10500 created, 10500 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady



Pod启动e2e耗时结果:

Jan 9 14:09:31.598: INFO: perc50: 1614, perc90: 2241, perc99: 2541

Jan 9 14:09:31.598: INFO: Approx throughput: 7486.052881923156 pods/min


report文件夹

详细的性能测试结果clusterloader2收集在report文件夹下,如果按本文方法运行,则结果存放在{$REPORT_DIR}中。因为涉及到的性能测试结果非常多,本文不在此一一列举。目前Kubernetes性能分析最主要的性能指标如下:

APIServer Restful API响应时间 - APIResponsiveness

Pod 启动总耗时 - PodStartupLatency

Scheduler调度性能 - SchedulingThroughput, SchedulingMetrics

ETCD指标 - EtcdMetrics