Kubemark学习文档
搭建k8s集群
搭建细节参考:https://blog.51cto.com/u_12040959/5043212
什么是kubemark
kubemark是K8S官方给出的性能测试工具,能够利用相当小的资源,模拟出一个大规模K8S集群。其主要架构如图所示:需要一个外部 K8S集群(external cluster,需要带有worker节点)以及一套完整的kubemark master控制面(可以是单节点也可以是多节点),即另外一个 K8S集群(kubemark集群),但是该只有master节点,没有worker节点。我们需要在external cluster中部署运行hollow pod,这些pod会主动向kubemark集群中的master注册,成为kubemark集群中的 hollow node(虚拟节点)。然后我们就可以在kubemark集群中进行性能测试。虽然与真实集群的稍微有点误差,不过可以代表真实集群的数据。
总之需要两个集群,一个external cluster集群,需要工作节点,一个kubemark集群只需要master节点。
搭建过程
3.1拉取镜像
在external cluster集群手动拉取kubemark镜像。
官网地址:https://hub.docker.com/ 搜索kubemark
找到合适的镜像 例:docker pull ss104301/kubemark
3.2创建secret
将kubemark k8s的config文件(任意kubemark k8s主节点的/root/.kube/config)拷贝到当前路径(/home/hdf)
kubectl create ns kubemark
kubectl create secret generic kubeconfig --type=Opaque --namespace=kubemark --from-file=kubelet.kubecnotallow=config --from-file=kubeproxy.kubecnotallow=config
3.3 创建ReplicationController
镜像都需要提前手动下载,以及替换成自己需要的镜像
apiVersion: v1
kind: ReplicationController
metadata:
name: hollow-node
namespace: kubemark
labels:
name: hollow-node
spec:
replicas: 3
selector:
name: hollow-node
template:
metadata:
labels:
name: hollow-node
spec:
initContainers:
- name: init-inotify-limit
image: busybox:1.32
imagePullPolicy: IfNotPresent
command: ['sysctl', '-w', 'fs.inotify.max_user_instances=1000']
securityContext:
privileged: true
volumes:
- name: kubeconfig-volume
secret:
secretName: kubeconfig
- name: logs-volume
hostPath:
path: /var/log
- name: no-serviceaccount-access-to-real-master
emptyDir: {}
containers:
- name: hollow-kubelet
image: staging-k8s.gcr.io/kubemark:v1.20.10
imagePullPolicy: IfNotPresent
ports:
- containerPort: 4194
- containerPort: 10250
- containerPort: 10255
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
command: [
"/kubemark",
"--morph=kubelet",
"--name=$(NODE_NAME)",
"--kubecnotallow=/kubeconfig/kubelet.kubeconfig",
"--log-file=/var/log/kubelet-$(NODE_NAME).log",
"--logtostderr=false"
]
volumeMounts:
- name: kubeconfig-volume
mountPath: /kubeconfig
readOnly: true
- name: logs-volume
mountPath: /var/log
resources:
requests:
cpu: 50m
memory: 100M
securityContext:
privileged: true
- name: hollow-proxy
image: staging-k8s.gcr.io/kubemark:v1.20.10
imagePullPolicy: IfNotPresent
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
command: [
"/kubemark",
"--morph=proxy",
"--name=$(NODE_NAME)",
"--kubecnotallow=/kubeconfig/kubeproxy.kubeconfig",
"--log-file=/var/log/kubeproxy-$(NODE_NAME).log",
"--logtostderr=false"
]
volumeMounts:
- name: kubeconfig-volume
mountPath: /kubeconfig
readOnly: true
- name: logs-volume
mountPath: /var/log
resources:
requests:
cpu: 50m
memory: 100M
3.4效果图
External集群
Kubemark集群
基于perf-test测试性能
4.1安装go基础环境
官网下载对于系统的golang安装包,然后解压到/usr/local目录即可使用
cd /usr/local
wget https://golang.google.cn/dl/go1.17.9.linux-amd64.tar.gz
tar -zxvf go1.17.9.linux-amd64.tar.g
vi /etc/profile
export PATH=$PATH:/usr/local/go/bin
source /etc/profile
4.2下载perf-tests源码
官网地址:https://github.com/kubernetes/perf-tests/
cd perf-tests/
go build -o clusterloader './cmd/'
# kube config for kubernetes api
KUBE_CONFIG=${HOME}/.kube/config
# Provider setting
# Supported provider for xiaomi: local, kubemark, lvm-local, lvm-kubemark
PROVIDER='kubemark'
# SSH config for metrics' collection
KUBE_SSH_KEY_PATH=$HOME/.ssh/id_rsa
MASTER_SSH_IP=10.142.43.51
MASTER_SSH_USER_NAME=root
# Clusterloader2 testing strategy config paths
# It supports setting up multiple test strategy. Each testing strategy is individual and serial.
TEST_CONFIG='configs/examples/density/config.yaml'
# Clusterloader2 testing override config paths
# It supports setting up multiple override config files. All of override config files will be applied to each testing strategy.
# OVERRIDE_CONFIG='testing/density/override/200-nodes.yaml'
# Log config
REPORT_DIR='./reports'
LOG_FILE='logs/tmp.log'
4.3运行clusterloader2
$CLUSTERLOADER_BIN --kubecnotallow=$KUBE_CONFIG \
--provider=$PROVIDER \
--masterip=$MASTER_SSH_IP --mastername=$MASTER_SSH_USER_NAME \
--testcnotallow=$TEST_CONFIG \
--report-dir=$REPORT_DIR \
--alsologtostderr 2>&1 | tee $LOG_FILE
# --testoverrides="${OVERRIDE_CONFIG:-}" \
4.4测试结果
整个性能测试过程以及部分测试结果会打印到stdout中,如果按本文所述方法运行clusterloader2,打印结果会保存一份至${LOG_FILE}。
调度吞吐量结果:
Jan 9 13:55:47.876: INFO: E2E startup time for 10500 pods: 28m40.333662404s
Jan 9 13:55:47.876: INFO: Throughput (pods/s) during cluster saturation phase: 6.1034675
^[[1mSTEP^[[0m: Printing Pod to Node allocation data
Jan 9 13:55:47.884: INFO: Density Pods: 10500 out of 10500 created, 10500 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady
Pod启动e2e耗时结果:
Jan 9 14:09:31.598: INFO: perc50: 1614, perc90: 2241, perc99: 2541
Jan 9 14:09:31.598: INFO: Approx throughput: 7486.052881923156 pods/min
report文件夹
详细的性能测试结果clusterloader2收集在report文件夹下,如果按本文方法运行,则结果存放在{$REPORT_DIR}中。因为涉及到的性能测试结果非常多,本文不在此一一列举。目前Kubernetes性能分析最主要的性能指标如下:
APIServer Restful API响应时间 - APIResponsiveness
Pod 启动总耗时 - PodStartupLatency
Scheduler调度性能 - SchedulingThroughput, SchedulingMetrics
ETCD指标 - EtcdMetrics