Zero to JupyterHub with Kubernetes上篇 - Kubernetes 离线二进制部署

前言： 纯个人记录使用。

• 搭建 Zero to JupyterHub with Kubernetes 上篇 - Kubernetes 离线二进制部署。
• 搭建 Zero to JupyterHub with Kubernetes 中篇 - Kubernetes 常规使用记录。
• 搭建 Zero to JupyterHub with Kubernetes 下篇 - Jupyterhub on k8s。

k8s二进制部署所需离线包和镜像
链接：https://pan.baidu.com/s/1z8quvOEoLgH0x7jkZWfVEw
提取码：1234

参考：

https://www.yuque.com/fairy-era/yg511q/xyqxge

https://blog.****.net/2301_77428746/article/details/140032125

文章目录

• • 1、 集群架构
  • 2、 cfssl证书生成工具
  • 3、Etcd集群部署
  • • 3.1 使用自签CA机构签发Etcd服务ssl证书
    • 3.2 部署Etcd集群
  • 4、安装docker
  • 5、 Master节点部署
  • • 5.1 使用自签CA签发kube-apiserver HTTPS证书
    • 5.2 部署kube-apiserver
    • • 5.2.1 服务启动错误
    • 5.3 部署kube-controller-manager
    • 5.4 部署kube-scheduler
    • 5.5 查看集群状态
  • 6、 Node节点部署
  • • 6.1 kubelet 部署
    • 6.2 kube-proxy部署
  • 7、 网络插件部署calico
  • 8、 授权apiserver访问kubelet
  • 9、 node1、node2 节点加入 woker node
  • 10、 部署CoreDNS和Dashboard
  • • 10.1 部署CoreDNS
    • 10.2 部署Dashboard

1、 集群架构

主机	角色	组件	主机版本
10.34.X.10	k8s-Master	Kube-apiserver、Kube-controller-manager、Kube-Scheduler\docker、calico、Etcd	centos7.9
10.34.X.11	k8s-Node1	Kubelet、Kube-proxy、docker、calico、Etcd	centos7.9
10.34.X.12	k8s-Node2	Kubelet、Kube-proxy、docker、calico、Etcd	centos7.9

软件	版本
Docker	19.03.9
Kubernetes	v1.20.4
calico	v3.15.1
etcd	v3.4.9

环境准备

# 1、3台机器配置ssh免密登录
> ssh-keygen -t rsa -b 4096
> ssh-copy-id username@hostname

# 2、主机名映射
> cat /etc/hosts
  10.34.X.10      k8s-Master
  10.34.X.11      k8s-Node1
  10.34.X.12      k8s-Node2  
  
# 3、 机器防火墙状态（未启用）
> systemctl status firewalld   # dead

# 4、 安全模块selinux状态
> getenforce    # Disabled

# 5、 swap分区禁用
> swapoff -a        # 临时关闭swap 
> vim /etc/fstab    # 注销掉swap分区配置
> free -h              total        used        free      shared  buff/cache   available
Mem:           251G         78G        2.4G        794M        169G        170G
Swap:            0B          0B          0B

# 6、 将桥接的IPv4流量传递到iptables的链 
> vim /etc/sysctl.d/k8s.conf 
'''
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
'''
> sysctl --system  # 生效

2、 cfssl证书生成工具

## cfssl 工具
[root@k8s-master /data/kubernetes/cfssl]$ tar -xzf cfssl.tar.gz
[root@k8s-master /data/kubernetes/cfssl]$ mv cfssl /usr/local/bin/cfssl               # 用于签发证书
[root@k8s-master /data/kubernetes/cfssl]$ mv cfssljson /usr/local/bin/cfssljson       # 将cfssl生成的证书(json)变成证书文件(pem)     
[root@k8s-master /data/kubernetes/cfssl]$ mv cfssl-certinfo /usr/bin/cfssl-certinfo   # 验证或查看证书

## 生成Etcd证书
#创建目录
[root@k8s-master ~]$ mkdir -p ca/etcd 
[root@k8s-master ~]$ cd ca/etcd
#自签CA机构配置文件：定义证书颁发机构（CA）的签名配置和策略。它通常包含关于证书过期时间、用途、签名配置等设置
[root@k8s-master ~/ca/etcd]$ vim ca-config.json
{
	"signing": {
		"default": {                // 默认签名配置
			"expiry": "87600h"      // 所有签发证书的默认有效期10年
		},
		"profiles": {                    // 定义不同类型证书的详细签名配置
			"www": {
				"expiry": "87600h",
				"usages": [              // 定义证书的用途
					"signing",
					"key encipherment",
					"server auth",
					"client auth"
				]
			}
		}
	}
}
#自签ca机构根证书签名申请文件
[root@k8s-master ~/ca/etcd]$ vim ca-csr.json
{
	"CN": "etcd CA",
	"key": {
		"algo": "rsa",
		"size": 2048
	},
	"names": [{
		"C": "CN",
		"L": "Beijing",
		"ST": "Beijing"
	}]
}

#生成ca机构证书
[root@k8s-master ~/ca/etcd]$ cfssl gencert -initca ca-csr.json  | cfssljson -bare ca -
ca.csr   ca-key.pem   ca.pem
# ca.csr 证书请求文件 ca.pem、ca-key.pem CA根证书文件及其私钥文件 

3、Etcd集群部署

3.1 使用自签CA机构签发Etcd服务ssl证书

## 使用自签CA签发Etcd HTTPS证书
#创建etcd服务证书申请文件
[root@k8s-master ~/ca/etcd]$ vim server-csr.json
{
    "CN": "etcd",
    "hosts": [             // 列出了该证书应该支持的所有主机名或域名
    "10.34.x.10",
    "10.34.x.11",
    "10.34.x.12"
    ],
    "key": {               // 指定秘钥算法及秘钥长度
        "algo": "rsa",
        "size": 2048
    },
    "names": [             // 该服务机构信息
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}

# CN（common name）：申请者名称
# hosts: 网络请求url中的合法主机名或域名集合
# key: 加密说明
# names: 所在国家、省市等信息

# 生成Etcd服务证书
[root@k8s-master ~/ca/etcd]$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
server.csr server-key.pem  server.pem

3.2 部署Etcd集群

# 解压
[root@k8s-master /data/s0/kubernetes/etcd]$ tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
# 创建etcd配置文件
[root@k8s-master /data/s0/kubernetes/etcd]$ vim etcd.conf
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.34.x.10:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.34.x.10:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.34.x.10:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.34.x.10:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://10.34.x.10:2380,etcd-2=https://10.34.x.11:2380,etcd-3=https://10.34.x.12:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

*参数解释
•	ETCD_NAME：节点名称，集群中唯一
•	ETCD_DATA_DIR：数据目录
•	ETCD_LISTEN_PEER_URLS：集群通信监听地址
•	ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
•	ETCD_INITIAL_ADVERTISE_PEERURLS：集群通告地址
•	ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
•	ETCD_INITIAL_CLUSTER：集群节点地址
•	ETCD_INITIALCLUSTER_TOKEN：集群Token
•	ETCD_INITIALCLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

# 配置系统服务
[root@k8s-master /data/s0/kubernetes/etcd]$ vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/data/s0/kubernetes/etcd/etcd.conf
ExecStart=/data/s0/kubernetes/etcd/etcd-v3.4.9-linux-amd64/etcd \
--cert-file=/root/ca/etcd/server.pem \
--key-file=/root/ca/etcd/server-key.pem \
--peer-cert-file=/root/ca/etcd/server.pem \
--peer-key-file=/root/ca/etcd/server-key.pem \
--trusted-ca-file=/root/ca/etcd/ca.pem \
--peer-trusted-ca-file=/root/ca/etcd/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target


# 将k8s-master的配置copy到k8s-node1、k8s-node2
[root@k8s-master ~]$ scp -r ~/ca 10.34.x.11:~/
[root@k8s-master ~]$ scp -r ~/ca 10.34.x.12:~/

[root@k8s-master ~]$ scp -r /data/s0/kubernetes/etcd 10.34.x.11:/data/s0/kubernetes
[root@k8s-master ~]$ scp -r /data/s0/kubernetes/etcd 10.34.x.12:/data/s0/kubernetes

[root@k8s-master ~]$ scp /usr/lib/systemd/system/etcd.service  10.34.x.11:/usr/lib/systemd/system
[root@k8s-master ~]$ scp /usr/lib/systemd/system/etcd.service  10.34.x.12:/usr/lib/systemd/system

# node1、node2 节点修改配置
[root@k8s-node1 ~]$ vim /data/s0/kubernetes/etcd/etcd.conf
#[Member]
ETCD_NAME="etcd-2" # 名称各节点不一样，注意
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.34.x.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.34.x.11:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.34.x.11:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.34.x.11:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://10.34.x.10:2380,etcd-2=https://10.34.x.11:2380,etcd-3=https://10.34.x.12:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

[root@k8s-node2 ~]$ vim /data/s0/kubernetes/etcd/etcd.conf
#[Member]
ETCD_NAME="etcd-3"   # 名称各节点不一样，注意
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.34.x.12:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.34.x.12:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.34.x.12:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.34.x.12:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://10.34.x.10:2380,etcd-2=https://10.34.x.11:2380,etcd-3=https://10.34.x.12:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

# 启动Etcd服务
[root@k8s-master ~]$ systemctl start etcd
[root@k8s-node1 ~]$ systemctl start etcd
[root@k8s-node2 ~]$ systemctl start etcd

# 查看集群状态
[root@k8s-master /data/s0/kubernetes/etcd/etcd-v3.4.9-linux-amd64]$./etcdctl --cacert=/root/ca/etcd/ca.pem --cert=/root/ca/etcd/server.pem --key=/root/ca/etcd/server-key.pem --endpoints="https://10.34.x.10:2379,https://10.34.x.11:2379,https://10.34.x.12:2379" endpoint health --write-out=table

+--------------------------+--------+-------------+-------+
|         ENDPOINT         | HEALTH |    TOOK     | ERROR |
+--------------------------+--------+-------------+-------+
| https://10.34.x.10:2379 |   true | 28.399299ms |       |
| https://10.34.x.11:2379 |   true | 28.433169ms |       |
| https://10.34.x.12:2379 |   true | 28.925481ms |       |
+--------------------------+--------+-------------+-------+

 

4、安装docker

# 解压安装
[root@k8s-master /data/s0/kubernetes/docker]$ tar zxvf docker-19.03.9.tgz
[root@k8s-master /data/s0/kubernetes/docker]$ cp docker/* /usr/bin
# 配置
[root@k8s-master /data/s0/kubernetes/docker]$ mkdir /etc/docker
[root@k8s-master /data/s0/kubernetes/docker]$ vim /etc/docker/daemon.json

{
    "data-root": "/data/s0/kubernetes/docker/docker_data"     # docker 数据保存地址默认保存地址/var/lib/docker
    "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]  # 镜像源，离线机器应该使用不到
}

# 配置系统服务
[root@k8s-master /data/s0/kubernetes/docker]$ vim /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

# 启动docker服务
[root@k8s-master /data/s0/kubernetes/docker]$ systemctl start docker

# node1、node2同时开启docker服务
[root@k8s-master /data/s0/kubernetes/docker]$ scp docker/* 10.34.x.11:/usr/bin
[root@k8s-master /data/s0/kubernetes/docker]$ scp docker/* 10.34.x.12:/usr/bin

[root@k8s-master /data/s0/kubernetes/docker]$ scp -r /etc/docker 10.34.x.11:/etc
[root@k8s-master /data/s0/kubernetes/docker]$ scp -r /etc/docker 10.34.x.12:/etc

[root@k8s-master /data/s0/kubernetes/docker]$ scp /usr/lib/systemd/system/docker.service 10.34.x.11:/usr/lib/systemd/system
[root@k8s-master /data/s0/kubernetes/docker]$ scp /usr/lib/systemd/system/docker.service 10.34.x.12:/usr/lib/systemd/system

[root@k8s-node1 ~]$ systemctl start docker
[root@k8s-node2 ~]$ systemctl start docker

5、 Master节点部署

5.1 使用自签CA签发kube-apiserver HTTPS证书

# 创建目录
[root@k8s-master ~]$ mkdir ca/k8s

# 服务签名配置文件
[root@k8s-master ~/ca/k8s]$ vim ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
# ca自签机构根证书签名请求
[root@k8s-master ~/ca/k8s]$ vim ca-csr.json
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

# 生成证书
[root@k8s-master1 ~/ca/k8s]$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
# 创建kube-apiserver服务证书申请文件
[root@k8s-master ~/ca/k8s]$ vim server-csr.json
{
	"CN": "kubernetes",
	"hosts": [
		"10.0.0.1",
		"127.0.0.1",
		"10.34.x.10", // master
		"10.34.x.11", // node1
		"10.34.x.12", // node2
		"kubernetes",
		"kubernetes.default",
		"kubernetes.default.svc",
		"kubernetes.default.svc.cluster",
		"kubernetes.default.svc.cluster.local"
	],
	"key": {
		"algo": "rsa",
		"size": 2048
	},
	"names": [{
		"C": "CN",
		"L": "BeiJing",
		"ST": "BeiJing",
		"O": "k8s",
		"OU": "System"
	}]
}

# 生成server.pem和server-key.pem文件
[root@k8s-master ~/ca/k8s]$  cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

5.2 部署kube-apiserver

# 解压
[root@k8s-master /data/s0/kubernetes/k8s]$ tar -zxvf kubernetes-v1.20.4-server-linux-amd64.tar.gz
[root@k8s-master /data/s0/kubernetes/k8s]$ cp kubernetes/server/bin/kubectl  /usr/bin
[root@k8s-master /data/s0/kubernetes/k8s]$ mkdir {bin,cfg,logs}
[root@k8s-master /data/s0/kubernetes/k8s]$ cp kubernetes/server/bin/{kube-apiserver,kube-scheduler,kube-controller-manager,kubelet,kube-proxy} ./bin

# 创建配置文件
[root@k8s-master /data/s0/kubernetes/k8s]$ vim kube-apiserver.conf
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/s0/kubernetes/k8s/logs \
--etcd-servers=https://10.34.x.10:2379,https://10.34.x.11:2379,https://10.34.x.12:2379 \
--bind-address=10.34.x.10 \
--secure-port=6443 \
--advertise-address=10.34.x.10 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/data/s0/kubernetes/k8s/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/root/ca/k8s/server.pem \
--kubelet-client-key=/root/ca/k8s/server-key.pem \
--tls-cert-file=/root/ca/k8s/server.pem  \
--tls-private-key-file=/root/ca/k8s/server-key.pem \
--client-ca-file=/root/ca/k8s/ca.pem \
--service-account-key-file=/root/ca/k8s/ca-key.pem \
--service-account-issuer=https://kubernetes.default.svc.cluster.local  \
--service-account-signing-key-file=/root/ca/k8s/ca-key.pem \
--etcd-cafile=/root/ca/etcd/ca.pem \
--etcd-certfile=/root/ca/etcd/server.pem \
--etcd-keyfile=/root/ca/etcd/server-key.pem \
--requestheader-client-ca-file=/root/ca/k8s/ca.pem \
--proxy-client-cert-file=/root/ca/k8s/server.pem \
--proxy-client-key-file=/root/ca/k8s/server-key.pem \
--requestheader-allowed-names=kubernetes \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=true \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/data/s0/kubernetes/k8s/logs/k8s-audit.log"


参考说明
•	--logtostderr：启用日志
•	---v：日志等级
•	--log-dir：日志目录
•	--etcd-servers：etcd集群地址
•	--bind-address：监听地址
•	--secure-port：https安全端口
•	--advertise-address：集群通告地址
•	--allow-privileged：启用授权
•	--service-cluster-ip-range：Service虚拟IP地址段
•	--enable-admission-plugins：准入控制模块
•	--authorization-mode：认证授权，启用RBAC授权和节点自管理
•	--enable-bootstrap-token-auth：启用TLS bootstrap机制
•	--token-auth-file：bootstrap token文件
•	--service-node-port-range：Service nodeport类型默认分配端口范围
•	--kubelet-client-xxx：apiserver访问kubelet客户端证书
•	--tls-xxx-file：apiserver https证书
•	1.20版本必须加的参数：--service-account-issuer，--service-account-signing-key-file
•	--etcd-xxxfile：连接Etcd集群证书
•	--audit-log-xxx：审计日志
•	启动聚合层相关配置：--requestheader-client-ca-file，--proxy-client-cert-file，--proxy-client-key-file，--requestheader-allowed-names，--requestheader-extra-headers-prefix，--requestheader-group-headers，--requestheader-username-headers，--enable-aggregator-routing

# 配置token文件
[root@k8s-master /data/s0/kubernetes/k8s]$ vim cfg/token.csv
bfd627b0217a49e8626ba1caf1259e0c,kubelet-bootstrap,10001,system:node-bootstrapper

#注：上述token可自行生成替换，但一定要与后续配置对应
> head -c 16 /dev/urandom | od -An -t x | tr -d ' '

# 配置系统服务
[root@k8s-master /data/s0/kubernetes/k8s]$ vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/data/s0/kubernetes/k8s/kube-apiserver.conf
ExecStart=/data/s0/kubernetes/k8s/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

# 启动kube-apiserver服务
[root@k8s-master /data/s0/kubernetes/k8s]$ systemctl start kube-apiserver

5.2.1 服务启动错误

• 错误1：Error: parse error on line 1, column 83: extraneous or missing " in quoted-field’

  修改 token.csv,角色：system:node-bootstrapper 去掉引号

• 错误2：Could not construct pre-rendered responses for ServiceAccountIssuerDiscovery endpoints. Endpoints will not be enabled.

  –service-account-issuer=https://kubernetes.default.svc.cluster.local

  –service-account-signing-key-file=/root/ca/k8s/ca-key.pem

• 错误3：Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/

  服务关闭后，再次重启，log文件错误，不影响使用，未处理。

5.3 部署kube-controller-manager

# 配置文件
[root@k8s-master /data/s0/kubernetes/k8s]$ vim cfg/kube-controller-manager.conf
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/s0/kubernetes/k8s/logs \
--leader-elect=true \
--kubeconfig=/data/s0/kubernetes/k8s/cfg/kube-controller-manager.kubeconfig \
--bind-address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-signing-cert-file=/root/ca/k8s/ca.pem \
--cluster-signing-key-file=/root/ca/k8s/ca-key.pem  \
--root-ca-file=/root/ca/k8s/ca.pem \
--service-account-private-key-file=/root/ca/k8s/ca-key.pem \
--cluster-signing-duration=87600h0m0s"   #证书过期时间10年

参数说明
•	--kubeconfig：连接apiserver配置文件
•	--leader-elect：当该组件启动多个时，自动选举（HA）
•	--cluster-signing-cert-file/--cluster-signing-key-file：为kubelet颁发证书的CA，与apiserver保持一致


# 生成kube-controller-manager证书
[root@k8s-master /data/s0/kubernetes/k8s]$ cd ~/ca/k8s/
[root@k8s-master ~/ca/k8s]$ vim kube-controller-manager-csr.json
{
	"CN": "system:kube-controller-manager",
	"hosts": [],
	"key": {
		"algo": "rsa",
		"size": 2048
	},
	"names": [{
		"C": "CN",
		"L": "BeiJing",
		"ST": "BeiJing",
		"O": "system:masters",
		"OU": "System"
	}]
}
# 证书生成
[root@k8s-master ~/ca/k8s]$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

#生成kubeconfig文件
[root@k8s-master /data/s0/kubernetes/k8s]$ KUBE_CONFIG="/data/s0/kubernetes/k8s/cfg/kube-controller-manager.kubeconfig"
[root@k8s-master /data/s0/kubernetes/k8s]$ KUBE_APISERVER="https://10.34.x.10:6443"

# 终端执行（4条）
# 将集群及证书信息写入kube-controller-manager的配置文件中
[root@k8s-master /data/s0/kubernetes/k8s]$ kubectl config set-cluster kubernetes \
  --certificate-authority=/root/ca/k8s/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
# 配置 kube-controller-manager 用户的证书和私钥
[root@k8s-master /data/s0/kubernetes/k8s]$ kubectl config set-credentials kube-controller-manager \
  --client-certificate=/root/ca/k8s/kube-controller-manager.pem \
  --client-key=/root/ca/k8s/kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
# 创建上下文，关联 kubernetes集群和 kube-controller-manager用户
[root@k8s-master /data/s0/kubernetes/k8s]$ kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-controller-manager \
  --kubeconfig=${KUBE_CONFIG}
# 切换上下文
[root@k8s-master /data/s0/kubernetes/k8s]$ kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

# 配置系统服务
[root@k8s-master /data/s0/kubernetes/k8s]$ vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/data/s0/kubernetes/k8s/cfg/kube-controller-manager.conf
ExecStart=/data/s0/kubernetes/k8s/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target


# kube-controller-manager服务启动
[root@k8s-master /data/s0/kubernetes/k8s]$ systemctl start kube-controller-manager

5.4 部署kube-scheduler

# 创建配置文件
[root@k8s-master /data/s0/kubernetes/k8s]$ vim ./cfg/kube-scheduler.conf
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/s0/kubernetes/k8s/logs \
--leader-elect \
--kubeconfig=/data/s0/kubernetes/k8s/cfg/kube-scheduler.kubeconfig \
--bind-address=127.0.0.1"

参数说明
•	--kubeconfig：连接apiserver配置文件
•	--leader-elect：当该组件启动多个时，自动选举（HA）

#生成kube-scheduler证书
[root@k8s-master /data/s0/kubernetes/k8s]$ cd ~/ca/k8s
[root@datanode40 ~/ca/k8s]$ vim kube-scheduler-csr.json
{
	"CN": "system:kube-scheduler",
	"hosts": [],
	"key": {
		"algo": "rsa",
		"size": 2048
	},
	"names": [{
		"C": "CN",
		"L": "BeiJing",
		"ST": "BeiJing",
		"O": "system:masters",
		"OU": "System"
	}]
}

#生成证书
[root@k8s-master ~/ca/k8s]$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

# kube-scheduler配置文件
[root@k8s-master ~/ca/k8s]$ KUBE_CONFIG="/data/s0/kubernetes/k8s/cfg/kube-scheduler.kubeconfig"
[root@k8s-master ~/ca/k8s]$ KUBE_APISERVER="https://10.34.x.10:6443"


#终端执行（4条）
[root@k8s-master ~/ca/k8s]$ kubectl config set-cluster kubernetes \
  --certificate-authority=/root/ca/k8s/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}

[root@k8s-master ~/ca/k8s]$ kubectl config set-credentials kube-scheduler \
  --client-certificate=/root/ca/k8s/kube-scheduler.pem \
  --client-key=/root/ca/k8s/kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CO