K8S二进制部署
K8S网络规划：
API server是整个集群管理的大脑。
主控节点和运算节点只是逻辑上的概念，物理上完全可以布在一台服务器上。
安装
集群规划
| IP | 节点 |
|---|---|
| 10.4.7.11 | hdss7-11.host.com |
| 10.4.7.12 | hdss7-12.host.com |
| 10.4.7.21 | hdss7-21.host.com |
| 10.4.7.22 | hdss7-22.host.com |
| 10.4.7.200 | hdss7-200.host.com |
环境初始化
这里使用了ansible做批量管理，ansible简单配置可以参考文章：centos7安装ansible并简单设置k8s集群节点hosts文件
关闭并禁用所有机器防火墙：
ansible all -m shell -a "systemctl stop firewalld && systemctl disable firewalld"
关闭selinux
ansible all -m shell -a "rm -rf /etc/selinux/config"
echo "SELINUX=disabled
SELINUXTYPE=targeted
" >> /root/config
ansible all -m copy -a "src=/root/config dest=/etc/selinux/config owner=root mode=644"
安装工具
ansible all -m shell -a "yum install -y epel-release"
ansible all -m shell -a "yum install vim wget net-tools telnet tree nmap sysstat lrzsz dos2unix bind-utils ntp -y"
时间同步
ansible all -m shell -a "systemctl enable ntpd && systemctl start ntpd"
ansible all -m shell -a "timedatectl set-timezone Asia/Shanghai && timedatectl set-ntp yes && ntpq -p"
timedatectl set-timezone Asia/Shanghai //更改时区
timedatectl set-ntp yes //启用ntp同步
ntpq -p //同步时间
重启
ansible all -m shell -a "reboot"
各节点设置
设置主机名并安装相关工具
[root@hdss7-11 ~]# hostnamectl set-hostname hdss7-11.host.com
安装bind9
[root@hdss7-11 ~]# yum install bind -y
[root@hdss7-11 ~]# rpm -qa bind
[root@hdss7-11 ~]# vim /etc/named.conf
bind语法非常严格，每一行末尾必须加上分号;
options {
listen-on port 53 { 10.4.7.11; };
allow-query { any; };
forwarders { 10.4.7.254; };
recursion yes; /*表示采用递归算法提供对dns查询 */
dnssec-enable no;
dnssec-validation no;
[root@hdss7-11 ~]# vim /etc/named.rfc1912.zones
zone "host.com" IN {
type master;
file "host.com.zone";
allow-update { 10.4.7.11; };
};
zone "od.com" IN {
type master;
file "od.com.zone";
allow-update { 10.4.7.11; };
};
添加/var/named/host.com.zone配置文件
[root@hdss7-11 ~]# vim /var/named/host.com.zone
$ORIGIN host.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.host.com. dnsadmin.host.com. (
2020080101 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.host.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
HDSS7-11 A 10.4.7.11
HDSS7-12 A 10.4.7.12
HDSS7-21 A 10.4.7.21
HDSS7-22 A 10.4.7.22
HDSS7-200 A 10.4.7.200
添加/var/named/host.od.zone配置文件，注意下面的dns与dnsadmin后面的一定是od
[root@hdss7-11 ~]# vim /var/named/od.com.zone
$ORIGIN od.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.od.com. dnsadmin.od.com. (
2020080101; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.od.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
# 检查配置文件是否正确，正确的话不会弹出任何内容
[root@hdss7-11 ~]# named-checkconf
[root@hdss7-11 ~]# systemctl status named
# 启动named服务
[root@hdss7-11 ~]# systemctl start named
[root@hdss7-11 ~]# systemctl enable named
# 检查named服务是否运行在53端口上
[root@hdss7-11 ~]# netstat -nltp | grep 53
tcp 0 0 172.26.91.11:53 0.0.0.0:* LISTEN 11642/named
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 11642/named
检查域名是否正常解析
[root@hdss7-11 ~]# dig -t A hdss7-21.host.com @10.4.7.11 +short
10.4.7.21
[root@hdss7-11 ~]# dig -t A hdss7-200.host.com @10.4.7.11 +short
10.4.7.200
修改/etc/resolv.conf文件，;是注释的意思，增加search host.com，并修改nameserver参数。
主机域用短域名，就像hdss7-11.host.com，业务域则用全域名。
[root@hdss7-11 ~]# vim /etc/resolv.conf
options timeout:2 attempts:3 rotate single-request-reopen
; generated by /usr/sbin/dhclient-script
search host.com
; nameserver 100.100.2.138
; nameserver 100.100.2.136
nameserver = 10.4.7.11
修改/etc/resolv.conf之后保存退出，继续ping
[root@hdss7-11 ~]# ping hdss7-200
PING HDSS7-200.host.com (172.26.91.200) 56(84) bytes of data.
64 bytes from 172.26.91.200 (172.26.91.200): icmp_seq=1 ttl=64 time=0.303 ms
64 bytes from 172.26.91.200 (172.26.91.200): icmp_seq=2 ttl=64 time=0.216 ms
[root@hdss7-11 ~]# ping hdss7-11
PING HDSS7-11.host.com (172.26.91.11) 56(84) bytes of data.
64 bytes from node001 (172.26.91.11): icmp_seq=1 ttl=64 time=0.007 ms
64 bytes from node001 (172.26.91.11): icmp_seq=2 ttl=64 time=0.019 ms
在HDSS7-200机器上操作，如果发现404，到这里下载对应的即可https://pkg.cfssl.org
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/bin/cfssl-json
wget wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/bin/cfssl-certinfo
授权
chmod +x /usr/bin/cfssl*
查看
[root@hdss7-200 ~]# which cfssl
/usr/bin/cfssl
[root@hdss7-200 ~]# which cfssl-json
/usr/bin/cfssl-json
[root@hdss7-200 ~]# which cfssl-certinfo
/usr/bin/cfssl-certinfo
证书配置
进入opt创建证书目录
[root@hdss7-200 ~]# cd /opt/
[root@hdss7-200 ~]# mkdir certs
[root@hdss7-200 ~]# cd certs/
[root@hdss7-200 ~]# vim /opt/certs/ca-csr.json
{
"CN": "OldboyEdu",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
],
"ca": {
"expiry": "175200h"
}
}
参数解读：
CN: Common Name，浏览器使用该字段验证网站是否合法，一般写的是域名。非常重要。浏览器使用该字段验证网站是否合法。
C：Country，国家
ST：State，州，省
L：Locality，地区，城市
O：Organization Name，组织名称，公司名称
OU：Organization Uint Name，组织单位名称，公司部门
expiry：过期时间
生成CA证书和私钥
参数：-bare 承载
[root@hdss7-200 certs]# cfssl gencert -initca ca-csr.json | cfssl-json -bare ca
2020/08/01 23:57:35 [INFO] generating a new CA key and certificate from CSR
2020/08/01 23:57:35 [INFO] generate received request
2020/08/01 23:57:35 [INFO] received CSR
2020/08/01 23:57:35 [INFO] generating key: rsa-2048
2020/08/01 23:57:35 [INFO] encoded CSR
2020/08/01 23:57:35 [INFO] signed certificate with serial number 321246988168182570941994946759579004322894455107
[root@hdss7-200 certs]# ll
total 16
-rw-r--r--. 1 root root 997 Aug 1 23:57 ca.csr
-rw-r--r--. 1 root root 324 Aug 1 23:34 ca-csr.json
-rw-------. 1 root root 1679 Aug 1 23:57 ca-key.pem
-rw-r--r--. 1 root root 1346 Aug 1 23:57 ca.pem
重点的两个文件
# ca-key.pem
-rw-------. 1 root root 1679 Aug 1 23:57 ca-key.pem
# ca.pem：私钥，nginx里面配的
-rw-r--r--. 1 root root 1346 Aug 1 23:57 ca.pem
部署docker环境
部署docker到以下机器
hdss7-200.host.com，hdss7-21.host.com，hdss7-22.host.com
下载docker
[root@hdss7-200 ~]# curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
创建docker相关配置文件
[root@hdss7-200 ~]# mkdir /etc/docker
[root@hdss7-200 ~]# mkdir -p /data/docker
创建/etc/docker/daemon.json配置文件
[root@hdss7-200 ~]# vim /etc/docker/daemon.json
{
"graph": "/data/docker",
"storage-driver": "overlay2",
"insecure-registries": ["registry.access.redhat.com", "quay.io", "harbor.od.com"],
"registry-mirrors": ["https://q2gr04ke.mirror.aliyuncs.com"],
"bip": "172.7.200.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
注意：三台机器的"bip": "172.7.xx.1/24"要填写所在机器的IP，比如172.7.200.1\24是10.4.7.200主机的配置。
在其他两台机器上也是同样的操作，"bip": "172.7.xx.1/24"需要改。
部署docker私有镜像仓库（harbor）
hdss7-200.host.com上
下载软件二进制包并解压
建议选择1.7.5以上的版本
[root@hdss7-200 ~]# cd /opt/
[root@hdss7-200 opt]# mkdir src
[root@hdss7-200 opt]# cd src/
可以先用阿里云按量付费进行下载，然后提升带宽，之后在7-200等虚拟机上使用scp再从阿里云服务器进行下载，可大大节省时间。
[root@hdss7-21 src]# wget https://storage.googleapis.com/harbor-releases/release-1.8.0/harbor-offline-installer-v1.8.3.tgz
解压
[root@hdss7-21 src]# tar xf harbor-offline-installer-v1.8.3.tgz -C /opt/
[root@hdss7-200 opt]# ll
drwxr-xr-x. 2 root root 71 Aug 2 11:57 certs
drwx--x--x. 4 root root 28 Aug 2 13:00 containerd
drwxr-xr-x. 2 root root 100 Aug 2 13:51 harbor
drwxr-xr-x. 2 root root 49 Aug 2 13:48 src
# 指定版本号
[root@hdss7-200 opt]# mv harbor/ harbor-v1.8.3
# 生成一个软链接，便于将来升级
[root@hdss7-200 opt]# ln -s /opt/harbor-v1.8.3/ /opt/harbor
查看
[root@hdss7-200 opt]# cd harbor
[root@hdss7-200 harbor]# ll
total 569632
-rw-r--r--. 1 root root 583269670 Sep 16 2019 harbor.v1.8.3.tar.gz
-rw-r--r--. 1 root root 4519 Sep 16 2019 harbor.yml
-rwxr-xr-x. 1 root root 5088 Sep 16 2019 install.sh
-rw-r--r--. 1 root root 11347 Sep 16 2019 LICENSE
-rwxr-xr-x. 1 root root 1654 Sep 16 2019 prepare
编辑harbor.yml，必须要找到对应的配置参数，修改该参数，不能自己手动添加，否则会报错。
[root@hdss7-200 harbor]# vim harbor.yml
hostname: harbor.od.com
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 180
# 生产环境中要把密码改成足够复杂的密码
harbor_admin_password: Harbor12345
# log级别，自己调
# rotate_count：滚动的数量
data_volume: /data/harbor
location: /data/harbor/logs
保存退出，创建对应目录
[root@hdss7-200 harbor]# mkdir -p /data/harbor/logs
# 安装docker-compose
[root@hdss7-200 harbor]# yum install docker-compose -y
# 如果报错No package，执行下面这条命令再重新安装
[root@hdss7-200 harbor]# yum install -y epel-release
[root@hdss7-200 harbor]# rpm -qa docker-compose
docker-compose-1.18.0-4.el7.noarch
[root@hdss7-200 harbor]# ./install.sh
✔ ----Harbor has been installed and started successfully.----
Now you should be able to visit the admin portal at http://harbor.od.com .
For more details, please visit https://github.com/goharbor/harbor .
[root@hdss7-200 harbor]# docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------
harbor-core /harbor/start.sh Up
harbor-db /entrypoint.sh postgres Up 5432/tcp
harbor-jobservice /harbor/start.sh Up
harbor-log /bin/sh -c /usr/local/bin/ ... Up 127.0.0.1:1514->10514/tcp
harbor-portal nginx -g daemon off; Up 80/tcp
nginx nginx -g daemon off; Up 0.0.0.0:180->80/tcp
redis docker-entrypoint.sh redis ... Up 6379/tcp
registry /entrypoint.sh /etc/regist ... Up 5000/tcp
registryctl /harbor/start.sh Up
Nginx配置
[root@hdss7-200 harbor]# yum install nginx -y
[root@hdss7-200 harbor]# vim /etc/nginx/conf.d/harbor.od.com.conf
server {
listen 80;
server_name harbor.od.com;
# 镜像最大大小1个G
client_max_body_size 1000m;
location / {
proxy_pass http://127.0.0.1:180;
}
}
[root@hdss7-200 harbor]# systemctl start nginx && systemctl enable nginx
修改hdss7-11机器上的/var/named/od.com.zone文件
[root@hdss7-11 ~]# vim /var/named/od.com.zone
$ORIGIN od.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.od.com. dnsadmin.od.com. (
2020080102; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.od.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
harbor A 10.4.7.200
修改如下
将2020080101最后的数字1前滚为数字2（serial前滚一个序号）
最下面增加：
harbor A 10.4.7.200
[root@hdss7-11 ~]# systemctl restart named
[root@hdss7-11 ~]# dig -t A harbor.od.com +short
10.4.7.200
Windows配置
打开控制面板——网络和 Internet——网络和共享中心——更改适配器设置
然后打开Windows的终端，ping我们的主机。
C:\Users\huan>ping hdss7-11.host.com
正在 Ping HDSS7-11.host.com [10.4.7.11] 具有 32 字节的数据:
来自 10.4.7.11 的回复: 字节=32 时间<1ms TTL=64
来自 10.4.7.11 的回复: 字节=32 时间<1ms TTL=64
来自 10.4.7.11 的回复: 字节=32 时间<1ms TTL=64
10.4.7.11 的 Ping 统计信息:
数据包: 已发送 = 3，已接收 = 3，丢失 = 0 (0% 丢失)，
往返行程的估计时间(以毫秒为单位):
最短 = 0ms，最长 = 0ms，平均 = 0ms
如果发现无法访问harbor，报错如下
[root@hdss7-200 harbor]# curl harbor.od.com
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.16.1</center>
</body>
</html>
检查nginx日志发现：
2020/08/02 16:25:51 [crit] 21757#0: *1 connect() to 127.0.0.1:180 failed (13: Permission denied) while connecting to upstream, client: 10.4.7.1, server: harbor.od.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:180/favicon.ico", host: "harbor.od.com", referrer: "http://harbor.od.com/"
注意permission denied，请将/etc/selinux/config 文件将SELINUX=enforcing改为SELINUX=disabled并重启200这台机器。
访问页面
如果出现502错误，并且确认上面的配置全部正确，那么请到harbor的目录下重新启动docker-compose相关的容器
访问页面：harbor.od.com，输入admin和密码Harbor12345，新建项目——公开，项目名称为public。
public对应的repository，而不是用户名。
docker的仓库不论是公有还是私有的，默认应该走https协议，
```bash
[root@hdss7-200 ~]# cat /etc/docker/daemon.json
{
"insecure-registries": ["registry.access.redhat.com", "quay.io", "harbor.od.com"],
}
可以看到在这里配置了一个"harbor.od.com"，如果是在生产环境，可以配置证书等，使其通过https访问。
因为这里是演示和学习环境，假的一个域名，只需要访问80端口即可。
下载nginx镜像
[root@hdss7-200 ~]# docker pull nginx:1.7.9
docker pull xxx == docker pull docker.io/library/xxx
[root@hdss7-200 ~]# docker images | grep 1.7.9
nginx 1.7.9 84581e99d807 5 years ago 91.7MB
[root@hdss7-200 ~]# docker tag 84581e99d807 harbor.od.com/public/nginx:v1.7.9
[root@hdss7-200 ~]# docker push harbor.od.com/public/nginx:v1.7.9
The push refers to repository [harbor.od.com/public/nginx]
5f70bf18a086: Preparing
4b26ab29a475: Preparing
ccb1d68e3fb7: Preparing
e387107e2065: Preparing
63bf84221cce: Preparing
e02dce553481: Waiting
dea2e4984e29: Waiting
denied: requested access to the resource is denied
# 可以看到这里报错，因为我们还没有等到到harbor仓库
# 用户名admin，passowrd: Harbor12345，可以在harbor.yml中查看
[root@hdss7-200 ~]# docker login harbor.od.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
# 再次push
[root@hdss7-200 harbor]# docker push harbor.od.com/public/nginx:v1.7.9
........
v1.7.9: digest: sha256:b1f5935eb2e9e2ae89c0b3e2e148c19068d91ca502e857052f14db230443e4c2 size: 3012
安装部署etcd
| 主机名 | 角色 | IP |
|---|---|---|
| HDSS7-12.host.com | etcd leader | 10.4.7.12 |
| HDSS7-21.host.com | etcd follower | 10.4.7.21 |
| HDSS7-21.host.com | etcd follower | 10.4.7.22 |
注意：这里部署文档以HDSS7-12.host.com主机为例，另外两台主机安装部署方法类似
创建基于根证书的config配置文件，所在路径/opt/certs/ca-config.json
[root@hdss7-200 ~]# vim /opt/certs/ca-config.json
{
"signing": {
"default": {
"expiry": "175200h"
},
"profiles": {
"server": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
创建etcd-peer-csr.json配置文件
[root@hdss7-200 ~]# vim /opt/certs/etcd-peer-csr.json
{
"CN": "k8s-etcd",
"hosts": [
"10.4.7.11",
"10.4.7.12",
"10.4.7.21",
"10.4.7.22"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
其中hosts主机包括可能要通信的主机，如果不加上，通信的时候就会报错。
必须把每个IP都填进去，不能用10.4.7.*等方式表示。
手撕证书
[root@hdss7-200 certs]# pwd
/opt/certs
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json | cfssl-json -bare etcd-peer
2020/08/02 18:33:54 [INFO] generate received request
2020/08/02 18:33:54 [INFO] received CSR
2020/08/02 18:33:54 [INFO] generating key: rsa-2048
2020/08/02 18:33:54 [INFO] encoded CSR
2020/08/02 18:33:54 [INFO] signed certificate with serial number 27799819669757823513778695090866250260852383558
2020/08/02 18:33:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@hdss7-200 certs]# ll
total 36
-rw-r--r-- 1 root root 836 Aug 2 18:22 ca-config.json
-rw-r--r--. 1 root root 997 Aug 2 11:57 ca.csr
-rw-r--r--. 1 root root 324 Aug 2 11:34 ca-csr.json
-rw-------. 1 root root 1679 Aug 2 11:57 ca-key.pem
-rw-r--r--. 1 root root 1346 Aug 2 11:57 ca.pem
-rw-r--r-- 1 root root 1062 Aug 2 18:33 etcd-peer.csr
-rw-r--r-- 1 root root 363 Aug 2 18:25 etcd-peer-csr.json
-rw------- 1 root root 1679 Aug 2 18:33 etcd-peer-key.pem
-rw-r--r-- 1 root root 1428 Aug 2 18:33 etcd-peer.pem
7-12机器部署
[root@hdss7-12 src]# useradd -s /sbin/nologin -M etcd
[root@hdss7-12 src]# id etcd
uid=1000(etcd) gid=1000(etcd) groups=1000(etcd)
使用比较稳定的3.1.x版本
[root@hdss7-12 src]# wget https://github.com/etcd-io/etcd/releases/download/v3.1.20/etcd-v3.1.20-linux-amd64.tar.gz
[root@hdss7-12 src]# tar -zxvf etcd-v3.1.20-linux-amd64.tar.gz -C /opt/
[root@hdss7-12 src]# mv /opt/etcd-v3.1.20-linux-amd64/ /opt/etcd-v3.1.20
[root@hdss7-12 src]# ln -s /opt/etcd-v3.1.20/ /opt/etcd
[root@hdss7-12 src]# ll /opt/
total 0
lrwxrwxrwx 1 root root 18 Aug 2 19:04 etcd -> /opt/etcd-v3.1.20/
drwxr-xr-x 3 478493 89939 123 Oct 11 2018 etcd-v3.1.20
drwxr-xr-x 2 root root 45 Aug 2 18:44 src
最重要的两个文件
-rwxr-xr-x 1 478493 89939 16406432 Oct 11 2018 etcd # 启动程序
-rwxr-xr-x 1 478493 89939 14327712 Oct 11 2018 etcdctl # 命令行工具
创建目录
[root@hdss7-12 src]# cd /opt/etcd
[root@hdss7-12 etcd]# mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
从7-200拷贝证书
[root@hdss7-12 etcd]# scp hdss7-200:/opt/certs/ca.pem certs/
[root@hdss7-12 etcd]# scp hdss7-200:/opt/certs/etcd-peer.pem certs/
[root@hdss7-12 etcd]# scp hdss7-200:/opt/certs/etcd-peer-key.pem certs/
注意：私钥一定要保管好，需要万分小心。
创建etcd服务启动脚本
hdss7-12.host.com
vim /opt/etcd/etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-7-12 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://10.4.7.12:2380 \
--listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://10.4.7.12:2380 \
--advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
listen-peer-urls： 内部通信 2380
listen-client-urls：外部通信 2379
quota-backend-bytes 后端配额
文件名和配置一定要保持一致，否则会报错。
更改执行权限等
[root@hdss7-12 etcd]# chmod +x etcd-server-startup.sh
[root@hdss7-12 etcd]# chown -R etcd.etcd /opt/etcd-v3.1.20/
[root@hdss7-12 etcd]# chown -R etcd.etcd /data/etcd/
[root@hdss7-12 etcd]# chown -R etcd.etcd /data/logs/etcd-server/
下载supervisor，这是一个管理后台进程的软件，可以自动拉起挂掉的服务，将etcd启动脚本交给supervisor管理
[root@hdss7-12 etcd]# yum install supervisor -y
[root@hdss7-12 etcd]# systemctl start supervisord && systemctl enable supervisord
编辑supervisor启动文件
[root@hdss7-12 etcd]# vim /etc/supervisord.d/etcd-server.ini
[program:etcd-server-7-12]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected
quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; \'expected\' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in \'capturemode\' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true
[root@hdss7-12 etcd]# supervisorctl update
etcd-server-7-12: added process group
[root@hdss7-12 etcd]# supervisorctl status
etcd-server-7-12 RUNNING pid 1272, uptime 0:00:36
[root@hdss7-12 etcd]# tail -fn 200 /data/logs/etcd-server/etcd.stdout.log
2020-08-02 19:30:46.546707 I | raft: f4a0cb0a765574a8 received MsgVoteResp from f4a0cb0a765574a8 at term 15
2020-08-02 19:30:46.546713 I | raft: f4a0cb0a765574a8 [logterm: 1, index: 3] sent MsgVote request to 5a0ef2a004fc4349 at term 15
2020-08-02 19:30:46.546723 I | raft: f4a0cb0a765574a8 [logterm: 1, index: 3] sent MsgVote request to 988139385f78284 at term 15
2020-08-02 19:30:48.246456 I | raft: f4a0cb0a765574a8 is starting a new election at term 15
2020-08-02 19:30:48.246489 I | raft: f4a0cb0a765574a8 became candidate at term 16
查看服务启动状态，只有2379和2380开启才算启动成功
[root@hdss7-12 etcd]# netstat -nltp | grep etcd
tcp 0 0 10.4.7.12:2379 0.0.0.0:* LISTEN 1273/./etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 1273/./etcd
tcp 0 0 10.4.7.12:2380 0.0.0.0:* LISTEN 1273/./etcd
在其他两台机器上安装，（7.21，7.22）
[root@hdss7-21 ~]# mkdir -p /opt/src
[root@hdss7-22 ~]# mkdir -p /opt/src
拷贝安装包
[root@hdss7-21 ~]# scp root@10.4.7.12:/opt/src/etcd-v3.1.20-linux-amd64.tar.gz /opt/src/
[root@hdss7-22 ~]# scp root@10.4.7.12:/opt/src/etcd-v3.1.20-linux-amd64.tar.gz /opt/src/
解压，重命名，软连接
[root@hdss7-22 src]# mv /opt/etcd-v3.1.20-linux-amd64/ /opt/etcd-v3.1.20
[root@hdss7-22 src]# ln -s /opt/etcd-v3.1.20/ /opt/etcd
[root@hdss7-21 src]# mv /opt/etcd-v3.1.20-linux-amd64/ /opt/etcd-v3.1.20
[root@hdss7-21 src]# ln -s /opt/etcd-v3.1.20/ /opt/etcd
添加etcd用户
[root@hdss7-21 opt]# useradd -s /sbin/nologin -M etcd
[root@hdss7-22 src]# useradd -s /sbin/nologin -M etcd
分别给两台机器添加证书目录
[root@hdss7-22 etcd]# mkdir certs
[root@hdss7-22 certs]# scp hdss7-200:/opt/certs/*.pem .
ca-key.pem
ca.pem
etcd-peer-key.pem
etcd-peer.pem
[root@hdss7-22 certs]# rm -rf ca-key.pem
[root@hdss7-21 certs]# scp hdss7-200:/opt/certs/*.pem .
ca-key.pem
ca.pem
etcd-peer-key.pem
etcd-peer.pem
[root@hdss7-21 certs]# rm -rf ca-key.pem
这里是为了方便直接以*.pem进行匹配，最后删掉多余的pem文件，生产环境慎用。
编辑startup文件，直接从上面拷贝就可以了，需要修改相关参数
[root@hdss7-21 certs]# cd ..
[root@hdss7-21 etcd]# vim etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-7-21 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://10.4.7.21:2380 \
--listen-client-urls https://10.4.7.21:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://10.4.7.21:2380 \
--advertise-client-urls https://10.4.7.21:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
[root@hdss7-21 etcd]# chmod +x etcd-server-startup.sh
在7-22上。
[root@hdss7-22 etcd]# vim etcd-server-startup.sh
#!/bin/sh
./etcd --name etcd-server-7-22 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://10.4.7.22:2380 \
--listen-client-urls https://10.4.7.22:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://10.4.7.22:2380 \
--advertise-client-urls https://10.4.7.22:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
[root@hdss7-22 etcd]# chmod +x etcd-server-startup.sh
创建目录
[root@hdss7-22 etcd]# mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
[root@hdss7-22 etcd]# chown -R etcd.etcd /opt/etcd-v3.1.20/
[root@hdss7-22 etcd]# chown -R etcd.etcd /data/etcd/
[root@hdss7-22 etcd]# chown -R etcd.etcd /data/logs/etcd-server/
[root@hdss7-21 etcd]# mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
[root@hdss7-21 etcd]# chown -R etcd.etcd /opt/etcd-v3.1.20/
[root@hdss7-21 etcd]# chown -R etcd.etcd /data/etcd/
[root@hdss7-21 etcd]# chown -R etcd.etcd /data/logs/etcd-server/
安装supervisor，启动并设置开机自启
[root@hdss7-21 etcd]# yum install supervisor -y
[root@hdss7-22 etcd]# yum install supervisor -y
[root@hdss7-22 etcd]# systemctl start supervisord && systemctl enable supervisord
[root@hdss7-21 etcd]# systemctl start supervisord && systemctl enable supervisord
编写supervisor配置文件，需要将program:etcd-server修改为对应的配置
[root@hdss7-22 etcd]# vim /etc/supervisord.d/etcd-server.ini
[program:etcd-server-7-22]
[root@hdss7-22 etcd]# supervisorctl update
[root@hdss7-21 etcd]# vim /etc/supervisord.d/etcd-server.ini
[program:etcd-server-7-21]
[root@hdss7-21 etcd]# supervisorctl update
查看集群状态
[root@hdss7-21 etcd]# ./etcdctl cluster-health
member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379
member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379
member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379
[root@hdss7-21 etcd]# ./etcdctl member list
988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.22:2379 isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.21:2379 isLeader=false
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=true
安装k8s
部署在10.4.7.21，10.4.7.22节点上
| 主机名 | 角色 | IP |
|---|---|---|
| HDSS7-21.host.com | kube-apiserver | 10.4.7.21 |
| HDSS7-22.host.com | kube-apiserver | 10.4.7.22 |
| HDSS7-11.host.com | 4层负载均衡 | 10.4.7.11 |
| HDSS7-12.host.com | 4层负载均衡 | 10.4.7.12 |
注意：这里10.4.7.12和10.4.7.11使用nginx做4层负载均衡，用keepalived跑一个vip：10.4.7.10，代理两个kube-apiserver，实现高可用
这里部署文档以HDSS7-21.host.com主机为例，另外一台运算节点安装部署方式类似
下载软件，解压，做软链接
HDSS7-21.host.com上：
[root@hdss7-21 opt]# wget https://dl.k8s.io/v1.15.4/kubernetes-server-linux-amd64.tar.gz
[root@hdss7-21 src]# tar -zxvf kubernetes-server-linux-amd64-v1.15.2.tar.gz -C /opt/
[root@hdss7-21 src]# mv /opt/kubernetes/ /opt/kubernetes-v1.15.2
[root@hdss7-21 src]# cd /opt/
[root@hdss7-21 opt]# ln -s kubernetes-v1.15.2/ kubernetes/
[root@hdss7-21 src]# cd /opt/kubernetes
[root@hdss7-21 kubernetes]# ll
total 27184
drwxr-xr-x 2 root root 6 Aug 5 2019 addons
-rw-r--r-- 1 root root 26625140 Aug 5 2019 kubernetes-src.tar.gz
-rw-r--r-- 1 root root 1205293 Aug 5 2019 LICENSES
drwxr-xr-x 3 root root 17 Aug 5 2019 server
# 删除k8s源码
[root@hdss7-21 kubernetes]# rm -rf kubernetes-src.tar.gz
[root@hdss7-21 kubernetes]# cd server/bin/
[root@hdss7-21 bin]# rm -f *.tar
[root@hdss7-21 bin]# rm -f *_tag
[root@hdss7-21 bin]# ll
total 884636
-rwxr-xr-x 1 root root 43534816 Aug 5 2019 apiextensions-apiserver
-rwxr-xr-x 1 root root 100548640 Aug 5 2019 cloud-controller-manager
-rwxr-xr-x 1 root root 200648416 Aug 5 2019 hyperkube
-rwxr-xr-x 1 root root 40182208 Aug 5 2019 kubeadm
-rwxr-xr-x 1 root root 164501920 Aug 5 2019 kube-apiserver
-rwxr-xr-x 1 root root 116397088 Aug 5 2019 kube-controller-manager
-rwxr-xr-x 1 root root 42985504 Aug 5 2019 kubectl
-rwxr-xr-x 1 root root 119616640 Aug 5 2019 kubelet
-rwxr-xr-x 1 root root 36987488 Aug 5 2019 kube-proxy
-rwxr-xr-x 1 root root 38786144 Aug 5 2019 kube-scheduler
-rwxr-xr-x 1 root root 1648224 Aug 5 2019 mounter
签发client证书
在运维主机hdss7-200.host.com上：
创建生成证书签名请求（csr）的json配置文件
etcd集群是server端，apiserver是客户端，所以需要做证书给apiserver。
[root@hdss7-200 certs]# vim /opt/certs/client-csr.json
{
"CN": "k8s-node",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
# 生成证书
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json | cfssl-json -bare client
创建生成证书签名请求（csr）的json配置文件
[root@hdss7-200 certs]# vim /opt/certs/apiserver-csr.json
{
"CN": "k8s-apiserver",
"hosts": [
"127.0.0.1",
"192.168.0.1",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"10.4.7.10",
"10.4.7.21",
"10.4.7.22",
"10.4.7.23"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json | cfssl-json -bare apiserver
"10.4.7.10"是vip，后续会讲到其作用
[root@hdss7-200 certs]# ll
total 68
-rw-r--r-- 1 root root 1249 Aug 2 22:01 apiserver.csr
-rw-r--r-- 1 root root 567 Aug 2 21:58 apiserver-csr.json
-rw------- 1 root root 1675 Aug 2 22:01 apiserver-key.pem
-rw-r--r-- 1 root root 1598 Aug 2 22:01 apiserver.pem
-rw-r--r-- 1 root root 836 Aug 2 18:22 ca-config.json
-rw-r--r--. 1 root root 997 Aug 2 11:57 ca.csr
-rw-r--r--. 1 root root 324 Aug 2 11:34 ca-csr.json
-rw-------. 1 root root 1679 Aug 2 11:57 ca-key.pem
-rw-r--r--. 1 root root 1346 Aug 2 11:57 ca.pem
-rw-r--r-- 1 root root 993 Aug 2 21:54 client.csr
-rw-r--r-- 1 root root 283 Aug 2 21:48 client-csr.json
-rw------- 1 root root 1675 Aug 2 21:54 client-key.pem
-rw-r--r-- 1 root root 1367 Aug 2 21:54 client.pem
-rw-r--r-- 1 root root 1062 Aug 2 18:33 etcd-peer.csr
-rw-r--r-- 1 root root 363 Aug 2 18:25 etcd-peer-csr.json
-rw------- 1 root root 1679 Aug 2 18:33 etcd-peer-key.pem
-rw-r--r-- 1 root root 1428 Aug 2 18:33 etcd-peer.pem
在/opt/kubernetes/server/bin目录下创建cert目录，注意是cert不是certs
[root@hdss7-21 bin]# mkdir cert
[root@hdss7-21 bin]# cd cert
[root@hdss7-21 cert]# scp hdss7-200:/opt/certs/ca-key.pem .
[root@hdss7-21 cert]# scp hdss7-200:/opt/certs/ca.pem .
[root@hdss7-21 cert]# scp hdss7-200:/opt/certs/client.pem .
[root@hdss7-21 cert]# scp hdss7-200:/opt/certs/client-key.pem .
[root@hdss7-21 cert]# scp hdss7-200:/opt/certs/apiserver.pem .
[root@hdss7-21 cert]# scp hdss7-200:/opt/certs/apiserver-key.pem .
在7-22上
[root@hdss7-22 cert]# scp hdss7-21:/opt/kubernetes/server/bin/cert/* .
创建conf文件
[root@hdss7-21 bin]# mkdir /opt/kubernetes-v1.15.2/server/bin/conf
[root@hdss7-21 bin]# cd conf
[root@hdss7-21 conf]# vim audit.yaml
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don\'t generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Log pod changes at RequestResponse level
- level: RequestResponse
resources:
- group: ""
# Resource "pods" doesn\'t match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources: ["pods"]
# Log "pods/log", "pods/status" at Metadata level
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
# Don\'t log requests to a configmap called "controller-leader"
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"]
# Don\'t log watch requests by the "system:kube-proxy" on endpoints or services
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
resources: ["endpoints", "services"]
# Don\'t log authenticated requests to certain non-resource URL paths.
- level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version"
# Log the request body of configmap changes in kube-system.
- level: Request
resources:
- group: "" # core API group
resources: ["configmaps"]
# This rule only applies to resources in the "kube-system" namespace.
# The empty string "" can be used to select non-namespaced resources.
namespaces: ["kube-system"]
# Log configmap and secret changes in all other namespaces at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"]
# Log all other resources in core and extensions at the Request level.
- level: Request
resources:
- group: "" # core API group
- group: "extensions" # Version of group should NOT be included.
# A catch-all rule to log all other requests at the Metadata level.
- level: Metadata
# Long-running requests like watches that fall under this rule will not
# generate an audit event in RequestReceived.
omitStages:
- "RequestReceived"
在conf的上级目录也就是/opt/kubernetes/server/bin目录编辑apiserver启动脚本配置文件
[root@hdss7-21 bin]# vim /opt/kubernetes/server/bin/kube-apiserver.sh
#!/bin/bash
./kube-apiserver \
--apiserver-count 2 \
--insecure-port 8080 \
--secure-port 6443 \
--audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log \
--audit-policy-file ./conf/audit.yaml \
--authorization-mode RBAC \
--client-ca-file ./cert/ca.pem \
--requestheader-client-ca-file ./cert/ca.pem \
--enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \
--etcd-cafile ./cert/ca.pem \
--etcd-certfile ./cert/client.pem \
--etcd-keyfile ./cert/client-key.pem \
--etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
--service-account-key-file ./cert/ca-key.pem \
--service-cluster-ip-range 192.168.0.0/16 \
--service-node-port-range 3000-29999 \
--target-ram-mb=1024 \
--kubelet-client-certificate ./cert/client.pem \
--kubelet-client-key ./cert/client-key.pem \
--log-dir /data/logs/kubernetes/kube-apiserver \
--tls-cert-file ./cert/apiserver.pem \
--tls-private-key-file ./cert/apiserver-key.pem \
--v 2
授予执行权限
[root@hdss7-21 bin]# chmod +x kube-apiserver.sh
配置apiserver.ini文件
[root@hdss7-21 bin]# vim /etc/supervisord.d/kube-apiserver.ini
[program:kube-apiserver-7-21]
command=/opt/kubernetes/server/bin/kube-apiserver.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; \'expected\' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in \'capturemode\' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
创建目录
[root@hdss7-21 bin]# mkdir -p /data/logs/kubernetes/kube-apiserver
[root@hdss7-21 bin]# supervisorctl update
kube-apiserver-7-21: added process group
接下来在7.22上安装kubernetes
[root@hdss7-21 ~]# scp hdss7-21:/opt/src/kubernetes-server-linux-amd64-v1.15.2.tar.gz /opt/src
[root@hdss7-21 ~]# tar -zxvf kubernetes-server-linux-amd64-v1.15.2.tar.gz -C /opt/
[root@hdss7-21 ~]# mv kubernetes/ kubernetes-v1.15.2
[root@hdss7-21 ~]# ln -s /opt/kubernetes-v1.15.2/ /opt/kubernetes
[root@hdss7-21 ~]# cd /opt/kubernetes
[root@hdss7-22 kubernetes]# rm -rf kubernetes-src.tar.gz
[root@hdss7-22 kubernetes]# cd server/bin/
[root@hdss7-22 bin]# rm -rf *.tar
[root@hdss7-22 bin]# rm -rf *_tag
[root@hdss7-22 bin]# mkdir cert
[root@hdss7-22 bin]#
[root@hdss7-22 bin]# mkdir conf
[root@hdss7-22 cert]# scp hdss7-21:/opt/kubernetes/server/bin/cert/*.pem .
[root@hdss7-22 cert]# ll
total 24
-rw------- 1 root root 1675 Aug 3 21:26 apiserver-key.pem
-rw-r--r-- 1 root root 1598 Aug 3 21:26 apiserver.pem
-rw------- 1 root root 1679 Aug 3 21:26 ca-key.pem
-rw-r--r-- 1 root root 1346 Aug 3 21:26 ca.pem
-rw------- 1 root root 1675 Aug 3 21:26 client-key.pem
-rw-r--r-- 1 root root 1367 Aug 3 21:26 client.pem
配置conf/audit.yaml等文件
[root@hdss7-22 cert]# cd ../conf/
[root@hdss7-22 conf]# scp hdss7-21:/opt/kubernetes/server/bin/conf/audit.yaml .
[root@hdss7-22 cert]# cd ../conf/
[root@hdss7-22 conf]# scp hdss7-21:/opt/kubernetes/server/bin/kube-apiserver.sh ..
[root@hdss7-22 bin]# scp hdss7-21:/etc/supervisord.d/kube-apiserver.ini /etc/supervisord.d/
注意把第一行的IP地址改为对应主机的IP地址
[program:kube-apiserver-7-22]
创建日志目录，启动并查看状态
[root@hdss7-22 supervisord.d]# mkdir -p /data/logs/kubernetes/kube-apiserver
[root@hdss7-22 supervisord.d]# supervisorctl update
kube-apiserver-7-22: added process group
[root@hdss7-22 supervisord.d]# supervisorctl status
etcd-server-7-22 RUNNING pid 1303, uptime 1 day, 1:24:15
kube-apiserver-7-22 RUNNING pid 2420, uptime 0:01:58
[root@hdss7-21 bin]# netstat -nltp | grep kube-api
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 2729/./kube-apiserv
tcp6 0 0 :::6443 :::*
利用L4 10.4.7.10：7443端口去反载10.4.7.21和10.4.7.22apiserver的6443端口
在7.11和7.12上安装nginx，并把下面的内容写入到两台机器对应文件的最后，不能放在http里面。
[root@hdss7-11 ~]# yum -y install nginx
[root@hdss7-12 ~]# yum -y install nginx
vim /etc/nginx/nginx.conf
# 将下面这段粘贴到最后
stream {
# kubernetes api-server ip地址以及https端口
upstream kube-apiserver {
server 10.4.7.21:6443 max_fails=3 fail_timeout=30s;
server 10.4.7.22:6443 max_fails=3 fail_timeout=30s;
}
# 监听7443端口，将其接收的流量转发至指定proxy_pass
server {
listen 7443;
proxy_connect_timeout 2s;
proxy_timeout 900s;
proxy_pass kube-apiserver;
}
}
检查配置，分别启动两台机器的nginx并设置为开机自启
[root@hdss7-12 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@hdss7-12 ~]# systemctl start nginx && systemctl enable nginx
# 在7-11上也要启动
[root@hdss7-11 ~]# systemctl start nginx && systemctl enable nginx
分别在两台机器上安装keepalived及配置相关文件，保证服务的高可用和节点宕机后的自动漂移
[root@hdss7-12 ~]# vim /etc/keepalived/check_port.sh
#!/bin/bash
# keepalived 监控端口脚本
# 使用方法：
# 在keepalived的配置文件中
# vrrp_script check_port {#创建一个vrrp_script脚本,检查配置
# script "/etc/keepalived/check_port.sh 6379" #配置监听的端口
# interval 2 #检查脚本的频率,单位（秒）
# }
CHK_PORT=$1
if [ -n "$CHK_PORT" ];then
PORT_PROCESS=`ss -lnt|grep $CHK_PORT|wc -l`
if [ $PORT_PROCESS -eq 0 ];then
echo "Port $CHK_PORT Is Not Used,End."
exit 1
fi
else
echo "Check Port Cant Be Empty!"
fi
[root@hdss7-12 ~]# chmod +x /etc/keepalived/check_port.sh
keepavlied主节点配置文件
[root@hdss7-11 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 10.4.7.11
}
vrrp_script chk_nginx {
# 调用脚本检测nginx监听的7443端口是否存在
script "/etc/keepalived/check_port.sh 7443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 251
priority 100
advert_int 1
# 当前主机IP
mcast_src_ip 10.4.7.11
# 非抢占式，重大失误的话算生产事故
nopreempt
# 高可用认证
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_nginx
}
# 虚拟IP
virtual_ipaddress {
10.4.7.10
}
}
keepavlied从节点配置文件
! Configuration File for keepalived
global_defs {
router_id 10.4.7.12
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 7443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 251
mcast_src_ip 10.4.7.12
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.4.7.10
}
}
设置开机自启并启动
[root@hdss7-12 ~]# systemctl start keepalived && systemctl enable keepalived
[root@hdss7-11 ~]# systemctl start keepalived && systemctl enable keepalived
[root@hdss7-11 ~]# systemctl status keepalived
[root@hdss7-12 ~]# systemctl status keepalived
[root@hdss7-12 ~]# netstat -nltp | grep 7443
tcp 0 0 0.0.0.0:7443 0.0.0.0:* LISTEN 2590/nginx: master
[root@hdss7-11 ~]# netstat -nltp | grep 7443
tcp 0 0 0.0.0.0:7443 0.0.0.0:* LISTEN 2461/nginx: master
[root@hdss7-11 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:29:91:2b brd ff:ff:ff:ff:ff:ff
inet 10.4.7.11/24 brd 10.4.7.255 scope global eth0
valid_lft forever preferred_lft forever
inet 10.4.7.10/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a767:dffd:3c13:2f24/64 scope link
valid_lft forever preferred_lft forever
# 如果没有看到这个，需要重启一下nginx，然后再次查看
# inet 10.4.7.10/32 scope global eth0
[root@hdss7-11 ~]# systemctl restart nginx
[root@hdss7-12 ~]# systemctl restart nginx
部署controller-manager
[root@hdss7-21 ~]# vim /opt/kubernetes/server/bin/kube-controller-manager.sh
#!/bin/sh
./kube-controller-manager \
--cluster-cidr 172.7.0.0/16 \
--leader-elect true \
--log-dir /data/logs/kubernetes/kube-controller-manager \
--master http://127.0.0.1:8080 \
--service-account-private-key-file ./cert/ca-key.pem \
--service-cluster-ip-range 192.168.0.0/16 \
--root-ca-file ./cert/ca.pem \
--v 2
配置参数说明
| 参数 | 说明 |
|---|---|
| --cluster-cidr | 集群中Pod的CIDR范围 |
| --master | kubernetes api server的地址，将会覆盖kubeconfig设置的值 |
| --service-cluster-ip-range | 集群service的cidr范围，需要--allocate-node-cidrs设置为true |
| --leader-elect | 多个master情况设置为true保证高可用，进行leader选举 |
| --leader-elect-lease-duration duration | 当leader-elect设置为true生效，选举过程中非leader候选等待选举的时间间隔（default 15s） |
| --leader-elect-renew-deadline duration | eader选举过程中在停止leading，再次renew时间间隔，小于或者等于leader-elect-lease-duration duration，也是leader-elect设置为true生效（default 10s） |
| --leader-elect-retry-period duration | 当leader-elect设置为true生效，获取leader或者重新选举的等待间隔（default 2s） |
创建相应的目录
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-controller-manager
[root@hdss7-21 ~]# chmod +x /opt/kubernetes/server/bin/kube-controller-manager.sh
在7-22上也是同样的操作
[root@hdss7-22 ~]# vim /opt/kubernetes/server/bin/kube-controller-manager.sh
[root@hdss7-22 ~]# mkdir -p /data/logs/kubernetes/kube-controller-manager
[root@hdss7-22 ~]# chmod +x /opt/kubernetes/server/bin/kube-controller-manager.sh
创建启动配置脚本
[root@hdss7-22 ~]# vim /etc/supervisord.d/kube-conntroller-manager.ini
[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-conntroller-manager.ini
记得[program:kube-controller-manager-7-xx]要改成对应所在的机器
[program:kube-controller-manager-7-21]
command=/opt/kubernetes/server/bin/kube-controller-manager.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; \'expected\' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in \'capturemode\' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true
分别在两台机器上执行supervisorctl命令
[root@hdss7-21 bin]# pwd
/opt/kubernetes/server/bin
[root@hdss7-21 bin]# supervisorctl update
kube-controller-manager-7-21: added process group
[root@hdss7-22 ~]# cd /opt/kubernetes/server/bin/
[root@hdss7-22 bin]# supervisorctl update
kube-controller-manager-7-22: added process group
检查状态
[root@hdss7-21 bin]# supervisorctl status
etcd-server-7-21 RUNNING pid 4299, uptime 2 days, 1:11:26
kube-apiserver-7-21 RUNNING pid 4300, uptime 23:48:49
kube-controller-manager-7-21 RUNNING pid 4298, uptime 0:00:35
[root@hdss7-22 bin]# supervisorctl update
kube-controller-manager-7-22: added process group
[root@hdss7-22 bin]# supervisorctl status
etcd-server-7-22 RUNNING pid 1303, uptime 2 days, 1:10:46
kube-apiserver-7-22 RUNNING pid 2420, uptime 23:50:02
kube-controller-manager-7-22 RUNNING pid 4030, uptime 0:02:20
创建kube-scheduler.sh脚本
[root@hdss7-22 bin]# vim /opt/kubernetes/server/bin/kube-scheduler.sh
#!/bin/sh
./kube-scheduler \
--leader-elect \
--log-dir /data/logs/kubernetes/kube-scheduler \
--master http://127.0.0.1:8080 \
--v 2
[root@hdss7-22 bin]# vim /opt/kubernetes/server/bin/kube-scheduler.sh
[root@hdss7-22 bin]# chmod +x /opt/kubernetes/server/bin/kube-scheduler.sh
[root@hdss7-21 bin]# chmod +x /opt/kubernetes/server/bin/kube-scheduler.sh
创建启动配置文件
[root@hdss7-21 bin]# vim /etc/supervisord.d/kube-scheduler.ini
[program:kube-scheduler-7-22]
command=/opt/kubernetes/server/bin/kube-scheduler.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; \'expected\' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in \'capturemode\' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true
创建对应的目录，启动并查看状态
[root@hdss7-22 bin]# vim /etc/supervisord.d/kube-scheduler.ini
[root@hdss7-22 bin]# mkdir -p /data/logs/kubernetes/kube-scheduler
[root@hdss7-22 bin]# supervisorctl update
kube-scheduler-7-22: added process group
supervisorctl status
[root@hdss7-21 bin]# supervisorctl update
kube-scheduler-7-21: added process group
[root@hdss7-21 bin]# mkdir -p /data/logs/kubernetes/kube-scheduler
[root@hdss7-21 bin]# supervisorctl status
[root@hdss7-22 bin]# supervisorctl status
现在，主备节点的所有服务都部署完了，可以检查集群的健康状态了。 先创建一个kubectl的软链接，方便后续操作。
[root@hdss7-22 bin]# ln -s /opt/kubernetes/server/bin/kubectl /usr/bin/kubectl
[root@hdss7-21 bin]# ln -s /opt/kubernetes/server/bin/kubectl /usr/bin/kubectl
[root@hdss7-21 bin]# which kubectl
/usr/bin/kubectl
[root@hdss7-22 bin]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-2 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
etcd-0 Healthy {"health": "true"}
部署node节点服务
在200机器上创建证书，要把所有可能的运算节点添加到hosts里面
[root@hdss7-200 certs]# pwd
/opt/certs
[root@hdss7-200 certs]# vim kubelet-csr.json
{
"CN": "k8s-kubelet",
"hosts": [
"127.0.0.1",
"10.4.7.10",
"10.4.7.21",
"10.4.7.22",
"10.4.7.23",
"10.4.7.24",
"10.4.7.25",
"10.4.7.26",
"10.4.7.27",
"10.4.7.28"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
手撕生成证书命令
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssl-json -bare kubelet
2020/08/04 22:07:19 [INFO] generate received request
2020/08/04 22:07:19 [INFO] received CSR
2020/08/04 22:07:19 [INFO] generating key: rsa-2048
2020/08/04 22:07:19 [INFO] encoded CSR
2020/08/04 22:07:19 [INFO] signed certificate with serial number 32307083622542829241342480939899677009596013704
2020/08/04 22:07:19 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
切换到21和22机器上，拷贝证书
[root@hdss7-22 bin]# cd cert/
[root@hdss7-22 cert]# scp hdss7-200:/opt/certs/kube*.pem .
kubelet-key.pem
kubelet.pem
[root@hdss7-21 bin]# cd cert/
[root@hdss7-21 cert]# scp hdss7-200:/opt/certs/kube*.pem .
kubelet-key.pem
kubelet.pem
分发证书到7-21和7-22上
[root@hdss7-22 cert]# cd ../conf/
[root@hdss7-22 conf]# ll
total 4
-rw-r--r-- 1 root root 2224 Aug 3 21:35 audit.yaml
第一步
# 指定根证书和api-server的vip
[root@hdss7-22 conf]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/cert/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=kubelet.kubeconfig
# 会有结果输出如下
Cluster "myk8s" set.
[root@hdss7-22 conf]# ll
-rw-r--r-- 1 root root 2224 Aug 3 21:35 audit.yaml
-rw------- 1 root root 1986 Aug 4 22:16 kubelet.kubeconfig
查看一下新增加的文件
[root@hdss7-22 conf]# cat kubelet.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR0akNDQXA2Z0F3SUJBZ0lVT0VVMmtZYWpmTDlPZWI0TDAwQlU3WXhHSVVNd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjJKbGFXcHBibWN4RURBT0JnTlZCQWNUQjJKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyOXNaREVNTUFvR0ExVUVDeE1EYjNCek1SSXdFQVlEVlFRREV3bFBiR1JpCmIzbEZaSFV3SGhjTk1qQXdPREF5TURNMU16QXdXaGNOTkRBd056STRNRE0xTXpBd1dqQmhNUXN3Q1FZRFZRUUcKRXdKRFRqRVFNQTRHQTFVRUNCTUhZbVZwYW1sdVp6RVFNQTRHQTFVRUJ4TUhZbVZwYW1sdVp6RU1NQW9HQTFVRQpDaE1EYjJ4a01Rd3dDZ1lEVlFRTEV3TnZjSE14RWpBUUJnTlZCQU1UQ1U5c1pHSnZlVVZrZFRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU4rQXVYWVdubEs3Ykw4RklMb0VZY2t6dmNWdEVmbEkKVFphUCtOcmt1Yy9SSVJ0VFpaZThMN3RKOEpnYVlzaVA2bDZ2SDAwMFZnMzlWK0V3ekYxT0FyMnB1Y2t2cm11dAo5b0pxN05jVndJV2o3cWF4eGdLRzkvVjZJNjl3MDJCcDk2cnpscnFyRjhoUXRpZU5vMWd5WUtlYXYrQWFZTXlJCnVUb1lWb1Q0ZHpuazlKdUtlRExJTkZhWStlTFd6a2IxdmdqbjdJbjZmNXdzek9uSW16dVFNUVFBRHQxL1N1ZXMKQW1QeHEyWVl6OTFPOXlWRi95SmtEVUZNT3NxNmZtZzlHVndDVnJEcUt2b0tTZlVpWHoxZXZ3M2dURlBmN2VzRQpmQzNwUXVoSEViMUE3QTdBcWVoQnZiaHgyaUZIZDNuWU0rZzRCN3NwT003ZjZBb1cvSEhZNjBNQ0F3RUFBYU5tCk1HUXdEZ1lEVlIwUEFRSC9CQVFEQWdFR01CSUdBMVVkRXdFQi93UUlNQVlCQWY4Q0FRSXdIUVlEVlIwT0JCWUUKRkZJc2lZYVY1aTZqSGI4bjBFeDc1RmJQTmxDcU1COEdBMVVkSXdRWU1CYUFGRklzaVlhVjVpNmpIYjhuMEV4Nwo1RmJQTmxDcU1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQzFHSEdWaW1mcS9yL2V1blRaTHpwc3M1Z3ZjT3dGClR5aTIrcE9YdmdLOUlLci9WMTBzUlVuTk10dHNHTnJoUk9mUUV0MFE5MjZwZDlQdC9ZUGJFeE55eWM4UUVQNGMKSm9JVXovYWN5VVEzeXNhTEJsallTR3JBa0kwRExGOXgySCtQLzluY0EyY3dqV1BnV05UZTd5dDZKNEdOekVUbApaOGlhRDZ5NU9mNzNHUEhXeGVZTkVBUDVXaTdIRUtROVdIcTBMdFBJQWFlYXVMYTV4ZVVCLzQvaXBGbExRaS85CkQ5SjIrdHRPUGFka21WR3VGZVZpUHlLdmY1TlJXSXNMTU5wSDVkSmRBL0xHaHVNeUwwMnBZZWR3bGNKcDZvMlAKME1VNEs2d0pWRVU3M2Q4QVVGTkNCODA2WUpWTnYwTUpNamZTZXlLTWpvay9uVHBMNmhSUUZ5RDkKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
server: https://10.4.7.10:7443
name: myk8s
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []
所以二进制安装的话，能够更加深刻的理解，各个组件的操作步骤和作用，配置文件所在目录等
第二步，把client-key也进行生成，拿客户端密钥和api-server通信
[root@hdss7-22 conf]# kubectl config set-credentials k8s-node \
--client-certificate=/opt/kubernetes/server/bin/cert/client.pem \
--client-key=/opt/kubernetes/server/bin/cert/client-key.pem \
--embed-certs=true \
--kubeconfig=kubelet.kubeconfig
以k8s-node用户去访问api-server（该用户需要授权）
[root@hdss7-22 conf]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=k8s-node \
--kubeconfig=kubelet.kubeconfig
kubectl config use-context myk8s-context --kubeconfig=kubelet.kubeconfig
总结一下，避免踩坑，上面一共执行了四条命令：
-
kubectl config set-cluster myk8s \ --certificate-authority=/opt/kubernetes/server/bin/cert/ca.pem \ --embed-certs=true \ --server=https://10.4.7.10:7443 \ --kubeconfig=kubelet.kubeconfig -
kubectl config set-credentials k8s-node \ --client-certificate=/opt/kubernetes/server/bin/cert/client.pem \ --client-key=/opt/kubernetes/server/bin/cert/client-key.pem \ --embed-certs=true \ --kubeconfig=kubelet.kubeconfig -
kubectl config set-context myk8s-context \ --cluster=myk8s \ --user=k8s-node \ --kubeconfig=kubelet.kubeconfig -
kubectl config use-context myk8s-context --kubeconfig=kubelet.kubeconfig
如果报错，就按上面总结的命令重新执行一遍即可。
[root@hdss7-22 conf]# cat kubelet.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CR1COE++++此处有省略++++0tLQo==
server: https://10.4.7.10:7443
name: myk8s
contexts:
- context:
cluster: myk8s
user: k8s-node
name: myk8s-context
current-context: ""
kind: Config
preferences: {}
users:
- name: k8s-node
user:
client-certificate-data: LS0tLS++++此处有省略++++tFWS0tLS0tCg==
关于kubeconfig文件
- 这是一个k8s用户的配置文件
- 它里面含有证书信息
- 证书过期或更换，需要同步替换该文件
角色绑定
k8s-node.yaml
创建资源配置文件
[root@hdss7-21 cert]# vim /opt/kubernetes/server/bin/conf/k8s-node.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: k8s-node
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: k8s-node
参数解读
apiversion：可以称为资源
roleRef：角色关系
让name: k8s-node的这样一个用户具有一个集群角色，让他具有运算节点的角色和权限
创建授权资源配置文件k8s-node.yaml节点
[root@hdss7-22 conf]# kubectl create -f k8s-node.yaml
clusterrolebinding.rbac.authorization.k8s.io/k8s-node created
查看节点
[root@hdss7-21 cert]# kubectl get clusterrolebinding k8s-node
NAME AGE
k8s-node 110s
继续查看yaml输出内容
[root@hdss7-21 cert]# kubectl get clusterrolebinding k8s-node -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: "2020-08-04T14:39:10Z"
name: k8s-node
resourceVersion: "22062"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/k8s-node
uid: 251b4021-04f4-4b1a-9f3f-91aa6929f39b
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: k8s-node
集群角色绑定策略，这个角色叫system：node，k8s一切皆资源。
kind: ClusterRoleBinding
在另一台机器上（7-21）继续操作
[root@hdss7-21 cert]# cd ../conf/
[root@hdss7-21 conf]# scp hdss7-22:/opt/kubernetes/server/bin/conf/kubelet.kubeconfig .
root@hdss7-22\'s password:
kubelet.kubeconfig
准备pause基础镜像
在运维主机7.200上操作
下载
[root@hdss7-200 certs]# docker login harbor.od.com
[root@hdss7-200 certs]# docker pull kubernetes/pause
[root@hdss7-200 certs]# docker images | grep pause
kubernetes/pause latest f9d5de079539 6 years ago 240kB
[root@hdss7-200 certs]# docker tag f9d5de079539 harbor.od.com/public/pause:latest
[root@hdss7-200 certs]# docker push harbor.od.com/public/pause:latest
The push refers to repository [harbor.od.com/public/pause]
5f70bf18a086: Mounted from public/nginx
e16a89738269: Pushed
latest: digest: sha256:b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105 size: 938
pause
在7-22机器上
他的作用就是kubenet启动的时候指定镜像，让所有的业务逻辑启动的时候，这个容器先于业务容器起来，因为pause镜像太小了，它可以给业务逻辑初始化网络空间，ipc空间，和utf空间。
编辑配置文件并启动kubelet服务
[root@hdss7-22 conf]# vim /opt/kubernetes/server/bin/kubelet.sh
#!/bin/sh
./kubelet \
--anonymous-auth=false \
--cgroup-driver systemd \
--cluster-dns 192.168.0.2 \
--cluster-domain cluster.local \
--runtime-cgroups=/systemd/system.slice \
--kubelet-cgroups=/systemd/system.slice \
--fail-swap-on="false" \
--client-ca-file ./cert/ca.pem \
--tls-cert-file ./cert/kubelet.pem \
--tls-private-key-file ./cert/kubelet-key.pem \
--hostname-override hdss7-22.host.com \
--image-gc-high-threshold 20 \
--image-gc-low-threshold 10 \
--kubeconfig ./conf/kubelet.kubeconfig \
--log-dir /data/logs/kubernetes/kube-kubelet \
--pod-infra-container-image harbor.od.com/public/pause:latest \
--root-dir /data/kubelet
cluster-dns：指定集群内部dns地址
hostname-override：当前机器主机名
pod-infra-container-image：pause镜像拉取地址
kubeconfig：指定上面创建的上下文配置文件
参数配置解析
| 参数 | 说明 |
|---|---|
| --anonymous-auth | 允许匿名请求到 kubelet 服务。未被另一个身份验证方法拒绝的请求被视为匿名请求。匿名请求包含系统的用户名: anonymous ，以及系统的组名: unauthenticated （默认 true ） |
| --cgroup-driver | 可选值有cgroupfs和systemd（默认cgroupfs）与docker驱动一致 |
| --cluster-dns | DNS 服务器的IP列表，多个用逗号分隔 |
| --cluster-domain | 集群域名, kubelet 将配置所有容器除了主机搜索域还将搜索当前域 |
| --fail-swap-on | 如果设置为true则启动kubelet失败(default true) |
| --hostname-override | cluster中的node name |
| --image-gc-high-threshold | 磁盘使用率最大值，超过此值将执行镜像垃圾回收(default 85) |
| --image-gc-low-threshold | 磁盘使用率最大值，低于此值将停止镜像垃圾回收(default 80) |
| --kubeconfig | 用来指定如何连接到 API server |
| --pod-infra-container-image | 每个 pod 中的 network/ipc 命名空间容器将使用的pause镜像 |
| --root-dir | kubelet 的工作目录 |
[root@hdss7-22 conf]# mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet
[root@hdss7-22 conf]# chmod +x /opt/kubernetes/server/bin/kubelet.sh
7-21机器上
[root@hdss7-21 conf]# pwd
/opt/kubernetes/server/bin/conf
[root@hdss7-21 conf]# ls -l | grep kubelet.kubeconfig
-rw------- 1 root root 6188 Aug 4 22:46 kubelet.kubeconfig
# 复制上面的内容
[root@hdss7-21 conf]# vim /opt/kubernetes/server/bin/kubelet.sh
[root@hdss7-21 conf]# mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet
[root@hdss7-21 conf]# chmod +x /opt/kubernetes/server/bin/kubelet.sh
[root@hdss7-21 conf]# vim /etc/supervisord.d/kube-kubelet.ini
[program:kube-kubelet-7-21]
command=/opt/kubernetes/server/bin/kubelet.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; \'expected\' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in \'capturemode\' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true
同时编辑7-22机器上的配置文件
[root@hdss7-22 conf]# vim /etc/supervisord.d/kube-kubelet.ini
[program:kube-kubelet-7-22]
command=/opt/kubernetes/server/bin/kubelet.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; \'expected\' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in \'capturemode\' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true
然后启动：
[root@hdss7-22 conf]# supervisorctl update
如果启动报错：
[root@hdss7-22 kube-kubelet]# supervisorctl status
etcd-server-7-22 RUNNING pid 3559, uptime 3 days, 23:57:09
kube-apiserver-7-22 RUNNING pid 3558, uptime 3 days, 23:57:09
kube-controller-manager-7-22 RUNNING pid 3555, uptime 3 days, 23:57:09
kube-kubelet-7-22 FATAL Exited too quickly (process log may have details)
kube-scheduler-7-22 RUNNING pid 3556, uptime 3 days, 23:57:09
查看日志发现：
failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"
请修改各个主机的docker的相关配置文件为如下:
[root@hdss7-22 conf]# vim /etc/docker/daemon.json
{
# ...
"exec-opts": ["native.cgroupdriver=systemd"],
}
然后重启
[root@hdss7-22 conf]# systemctl daemon-reload
[root@hdss7-22 conf]# systemctl status docker
再重启kube-kubelet-7-22和7-22
[root@hdss7-22 conf]# supervisorctl restart kube-kubelet-7-22
[root@hdss7-21 conf]# supervisorctl restart kube-kubelet-7-21
再次查看状态：
[root@hdss7-21 conf]# supervisorctl status
etcd-server-7-21 RUNNING pid 3839, uptime 4 days, 0:18:14
kube-apiserver-7-21 RUNNING pid 3840, uptime 4 days, 0:18:14
kube-controller-manager-7-21 RUNNING pid 3835, uptime 4 days, 0:18:14
kube-kubelet-7-21 RUNNING pid 13301, uptime 0:03:10
kube-scheduler-7-21 RUNNING pid 3837, uptime 4 days, 0:18:14
[root@hdss7-22 conf]# supervisorctl status
etcd-server-7-22 RUNNING pid 3559, uptime 4 days, 0:50:17
kube-apiserver-7-22 RUNNING pid 3558, uptime 4 days, 0:50:17
kube-controller-manager-7-22 RUNNING pid 3555, uptime 4 days, 0:50:17
kube-kubelet-7-22 RUNNING pid 11190, uptime 0:34:17
kube-scheduler-7-22 RUNNING pid 3556, uptime 4 days, 0:50:17
检查运算节点
[root@hdss7-22 conf]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready <none> 36m v1.15.2
hdss7-22.host.com Ready <none> 35m v1.15.
查看到角色是none，那就给运算节点打个标签
[root@hdss7-21 conf]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/master=
# 按回车，显示如下
node/hdss7-21.host.com labeled
[root@hdss7-21 conf]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready master 53m v1.15.2
hdss7-22.host.com Ready <none> 53m v1.15.2
让它既是主控节点，又是运算节点
[root@hdss7-21 conf]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/node=
# 按回车，显示如下
node/hdss7-21.host.com labeled
[root@hdss7-21 conf]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready master,node 53m v1.15.2
hdss7-22.host.com Ready <none> 53m v1.15.2
7-22也同样操作
[root@hdss7-22 conf]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/master=
# 按回车，显示如下
node/hdss7-22.host.com labeled
[root@hdss7-22 conf]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/node=
# 按回车，显示如下
node/hdss7-22.host.com labeled
[root@hdss7-22 conf]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready master,node 55m v1.15.2
hdss7-22.host.com Ready master,node 55m v1.15.2
签发kube-proxy证书
在运维主机HDSS7-200.host.com上：
创建生成证书签名请求（csr）的json配置文件。
[root@hdss7-200 certs]# vim kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
注意：system:kube-proxy不能改，在k8s中，system:后面的内容，是k8s默认的角色。
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-csr.json | cfssl-json -bare kube-proxy-client
.......
specifically, section 10.2.3 ("Information Requirements").
分发证书
[root@hdss7-21 cert]# pwd
/opt/kubernetes/server/bin/cert
[root@hdss7-21 cert]# scp hdss7-200:/opt/certs/kube-proxy-client*.pem .
[root@hdss7-22 cert]# scp hdss7-200:/opt/certs/kube-proxy-client*.pem .
创建kube-proxy配置
[root@hdss7-22 cert]# cd ../conf
[root@hdss7-22 conf]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/cert/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=kube-proxy.kubeconfig
Cluster "myk8s" set.
[root@hdss7-22 conf]# kubectl config set-credentials kube-proxy \
--client-certificate=/opt/kubernetes/server/bin/cert/kube-proxy-client.pem \
--client-key=/opt/kubernetes/server/bin/cert/kube-proxy-client-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
User "kube-proxy" set.
[root@hdss7-22 conf]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
Context "myk8s-context" created.
[root@hdss7-22 conf]# kubectl config use-context myk8s-context --kubeconfig=kube-proxy.kubeconfig
Switched to context "myk8s-context".
7-21直接从7-22进行考呗
[root@hdss7-21 conf]# scp hdss7-22:/opt/kubernetes/server/bin/conf/kube-proxy* .
加载ipvs模块
[root@hdss7-21 conf]# cd ..
[root@hdss7-21 bin]# vim /root/ipvs.sh
#!/bin/bash
ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"
for i in $(ls $ipvs_mods_dir|grep -o "^[^.]*")
do
/sbin/modinfo -F filename $i &>/dev/null
if [ $? -eq 0 ];then
/sbin/modprobe $i
fi
done
[root@hdss7-21 bin]# lsmod | grep ip_vs
[root@hdss7-21 bin]# chmod +x /root/ipvs.sh
[root@hdss7-21 bin]# ~/ipvs.sh
[root@hdss7-21 bin]# lsmod | grep ip_vs
ip_vs_wrr 12697 0
ip_vs_wlc 12519 0
ip_vs_sh 12688 0
ip_vs_sed 12519 0
ip_vs_rr 12600 0
ip_vs_pe_sip 12697 0
nf_conntrack_sip 33860 1 ip_vs_pe_sip
ip_vs_nq 12516 0
ip_vs_lc 12516 0
ip_vs_lblcr 12922 0
ip_vs_lblc 12819 0
ip_vs_ftp 13079 0
ip_vs_dh 12688 0
ip_vs 141092 24 ip_vs_dh,ip_vs_lc,ip_vs_nq,ip_vs_rr,ip_vs_sh,ip_vs_ftp,ip_vs_sed,ip_vs_wlc,ip_vs_wrr,ip_vs_pe_sip,ip_vs_lblcr,ip_vs_lblc
nf_nat 26787 3 ip_vs_ftp,nf_nat_ipv4,nf_nat_masquerade_ipv4
nf_conntrack 133387 8 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_sip,nf_conntrack_ipv4
libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack
7-22也是同样操作
推荐动态算法nq
接下来创建启动文件并授权
[root@hdss7-21 bin]# vim /opt/kubernetes/server/bin/kube-proxy.sh
#!/bin/sh
./kube-proxy \
--cluster-cidr 172.7.0.0/16 \
--hostname-override hdss7-21.host.com \
--proxy-mode=ipvs \
--ipvs-scheduler=nq \
--kubeconfig ./conf/kube-proxy.kubeconfig
[root@hdss7-21 bin]# chmod +x /opt/kubernetes/server/bin/kube-proxy.sh
创建proxy日志存放路径
mkdir -p /data/logs/kubernetes/kube-proxy
创建ini文件
[root@hdss7-21 bin]# vim /etc/supervisord.d/kube-proxy.ini
[program:kube-proxy-7-21]
command=/opt/kubernetes/server/bin/kube-proxy.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at
unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; \'expected\' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in \'capturemode\' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true
注意：在7-22上创建配置文件记得修改第一行，其他操作都一样
[program:kube-proxy-7-22]
然后启动并查看状态
[root@hdss7-21 bin]# supervisorctl update
kube-proxy-7-21: added process group
[root@hdss7-21 bin]# supervisorctl status
etcd-server-7-21 RUNNING pid 3839, uptime 4 days, 1:59:51
kube-apiserver-7-21 RUNNING pid 3840, uptime 4 days, 1:59:51
kube-controller-manager-7-21 RUNNING pid 3835, uptime 4 days, 1:59:51
kube-kubelet-7-21 RUNNING pid 13301, uptime 1:44:47
kube-proxy-7-21 RUNNING pid 33588, uptime 0:01:25
kube-scheduler-7-21 RUNNING pid 3837, uptime 4 days, 1:59:51
安装ipvsadm
[root@hdss7-21 bin]# yum install ipvsadm -y
kube-proxy其实就是在K8s内嵌了一套LVS
[root@hdss7-21 bin]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.1:443 nq
-> 10.4.7.21:6443 Masq 1 0 0
-> 10.4.7.22:6443 Masq 1 0 0
在另外一个运算节点7-22上也是同样操作。