解决项目迁移至Kubernetes集群中的代理问题

随着Kubernetes技术的日益成熟，越来越多的企业选择用Kubernetes集群来管理项目。新项目还好，可以选择合适的集群规模从零开始构建项目；旧项目迁移进Kubernetes集群就需要考虑很多因素，毕竟项目不能中断时间过久。

问题来源

近日在做项目迁移至Kubernetes集群时，遇到了一件有意思的问题：因为开发用的dubbo版本过低，在zookeeper注册不上，需要开发升级dobbo，然后在打包成镜像，所以要先把nodejs迁移进Kubernets集群。因为是部分业务迁移进Kubernets集群，所以要在traefik 前面还得加一层代理Nginx(Nginx为旧业务的入口，反向代理后面的微服务，阿里云的slb指向nginx，等到业务全部迁移完毕，slb就指向traefik)。此种架构为双层代理，即Slb-->Nginx-->Traefik-->Service 。

图解

解决项目迁移至Kubernetes集群中的代理问题

解决方案：

迁移至k8s集群的业务走Nodeport，Nginx --> Nodeport。业务应用直接Nodeport，不好管理，1万台机器的时候不能也Nodeport吧，端口自己要规划，机器多了每个机器还都暴露端口，想想都不现实
迁移至k8s集群的业务走Clusterip，Nginx --> Traefik --> Service。这种方式合理。

解决问题

总不能拿生产环境来写博文吧，用虚机讲明。其实把虚机和生产机也就网络环境存在差别。

思路分析

部署k8s集群
部署nginx
部署traefik
部署应用
联调联试

部署k8s集群

使用我之前的博文部署方法：https://www.cnblogs.com/zisefeizhu/p/12505117.html

部署nginx

下载必要的组件

# hostname -I

20.0.0.101

# cat /etc/redhat-release

CentOS Linux release 7.6.1810 (Core)

# uname -a

Linux fuxi-node02-101 4.4.186-1.el7.elrepo.x86_64 #1 SMP Sun Jul 21 04:06:52 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

# wget http://nginx.org/download/nginx-1.10.2.tar.gz

# wget http://www.openssl.org/source/openssl-fips-2.0.10.tar.gz

# wget http://zlib.net/zlib-1.2.11.tar.gz

# wget https://ftp.pcre.org/pub/pcre/pcre-8.40.tar.gz

# yum install gcc-c++

配置-编译-安装软件

# tar zxvf openssl-fips-2.0.10.tar.gz

# cd openssl-fips-2.0.10/

# ./config && make && make install

# cd ..

# ll

tar zxvf pcre-8.40.tar.gz

# cd pcre-8.40/

# ./configure && make && make install

# tar zxvf zlib-1.2.11.tar.gz

# cd zlib-1.2.11/

# ./configure && make && make install

# tar zxvf nginx-1.10.2.tar.gz

# cd nginx-1.10.2/

#./configure --with-http_stub_status_module --prefix=/opt/nginx

# make && make install

启动Nginx

# pwd

/opt/nginx

# ll

总用量 4

drwx------ 2 nobody root    6 4月  22 11:30 client_body_temp

drwxr-xr-x 2 root   root 4096 4月  22 12:53 conf

drwx------ 2 nobody root    6 4月  22 11:30 fastcgi_temp

drwxr-xr-x 2 root   root   40 4月  22 11:29 html

drwxr-xr-x 2 root   root   41 4月  22 14:24 logs

drwx------ 2 nobody root    6 4月  22 11:30 proxy_temp

drwxr-xr-x 2 root   root   19 4月  22 11:29 sbin

drwx------ 2 nobody root    6 4月  22 11:30 scgi_temp

drwx------ 2 nobody root    6 4月  22 11:30 uwsgi_temp

# sbin/nginx

解决项目迁移至Kubernetes集群中的代理问题

traefik 部署

https://www.cnblogs.com/zisefeizhu/p/12692979.html

环境检查

# kubectl get pods,svc -A | grep traefik

kube-system   pod/traefik-ingress-controller-z5qd7             1/1     Running       0          136m

kube-system   service/traefik                     ClusterIP   10.68.251.132   <none>        80/TCP,443/TCP,8080/TCP        4h14m

浏览器访问

解决项目迁移至Kubernetes集群中的代理问题

部署应用

这里的测试应用选择containous/whoami镜像

测试应用部署

# cat whoami.yaml

##########################################################################

#Author:                     zisefeizhu

#QQ:                         2********0

#Date:                       2020-04-22

#FileName:                   whoami.yaml

#URL:                        https://www.cnblogs.com/zisefeizhu/

#Description:                The test script

#Copyright (C):              2020 All rights reserved

###########################################################################

apiVersion: v1

kind: Service

metadata:

  name: whoami

spec:

  ports:

    - protocol: TCP

      name: web

      port: 80

  selector:

    app: whoami

---

kind: Deployment

apiVersion: apps/v1

metadata:

  name: whoami

  labels:

    app: whoami

spec:

  replicas: 2

  selector:

    matchLabels:

      app: whoami

  template:

    metadata:

      labels:

        app: whoami

    spec:

      containers:

        - name: whoami

          image: containous/whoami

          ports:

            - name: web

              containerPort: 80

# kubectl get svc,pod

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE

service/whoami       ClusterIP   10.68.109.151   <none>        80/TCP              3h30m

NAME                                 READY   STATUS    RESTARTS   AGE

pod/whoami-bd6b677dc-jvqc2           1/1     Running   0          3h30m

pod/whoami-bd6b677dc-lvcxp           1/1     Running   0          3h30m

联调联试

因为选择的解决问题的方案是：nginx --> traefik --> service

traefik -->service
nginx --> traefik
nginx --> service

traefik --> service

使用traefik 代理测试应用的资源清单：

# cat traefik-whoami.yaml

##########################################################################

#Author:                     zisefeizhu

#QQ:                         2********0

#Date:                       2020-04-22

#FileName:                   traefik-whoami.yaml

#URL:                        https://www.cnblogs.com/zisefeizhu/

#Description:                The test script

#Copyright (C):              2020 All rights reserved

###########################################################################

apiVersion: traefik.containo.us/v1alpha1

kind: IngressRoute

metadata:

  name: simpleingressroute

spec:

  entryPoints:

    - web

  routes:

  - match: Host(`who.linux.com`) && PathPrefix(`/notls`)

    kind: Rule

    services:

    - name: whoami

      port: 80

本地hosts解析

解决项目迁移至Kubernetes集群中的代理问题

traefik界面观察是代理成功：

访问who.linux.com/notls

nginx --> traefik

# cat conf/nginx.conf

user nobody;

worker_processes 4;

events {

	use epoll;

	worker_connections 2048;

}

http {

	upstream app {

		server  20.0.0.202;

	}

	server {

		listen 80;

#		server_name who2.linux.com;

  	access_log logs/access.log;

  	error_log logs/error.log;

  	location / {

   		proxy_set_header X-Forwarded-For $remote_addr;

    	proxy_set_header X-Real-IP $remote_addr;

    	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    	proxy_set_header Host $host;

			proxy_headers_hash_max_size 51200;

			proxy_headers_hash_bucket_size 6400;

    	proxy_redirect off;

    	proxy_read_timeout 600;

    	proxy_connect_timeout 600;

    	proxy_pass http://app;

  	}

	}

}

# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

20.0.0.202  who.linux.com   //k8s集群traefik所落节点，其实K8s任意节点都随便拉

# curl -iL who.linux.com/notls

HTTP/1.1 200 OK

Content-Length: 388

Content-Type: text/plain; charset=utf-8

Date: Wed, 22 Apr 2020 07:33:52 GMT

Hostname: whoami-bd6b677dc-lvcxp

IP: 127.0.0.1

IP: 172.20.46.67

RemoteAddr: 172.20.177.153:58168

GET /notls HTTP/1.1

Host: who.linux.com

User-Agent: curl/7.29.0

Accept: */*

Accept-Encoding: gzip

X-Forwarded-For: 20.0.0.101

X-Forwarded-Host: who.linux.com

X-Forwarded-Port: 80

X-Forwarded-Proto: http

X-Forwarded-Server: traefik-ingress-controller-z5qd7

X-Real-Ip: 20.0.0.101

nginx要是不熟悉就看这大佬的博文吧：https://www.cnblogs.com/kevingrace/p/6095027.html

nginx --> service

# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

20.0.0.101  who.linux.com

# curl -iL who.linux.com/notls

HTTP/1.1 200 OK       //响应信息

Server: nginx/1.10.2    //响应服务

Date: Wed, 22 Apr 2020 07:27:46 GMT

Content-Type: text/plain; charset=utf-8

Content-Length: 389

Connection: keep-alive

Hostname: whoami-bd6b677dc-jvqc2

IP: 127.0.0.1

IP: 172.20.46.111

RemoteAddr: 172.20.177.153:38298

GET /notls HTTP/1.1

Host: who.linux.com

User-Agent: curl/7.29.0

Accept: */*

Accept-Encoding: gzip

X-Forwarded-For: 20.0.0.101

X-Forwarded-Host: who.linux.com

X-Forwarded-Port: 80

X-Forwarded-Proto: http

X-Forwarded-Server: traefik-ingress-controller-z5qd7

X-Real-Ip: 20.0.0.101

nginx日志

# tail -f access.log

20.0.0.101 - - [22/Apr/2020:15:28:28 +0800] "GET /notls HTTP/1.1" 200 389 "-" "curl/7.29.0"

浏览器测试

解决项目迁移至Kubernetes集群中的代理问题

继续测试

把traefik应用给关了，然后再测试

# kubectl delete -f .

configmap "traefik-config" deleted

customresourcedefinition.apiextensions.k8s.io "ingressroutes.traefik.containo.us" deleted

customresourcedefinition.apiextensions.k8s.io "ingressroutetcps.traefik.containo.us" deleted

customresourcedefinition.apiextensions.k8s.io "middlewares.traefik.containo.us" deleted

customresourcedefinition.apiextensions.k8s.io "tlsoptions.traefik.containo.us" deleted

customresourcedefinition.apiextensions.k8s.io "traefikservices.traefik.containo.us" deleted

ingressroute.traefik.containo.us "traefik-dashboard-route" deleted

service "traefik" deleted

daemonset.apps "traefik-ingress-controller" deleted

serviceaccount "traefik-ingress-controller" deleted

clusterrole.rbac.authorization.k8s.io "traefik-ingress-controller" deleted

clusterrolebinding.rbac.authorization.k8s.io "traefik-ingress-controller" deleted

解决项目迁移至Kubernetes集群中的代理问题

# kubectl delete -f traefik-whoami.yaml   //关闭whoami traefik代理

ingressroute.traefik.containo.us "simpleingressroute" deleted

解决项目迁移至Kubernetes集群中的代理问题

没得说了测试结果很明确了：访问who.linux.com 流量走向：nginx-->traefik --> service 。

秒客网

解决项目迁移至Kubernetes集群中的代理问题