背景
在大规模集群场景中，部分服务需要通过公网Ingress对外提供服务访问，但是有部分服务只对内提供服务，不允许使用公网访问，仅支持内部服务间调用，此时可以通过部署两套独立的Ingress来实现，一套支持公网访问，一套仅支持内网访问。接下来，我们通过最佳实践进行实现喽！
架构图
最佳实践说明：
- 在Kubernetes集群中，部署两个Ingress控制器分别为int-ingress-nginx和ingress-nginx
- int-ingress-nginx负责承接内部流量
- ingress-nginx负责承接外部流量
Ingress 配置参数说明
- controller.image.repository：ingress-nginx镜像地址
- controller.image.registry：镜像仓库域名，该参数需要与controller.image.image同时填写
- controller.image.image：镜像名称。该参数需要与controller.image.registry同时填写
- controller.image.tag：ingress-nginx镜像版本，可通过已安装插件实例的YAML文件查看，需要根据插件版本进行替换
- controller.ingressClass：设置Ingress Controller所对应的IngressClass的名称，同一个集群中不同套Ingress Controller名称必须唯一
- controller.ingressClassResource.name：需要与ingressClass值相同
- controller.ingressClassResource.controllerValue：同一个集群中不同套Ingress Controller的监听标识必须唯一，且不能设置为k8s.io/ingress-nginx，（k8s.io/ingress-nginx是默认Nginx Ingress Controller的监听标识）。
- controller.resources.requests.cpu：Nginx controller的CPU资源申请值，可根据需求自定义
- controller.resources.requests.memory：Nginx controller的内存资源申请值，可根据需求自定义
- defaultBackend.image.tag：default-backend镜像版本
- controller.admissionWebhooks.enabled：是否开启admissionWebhooks，可以对Ingress对象进行有效性校验，避免因配置错误导致ingress-controller不断重新加载资源，导致业务中断。
- controller.admissionWebhooks.patch.enabled：同上，表示是否开启admissionWebhooks。此处设置为false。
Ingress控制器CRD的关联关系：
说明：
- Ingress-controller启动的时候指定controller-class名称
- 通过IngressClass CRD对象进行声明并与Ingress-controller进行绑定
- Ingress CRD对象关联IngressClassName
部署多个Ingress Nginx
本案例通过helm进行安装:
# 配置helm仓库
$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
"ingress-nginx" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "higress.io" chart repository
...Successfully got an update from the "ingress-nginx" chart repository
Update Complete. ⎈Happy Helming!⎈
安装Ingress-nginx（外网流量）：
$ helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --set controller.ingressClass=ingress-nginx,controller.ingressClassResource.name=ingress-nginx
NAME: ingress-nginx
LAST DEPLOYED: Tue Aug 22 14:26:25 2023
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'
An example Ingress that makes use of the controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
namespace: foo
spec:
ingressClassName: ingress-nginx
rules:
- host: www.example.com
http:
paths:
- pathType: Prefix
backend:
service:
name: exampleService
port:
number: 80
path: /
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- www.example.com
secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tls
安装int-ingress-nginx（内部流量）：
$ helm -n int-ingress-nginx install int-ingress-nginx ingress-nginx/ingress-nginx --create-namespace --set controller.ingressClass=int-ingress-nginx,controller.ingressClassResource.name=int-ingress-nginx
NAME: int-ingress-nginx
LAST DEPLOYED: Tue Aug 22 14:29:00 2023
NAMESPACE: int-ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace int-ingress-nginx get services -o wide -w int-ingress-nginx-controller'
An example Ingress that makes use of the controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
namespace: foo
spec:
ingressClassName: int-ingress-nginx
rules:
- host: www.example.com
http:
paths:
- pathType: Prefix
backend:
service:
name: exampleService
port:
number: 80
path: /
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- www.example.com
secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tls
验证：
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-controller-5db5867dcf-v9pfv 1/1 Running 0 3m
int-ingress-nginx int-ingress-nginx-controller-54585986c-pp9v4 1/1 Running 0 26s
kube-system coredns-5d78c9869d-5m4t5 1/1 Running 0 4d22h
kube-system coredns-5d78c9869d-bs7s4 1/1 Running 0 4d22h
kube-system etcd-kind-control-plane 1/1 Running 0 4d22h
kube-system kindnet-5cxpf 1/1 Running 0 4d22h
kube-system kube-apiserver-kind-control-plane 1/1 Running 0 4d22h
kube-system kube-controller-manager-kind-control-plane 1/1 Running 0 4d22h
kube-system kube-proxy-c9ztm 1/1 Running 0 4d22h
kube-system kube-scheduler-kind-control-plane 1/1 Running 0 4d22h
local-path-storage local-path-provisioner-6bc4bddd6b-7v8fm 1/1 Running 0 4d22h
$ kubectl get ingressClass
NAME CONTROLLER PARAMETERS AGE
ingress-nginx k8s.io/ingress-nginx <none> 3m57s
int-ingress-nginx k8s.io/ingress-nginx <none> 83s
到此为止，多个ingress安装成功，接下来，咱们可以安装demo进行验证了！
部署示例服务
- Auth服务：负责用户登陆授权返回Token
- Demo服务：负责认证通过后访问的微服务
# 部署Auth服务
$ cat auth.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth
labels:
app: auth
spec:
replicas: 1
selector:
matchLabels:
app: auth
template:
metadata:
labels:
app: auth
spec:
containers:
- name: auth
imagePullPolicy: Always
image: registry.cn-shanghai.aliyuncs.com/kubesre01/auth:v1
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: auth-svc
spec:
type: ClusterIP
selector:
app: auth
ports:
- port: 8080
targetPort: 8080
$ kubectl apply -f auth.yml
deployment.apps/auth unchanged
service/auth-svc unchanged
# 部署demo服务
$ cat demo.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo
labels:
app: demo
spec:
replicas: 1
selector:
matchLabels:
app: demo
template:
metadata:
labels:
app: demo
spec:
containers:
- name: demo
imagePullPolicy: Always
image: registry.cn-shanghai.aliyuncs.com/kubesre01/demo:v1
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: demo-svc
spec:
type: ClusterIP
selector:
app: demo
ports:
- port: 8080
targetPort: 8080
$ kubectl apply -f demo.yml
deployment.apps/demo unchanged
service/demo-svc unchanged
验证
创建内部流量的Ingress资源对象：
$ cat ingress.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo
spec:
rules:
- host: demo.kubesre.com
http:
paths:
- path: /info
pathType: Prefix
backend:
service:
name: demo-svc
port:
number: 8080
ingressClassName: int-ingress-nginx # 指定内部流量Ingress控制器
$ kubectl apply -f ingress.yml
ingress.networking.k8s.io/demo created
访问验证：
# 向内部流量Ingress发送请求
$ curl demo.kubesre.com/info
{"message":"云原生运维圈！"}
# 查看内部流量INgress日志进行确认
$ kubectl logs -n int-ingress-nginx -f int-ingress-nginx-controller-54585986c-pp9v4
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.8.1
Build: dc88dce9ea5e700f3301d16f971fa17c6cfe757d
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6
-------------------------------------------------------------------------------
W0822 06:29:03.903208 10 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0822 06:29:03.903300 10 main.go:209] "Creating API client" host="https://10.96.0.1:443"
I0822 06:29:03.906676 10 main.go:253] "Running in Kubernetes cluster" major="1" minor="27" git="v1.27.3" state="clean" commit="25b4e43193bcda6c7328a6d147b1fb73a33f1598" platform="linux/arm64"
I0822 06:29:03.989381 10 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0822 06:29:03.999403 10 ssl.go:533] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0822 06:29:04.004242 10 nginx.go:261] "Starting NGINX Ingress controller"
I0822 06:29:04.007613 10 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"int-ingress-nginx", Name:"int-ingress-nginx-controller", UID:"378c8763-6969-4a5b-8296-3de95dedd962", APIVersion:"v1", ResourceVersion:"135147", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap int-ingress-nginx/int-ingress-nginx-controller
I0822 06:29:05.206429 10 nginx.go:304] "Starting NGINX process"
I0822 06:29:05.206811 10 leaderelection.go:248] attempting to acquire leader lease int-ingress-nginx/int-ingress-nginx-leader...
I0822 06:29:05.207155 10 nginx.go:324] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0822 06:29:05.208516 10 controller.go:190] "Configuration changes detected, backend reload required"
I0822 06:29:05.220682 10 leaderelection.go:258] successfully acquired lease int-ingress-nginx/int-ingress-nginx-leader
I0822 06:29:05.221284 10 status.go:84] "New leader elected" identity="int-ingress-nginx-controller-54585986c-pp9v4"
I0822 06:29:05.283791 10 controller.go:207] "Backend successfully reloaded"
I0822 06:29:05.283853 10 controller.go:218] "Initial sync, sleeping for 1 second"
I0822 06:29:05.283888 10 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"int-ingress-nginx", Name:"int-ingress-nginx-controller-54585986c-pp9v4", UID:"26b115bc-5d52-47fa-927d-71d8ef8eaa54", APIVersion:"v1", ResourceVersion:"135176", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W0822 07:54:01.251833 10 controller.go:328] ignoring ingress demo-redirect in default based on annotation : no object matching key "nginx" in local store
I0822 07:54:01.251929 10 main.go:110] "successfully validated configuration, accepting" ingress="default/demo-redirect"
I0822 07:54:01.255287 10 store.go:428] "Ignoring ingress because of error while validating ingress class" ingress="default/demo-redirect" error="no object matching key \"nginx\" in local store"
I0822 07:54:14.491144 10 store.go:398] "Ignoring ingress because of error while validating ingress class" ingress="default/demo-redirect" error="no object matching key \"nginx\" in local store"
I0822 07:59:03.354868 10 admission.go:149] processed ingress via admission controller {testedIngressLength:1 testedIngressTime:0.06s renderingIngressLength:1 renderingIngressTime:0.001s admissionTime:25.5kBs testedConfigurationSize:0.061}
I0822 07:59:03.354915 10 main.go:110] "successfully validated configuration, accepting" ingress="default/demo"
I0822 07:59:03.358113 10 store.go:432] "Found valid IngressClass" ingress="default/demo" ingressclass="int-ingress-nginx"
I0822 07:59:03.358399 10 controller.go:190] "Configuration changes detected, backend reload required"
I0822 07:59:03.358596 10 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"demo", UID:"64bbeb49-e705-423c-9f86-1fcbf066f6ce", APIVersion:"networking.k8s.io/v1", ResourceVersion:"143749", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0822 07:59:03.402693 10 controller.go:207] "Backend successfully reloaded"
I0822 07:59:03.402902 10 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"int-ingress-nginx", Name:"int-ingress-nginx-controller-54585986c-pp9v4", UID:"26b115bc-5d52-47fa-927d-71d8ef8eaa54", APIVersion:"v1", ResourceVersion:"135176", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
127.0.0.1 - - [22/Aug/2023:08:07:28 +0000] "GET /info HTTP/1.1" 200 35 "-" "curl/7.88.1" 84 0.010 [default-demo-svc-8080] [] 10.244.0.27:8080 35 0.010 200 3364c68cbd67906e432a3428f175034
127.0.0.1 - - [22/Aug/2023:08:07:45 +0000] "GET /info HTTP/1.1" 200 35 "-" "curl/7.88.1" 84 0.007 [default-demo-svc-8080] [] 10.244.0.27:8080 35 0.006 200 02dc2168565fce906af19ad33949d836
# 可以看出是否流量进入的说明验证成功了
创建外部流量的Ingress资源对象：
$ cat ingress.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: auth
spec:
rules:
- host: demo.kubesre.com
http:
paths:
- path: /login
pathType: Prefix
backend:
service:
name: auth-svc
port:
number: 8080
ingressClassName: ingress-nginx # 指定外部流量Ingress控制器
$ kubectl apply -f ingress.yml
ingress.networking.k8s.io/auth created
验证：
# 向外部流量Ingress发送请求
$ curl demo.kubesre.com/login
404 page not found
# 查看外部流量INgress日志进行确认
kubectl logs -f -n ingress-nginx ingress-nginx-controller-5db5867dcf-v9pfv
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.8.1
Build: dc88dce9ea5e700f3301d16f971fa17c6cfe757d
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6
-------------------------------------------------------------------------------
W0822 06:26:30.280533 10 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0822 06:26:30.280652 10 main.go:209] "Creating API client" host="https://10.96.0.1:443"
I0822 06:26:30.284480 10 main.go:253] "Running in Kubernetes cluster" major="1" minor="27" git="v1.27.3" state="clean" commit="25b4e43193bcda6c7328a6d147b1fb73a33f1598" platform="linux/arm64"
I0822 06:26:30.520820 10 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0822 06:26:30.531350 10 ssl.go:533] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0822 06:26:30.543711 10 nginx.go:261] "Starting NGINX Ingress controller"
I0822 06:26:30.551968 10 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"c019e1aa-4bc2-42f9-a049-c1a714404030", APIVersion:"v1", ResourceVersion:"134801", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0822 06:26:31.745338 10 nginx.go:304] "Starting NGINX process"
I0822 06:26:31.745585 10 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I0822 06:26:31.746220 10 nginx.go:324] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0822 06:26:31.746577 10 controller.go:190] "Configuration changes detected, backend reload required"
I0822 06:26:31.751580 10 status.go:84] "New leader elected" identity="ingress-nginx-controller-5fcb5746fc-4ppzf"
I0822 06:26:31.827000 10 controller.go:207] "Backend successfully reloaded"
I0822 06:26:31.831564 10 controller.go:218] "Initial sync, sleeping for 1 second"
I0822 06:26:31.831562 10 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5db5867dcf-v9pfv", UID:"5572da28-300b-42c4-9faf-414b5211e65e", APIVersion:"v1", ResourceVersion:"134833", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0822 06:27:08.891616 10 leaderelection.go:258] successfully acquired lease ingress-nginx/ingress-nginx-leader
I0822 06:27:08.891679 10 status.go:84] "New leader elected" identity="ingress-nginx-controller-5db5867dcf-v9pfv"
W0822 07:54:01.251978 10 controller.go:328] ignoring ingress demo-redirect in default based on annotation : no object matching key "nginx" in local store
I0822 07:54:01.252103 10 main.go:110] "successfully validated configuration, accepting" ingress="default/demo-redirect"
I0822 07:54:01.255287 10 store.go:428] "Ignoring ingress because of error while validating ingress class" ingress="default/demo-redirect" error="no object matching key \"nginx\" in local store"
I0822 07:54:14.491144 10 store.go:398] "Ignoring ingress because of error while validating ingress class" ingress="default/demo-redirect" error="no object matching key \"nginx\" in local store"
I0822 07:59:03.354862 10 admission.go:149] processed ingress via admission controller {testedIngressLength:1 testedIngressTime:0.06s renderingIngressLength:1 renderingIngressTime:0.001s admissionTime:25.5kBs testedConfigurationSize:0.061}
I0822 07:59:03.354910 10 main.go:110] "successfully validated configuration, accepting" ingress="default/demo"
I0822 07:59:03.360144 10 store.go:432] "Found valid IngressClass" ingress="default/demo" ingressclass="int-ingress-nginx"
I0822 07:59:03.360620 10 controller.go:190] "Configuration changes detected, backend reload required"
I0822 07:59:03.362848 10 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"demo", UID:"64bbeb49-e705-423c-9f86-1fcbf066f6ce", APIVersion:"networking.k8s.io/v1", ResourceVersion:"143749", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0822 07:59:03.404657 10 controller.go:207] "Backend successfully reloaded"
I0822 07:59:03.404826 10 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5db5867dcf-v9pfv", UID:"5572da28-300b-42c4-9faf-414b5211e65e", APIVersion:"v1", ResourceVersion:"134833", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
127.0.0.1 - - [22/Aug/2023:08:01:56 +0000] "GET / HTTP/1.1" 404 146 "-" "curl/7.88.1" 80 0.002 [upstream-default-backend] [] 127.0.0.1:8181 146 0.002 404 eba5cdeb999b4cac9a00142e599cbb1c
127.0.0.1 - - [22/Aug/2023:08:01:59 +0000] "GET /info HTTP/1.1" 200 35 "-" "curl/7.88.1" 84 0.043 [default-demo-svc-8080] [] 10.244.0.27:8080 35 0.044 200 6ffcb6dd13f1b157b6b12e1ec8bd2336
127.0.0.1 - - [22/Aug/2023:08:03:39 +0000] "GET /info HTTP/1.1" 200 35 "-" "curl/7.88.1" 84 0.007 [default-demo-svc-8080] [] 10.244.0.27:8080 35 0.007 200 6846c357020f3beb6245c2d42f7d3c8c
I0822 08:10:19.216812 10 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.067s renderingIngressLength:2 renderingIngressTime:0.001s admissionTime:32.8kBs testedConfigurationSize:0.068}
I0822 08:10:19.216851 10 main.go:110] "successfully validated configuration, accepting" ingress="default/auth"
I0822 08:10:19.224537 10 store.go:432] "Found valid IngressClass" ingress="default/auth" ingressclass="ingress-nginx"
I0822 08:10:19.224973 10 controller.go:190] "Configuration changes detected, backend reload required"
I0822 08:10:19.225003 10 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"auth", UID:"dfd1450a-a1c6-45fa-9f77-d74ac29764a9", APIVersion:"networking.k8s.io/v1", ResourceVersion:"144811", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0822 08:10:19.271423 10 controller.go:207] "Backend successfully reloaded"
I0822 08:10:19.271788 10 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5db5867dcf-v9pfv", UID:"5572da28-300b-42c4-9faf-414b5211e65e", APIVersion:"v1", ResourceVersion:"134833", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
127.0.0.1 - - [22/Aug/2023:08:11:11 +0000] "GET /login HTTP/1.1" 404 18 "-" "curl/7.88.1" 85 0.036 [default-auth-svc-8080] [] 10.244.0.26:8080 18 0.036 404 aa8a7aa2bb41c704a5426bb851e967e6
127.0.0.1 - - [22/Aug/2023:08:11:14 +0000] "GET /login HTTP/1.1" 404 18 "-" "curl/7.88.1" 85 0.005 [default-auth-svc-8080] [] 10.244.0.26:8080 18 0.006 404 05a5c56c610ae2e3c11d8a1fe8ed7778
127.0.0.1 - - [22/Aug/2023:08:11:15 +0000] "GET /login HTTP/1.1" 404 18 "-" "curl/7.88.1" 85 0.004 [default-auth-svc-8080] [] 10.244.0.26:8080 18 0.004 404 2787d38a23c042e5145f62310c0be4b0
# 可以看出是否流量进入的说明验证成功了
总结
本文介绍了Kubernetes集群内部署多个Ingress控制器的应用场景，并通过实际案例的方式讲解，下一章将讲解Ingress更多企业级实战，请敬请期待！