1、ingress-nginx使用手册

1、匹配请求头

匹配请求头，主要用于根据请求头信息将用户请求转发到不同的应用，比如根据不同的客户端转发请求

 annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/server-snippet: |
        set $agentflag 0;
        if ($http_user_agent ~* "(iPhone)" ){
          set $agentflag 1;
        }
        if ( $agentflag = 1 ) {
          return 301 http://iphone.coolops.cn:30369;
        }

2、白名单及请求速率限制

设置 test.haha.com/login 登陆页为每秒100个连接数，192.168.1.0/24,192.168.2.8 IP段不在限速范围

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-dt
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/whitelist-source-range: 192.168.1.0/24,192.168.2.8
    nginx.ingress.kubernetes.io/limit-rps: '100'
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  tls:
  - hosts:
    - test.haha.com
    secretName: tls-https
  rules:
    - host: test.haha.com
      http:
        paths:
          - path: /login
            backend:
              serviceName: nginx-front
              servicePort: 80

3、redirect和rewrite-target

redirect主要用于域名重定向，比如访问a.com被重定向到b.com

nginx.ingress.kubernetes.io/permanent-redirect: "https://www.baidu.com"

rewrite主要用于地址重写，比如访问a.com/foo重写到a.com，访问a.com/foo/bbb重写到a.com/bbb

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-nginx
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: "/$1"
spec:
  rules:
  - host: ng.coolops.cn 
    http:
      paths:
      - path: /foo/?(.*)
        backend:
          serviceName: ng-svc
          servicePort: 80

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: rewrite
  namespace: default
spec:
  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /something(/|$)(.*)

4、跨域访问

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-dt
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true
    nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-headers："DNT,X-CustomHeader,Keep-Alive,User- 
Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization"
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"

5、支持websocket配置

nginx.ingress.kubernetes.io/configuration-snippet （用于插入 location 块代码段）
nginx.ingress.kubernetes.io/server-snippet （用于插入 server 块中的代码段）

annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Upgrade "websocket";
      proxy_set_header Connection "Upgrade";
      nginx.ingress.kubernetes.io/proxy-read-timeout 3600;  
      nginx.ingress.kubernetes.io/proxy-send-timeout 3600;

6、负载均衡

为方便上游服务器组的动态管理，Nginx Ingress 基于 Lua 实现了一致性哈希、基于子集的一致性哈希、轮询调度及峰值指数加权移动平均
通过客户端ip进行一致性hash的均衡算法

nginx.ingress.kubernetes.io/upstream-hash-by: "${remote_addr}"

通过请求uri进行一致性hash的均衡算法

nginx.ingress.kubernetes.io/upstream-hash-by: "${request_uri}"

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/load-balance: ip_hash
    nginx.ingress.kubernetes.io/server-snippet: |
      set $agentflag 0;
      if ($http_user_agent ~* "(Mobile)" ){
        set $agentflag 1;
      }
    nginx.ingress.kubernetes.io/upstream-hash-by: ip_hash
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ( $agentflag = 1 ) {
         proxy_pass http://default-cardinfo-homepage-80;
      }
  name: nginx-condition
  namespace: default
spec:
  rules:
  - host: stickyingress.example.com
    http:
      paths:
      - backend:
          serviceName: cardinfo-recommendation
          servicePort: 80
        path: /
  - host: mobile.example.com
    http:
      paths:
      - backend:
          serviceName: cardinfo-homepage
          servicePort: 80
        path: /

7、会话粘性

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: web-nginxbar-org
    annotations:
        nginx.ingress.kubernetes.io/affinity: "cookie"
        nginx.ingress.kubernetes.io/session-cookie-name: "route"
        nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
        nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"

spec:
    rules:
    - host: web.nginxbar.org
      http:
        paths:
        - backend:
            serviceName: nginx-web
            servicePort: 8080
        path: /

8、四层负载均衡

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: tcp-services
  namespace: ingress-nginx
data:     #通过data字段添加四层反向代理的service
  "6666": default/nginx-06:1992  #key为代理端口

9、https配置

# 创建TLS证书
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /data/apps/certs/dashboard.key -out /data/apps/certs/dashboard.crt -subj "/CN=dashboard.nginxbar.org/O=dashboard.nginxbar.org"
kubectl -n kube-system  create secret tls ingress-secret --key /data/apps/certs/dashboard.key --cert /data/apps/certs/dashboard.crt

# 创建HTTPS服务
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: dashboard-ingress
    namespace: kube-system
    annotations:
        nginx.ingress.kubernetes.io/ingress.class: nginx
        #设置当前虚拟主机支持 HTTPS 请求时，是否将 HTTP 的请求强制跳转到 HTTPS 端口，全局默认为 true
        nginx.ingress.kubernetes.io/ssl-redirect: true
        # 使用HTTPS协议代理后端服务器
        nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
        # 启用SSL透传
        nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
    tls:
    - hosts:
        - dashboard.nginxbar.org
        secretName: ingress-secret
    rules:
        - host: dashboard.nginxbar.org
          http:
            paths:
            - path: /
              backend:
                serviceName: kubernetes-dashboard
                servicePort: 443

10、客户端地址记录

可以在特定ingress资源下通过metadata.annotations字段下通过nginx.ingress.kubernetes.io/configuration-snippet 参数来定义

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-dt
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

11、全局常规参数

data:
  multi_accept: on;
  use: epoll;
  user: www;
  worker_connections: 65535;
  worker_cpu_affinity: auto;
  worker_processes: auto;
  worker_rlimit_nofile: 300000;

  # 把真实IP地址传给后端
  compute-full-forwarded-for: "true"
  forwarded-for-header: "X-Forwarded-For"
  use-forwarded-headers: "true"
  # 关闭版本显示
  server-tokens: "false"
  # 客户端请求头的缓冲区大小 
  client-header-buffer-size: "512k"
  # 设置用于读取大型客户端请求标头的最大值number和size缓冲区
  large-client-header-buffers: "16 512k"
  # 读取客户端请求body的缓冲区大小
  client-body-buffer-size: "968k"
  # 代理缓冲区大小
  proxy-buffer-size: "1024k"
  # 代理body大小
  proxy-body-size: "50m"
  # 服务器名称哈希大小
  server-name-hash-bucket-size: "128"
  # map哈希大小
  map-hash-bucket-size: "128"
  # SSL加密套件
  ssl-ciphers: "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA"
  # ssl 协议
  ssl-protocols: "TLSv1 TLSv1.1 TLSv1.2"
#定义json 访问日志格式
log-format-upstream: '{"time": "$time_iso8601", "remote_addr": "$proxy_protocol_addr", "x-forward-for": "$proxy_add_x_forwarded_for", "request_id": "$req_id", "remote_user": "$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status":$status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri", "request_query": "$args", "request_length": $request_length, "duration": $request_time,"method": "$request_method", "http_referrer": "$http_referer", "http_user_agent": "$http_user_agent"}'

12、Nginx Ingress 实现金丝雀发布

• 使用 Header 或 Cookie 来标识不同类型的用户
• 切一定比例的流量给新版本

12.1）注释说明

nginx.ingress.kubernetes.io/canary 启用“金丝雀”发布功能
nginx.ingress.kubernetes.io/canary-by-header 表示如果请求头中包含这里指定的 header 名称，并且值为 always 的话，就将该请求转发给该 Ingress 定义的对应后端服务；如果值为 never 就不转发，可以用于回滚到旧版；如果是其它值则忽略该 annotation
nginx.ingress.kubernetes.io/canary-by-header-value 这个可以作为 canary-by-header的补充，允许指定请求头的值可以自定义成其它值，不再只能是 always 或 never；当请求头的值命中这里的自定义值时，请求将会转发给该 Ingress 定义的对应后端服务
nginx.ingress.kubernetes.io/canary-by-cookie: 这个与 canary-by-header 类似，只是这个用于 cookie，同样也是只支持 always 和 never 的值。
nginx.ingress.kubernetes.io/canary-weight: 表示 Canary Ingress 所分配流量的比例的百分比，取值范围 [0-100]，比如设置为 10，意思是分配 10% 的流量给 Canary Ingress 对应的后端服务。

优先顺序如下： canary-by-header -> canary-by-cookie -> canary-weight

12.2）实践

部署v1版本

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
      version: v1
  template:
    metadata:
      labels:
        app: nginx
        version: v1
    spec:
      containers:
      - name: nginx
        image: "openresty/openresty:centos"
        ports:
        - name: http
          protocol: TCP
          containerPort: 80
        volumeMounts:
        - mountPath: /usr/local/openresty/nginx/conf/nginx.conf
          name: config
          subPath: nginx.conf
      volumes:
      - name: config
        configMap:
          name: nginx-v1
---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: nginx
    version: v1
  name: nginx-v1
data:
  nginx.conf: |-
    worker_processes  1;
    events {
        accept_mutex on;
        multi_accept on;
        use epoll;
        worker_connections  1024;
    }
    http {
        ignore_invalid_headers off;
        server {
            listen 80;
            location / {
                access_by_lua '
                    local header_str = ngx.say("nginx-v1")
                ';
            }
        }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-v1
spec:
  type: ClusterIP
  ports:
  - port: 80
    protocol: TCP
    name: http
  selector:
    app: nginx
    version: v1

部署v2版本

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
      version: v2
  template:
    metadata:
      labels:
        app: nginx
        version: v2
    spec:
      containers:
      - name: nginx
        image: "openresty/openresty:centos"
        ports:
        - name: http
          protocol: TCP
          containerPort: 80
        volumeMounts:
        - mountPath: /usr/local/openresty/nginx/conf/nginx.conf
          name: config
          subPath: nginx.conf
      volumes:
      - name: config
        configMap:
          name: nginx-v2
---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: nginx
    version: v2
  name: nginx-v2
data:
  nginx.conf: |-
    worker_processes  1;
    events {
        accept_mutex on;
        multi_accept on;
        use epoll;
        worker_connections  1024;
    }
    http {
        ignore_invalid_headers off;
        server {
            listen 80;
            location / {
                access_by_lua '
                    local header_str = ngx.say("nginx-v2")
                ';
            }
        }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-v2
spec:
  type: ClusterIP
  ports:
  - port: 80
    protocol: TCP
    name: http
  selector:
    app: nginx
    version: v2

再创建一个 Ingress，对外暴露服务，指向 v1 版本的服务:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: canary.example.com
    http:
      paths:
      - backend:
          serviceName: nginx-v1
          servicePort: 80
        path: /

基于 Header 的流量切分

创建 Canary Ingress，指定 v2 版本的后端服务，且加上一些 annotation，实现仅将带有名为 Region 且值为 cd 或 sz 的请求头的请求转发给当前 Canary Ingress，模拟灰度新版本给成都和深圳地域的用户:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "Region"
    nginx.ingress.kubernetes.io/canary-by-header-pattern: "cd|sz"
  name: nginx-canary
spec:
  rules:
  - host: canary.example.com
    http:
      paths:
      - backend:
          serviceName: nginx-v2
          servicePort: 80
        path: /

基于 Cookie 的流量切分

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    #这里以模拟灰度成都地域用户为例，仅将带有名为 user_from_cd 的 cookie 的请求转发给当前 Canary Ingress
    nginx.ingress.kubernetes.io/canary-by-cookie: "user_from_cd" 
  name: nginx-canary
spec:
  rules:
  - host: canary.example.com
    http:
      paths:
      - backend:
          serviceName: nginx-v2
          servicePort: 80
        path: /

基于服务权重的流量切分

基于服务权重的 Canary Ingress 就简单了，直接定义需要导入的流量比例，这里以导入 10% 流量到 v2 版本为例

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
  name: nginx-canary
spec:
  rules:
  - host: canary.example.com
    http:
      paths:
      - backend:
          serviceName: nginx-v2
          servicePort: 80
        path: /