配置 ingress nginx 日志持久化和 ssl 证书

修改日志格式并持久化

ingress-nginx 默认访问日志保存在 /var/log/nginx/ 目录下。

ingress-nginx 默认日志格式是普通日志格式，为了 efk 方便分析，可以修改为 json 格式的日志。

ingress rule 根据域名设置访问日志

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/enable-access-log: "true"
    nginx.ingress.kubernetes.io/configuration-snippet: |
       access_log /var/log/nginx/test.example.com.access.log upstreaminfo if=$loggable;
       error_log  /var/log/nginx/test.example.com.error.log;
spec:
  rules:
  - host: test.example.com
    http:
      paths:
      - backend:
          serviceName: demo
          servicePort: 8080

配置 json 格式日志

https://raw.githubusercontent.com/Simontage/k8s/main/ingress-nginx-demo/nginx-configuration-json-cm.yaml

kind:  ConfigMap
apiVersion:  v1
metadata: 
  name:  nginx-configuration
  namespace:  ingress-nginx
  labels: 
    app.kubernetes.io/name:  ingress-nginx
    app.kubernetes.io/part-of:  ingress-nginx
data:
  log-format-escape-json: "true"
  log-format-upstream: |-
    {
        "remote_addr":"$remote_addr",
        "remote_port":"$remote_port",
        "http_x_forwarded_for":"$http_x_forwarded_for",
        "time_local":"$time_local",
        "server_protocol":"$server_protocol",
        "request_method":"$request_method",
        "scheme":"$scheme",
        "host":"$host",
        "request_uri":"$request_uri",
        "url":"$scheme://$host$request_uri",
        "status":"$status",
        "body_bytes_sent":"$body_bytes_sent",
        "request_time":"$request_time",
        "http_referer":"$http_referer",
        "http_user_agent":"$http_user_agent",
        "upstream_addr":"$upstream_addr",
        "upstream_response_time":"$upstream_response_time",
        "upstream_status":"$upstream_status"
    }
  client_max_body_size: "2000m"

文本日志格式：

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  client-header-buffer-size: "512k"
  large-client-header-buffers: "4 512k"
  client-body-buffer-size: "128k"
  proxy-buffer-size: "256k"
  client-body-buffer-size: "128k"
  proxy-body-size: "50m"
  server-name-hash-bucket-size: "128"
  map-hash-bucket-size: "128"
  ssl-ciphers: "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA"
  ssl-protocols: "TLSv1 TLSv1.1 TLSv1.2"
  log-format-upstream: '[$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent $request_time "$http_referer" $host DIRECT/$upstream_addr $upstream_http_content_type "$http_user_agent" "$http_x_forwarded_for" $request_length [$proxy_upstream_name] $upstream_response_length $upstream_response_time $upstream_status $req_id'

配置日志持久化保存

ingress-nginx 添加 pvc 用于 ingress 日志存储，StorageClass 需要提前创建好。

kubectl apply -f ingress-nfs.yaml

apiVersion: v1      
kind: PersistentVolumeClaim      
metadata:      
  name: ingress-nfs      
spec:      
  accessModes:      
  - ReadWriteMany      
  resources:      
    requests:      
      storage: 10Gi      
  storageClassName: normal-nfs-storage

ingress 添加挂载

KUBE_EDITER='vim'
kubectl edit deployment/nginx-ingress-controller -n ingress-nginx

配置证书

创建 secret

apiVersion: v1
kind: Secret
data:
  tls.crt: base64 encoded cert
  tls.key: base64 encoded key
metadata:
  name: yellowdog-secret
  namespace: default
type: Opaque

因为编码的密码太长，这里直接使用命令行进行创建，操作比较简单。

kubectl create secret tls yellowdog-secret --cert=证书.crt --key=证书.key

查看创建好的 secret

kubectl describe secret yellowdog-secret

创建 ingress 规则

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: yellowdog-ingress
  namespace: app
spec:
  tls:
  - hosts:
    - www.yellowdog.com
    secretName: yellowdog-secret
  rules:
  - host: www.yellowdog.com
    http:
      paths:
      - path:
        backend:
          serviceName: www
          servicePort: 80

参考文档

ingress-nginx 官方文档

kubernetes nginx ingress controller日志efk收集分析

Ingress(Nginx)日志持久化与可视化(多图预警)

Ingress

kubernetes-api

Kubernetes Reference