Kubernetes 审计(Audit)提供了安全相关的时序操作记录,支持日志和 webhook 两种格式,并可以通过审计策略自定义事件类型。

审计日志

通过配置 kube-apiserver 的下列参数开启审计日志

  • audit-log-path:审计日志路径
  • audit-log-maxage:旧日志最长保留天数
  • audit-log-maxbackup:旧日志文件最多保留个数
  • audit-log-maxsize:日志文件最大大小(单位 MB),超过后自动做轮转(默认为 100MB)

每条审计记录包括两行

  • 请求行包括:唯一 ID 和请求的元数据(如源 IP、用户名、请求资源等)
  • 响应行包括:唯一 ID(与请求 ID 一致)和响应的元数据(如 HTTP 状态码)

比如,admin 用户查询默认 namespace 的 Pod 列表的审计日志格式为

  1. 2017-03-21T03:57:09.106841886-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" ip="127.0.0.1" method="GET" user="admin" groups="\"system:masters\",\"system:authenticated\""as="<self>"asgroups="<lookup>"namespace="default"uri="/api/v1/namespaces/default/pods"
  2. 2017-03-21T03:57:09.108403639-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" response="200"

审计策略

v1.7 + 支持实验性的高级审计特性,可以自定义审计策略(选择记录哪些事件)和审计存储后端(日志和 webhook)等。开启方法为

  1. kube-apiserver ... --feature-gates=AdvancedAuditing=true

注意开启 AdvancedAuditing 后,日志的格式有一些修改,如新增了 stage 字段(包括 RequestReceived,ResponseStarted ,ResponseComplete,Panic 等)。

审计策略

审计策略选择记录哪些事件,设置方法为

  1. kube-apiserver ... --audit-policy-file=/etc/kubernetes/audit-policy.yaml

其中,设计策略的配置格式为

  1. rules:
  2. # Don't log watch requests by the"system:kube-proxy" on endpoints or services
  3. - level: None
  4. users: ["system:kube-proxy"]
  5. verbs: ["watch"]
  6. resources:
  7. - group: "" # core API group
  8. resources: ["endpoints", "services"]
  9. # Don't log authenticated requests to certain non-resource URL paths.
  10. - level: None
  11. userGroups: ["system:authenticated"]
  12. nonResourceURLs:
  13. - "/api*" # Wildcard matching.
  14. - "/version"
  15. # Log the request body of configmap changes in kube-system.
  16. - level: Request
  17. resources:
  18. - group: "" # core API group
  19. resources: ["configmaps"]
  20. # This rule only applies to resources in the "kube-system" namespace.
  21. # The empty string "" can be used to select non-namespaced resources.
  22. namespaces: ["kube-system"]
  23. # Log configmap and secret changes in all other namespaces at the Metadata level.
  24. - level: Metadata
  25. resources:
  26. - group: "" # core API group
  27. resources: ["secrets", "configmaps"]
  28. # Log all other resources in core and extensions at the Request level.
  29. - level: Request
  30. resources:
  31. - group: "" # core API group
  32. - group: "extensions" # Version of group should NOT be included.
  33. # A catch-all rule to log all other requests at the Metadata level.
  34. - level: Metadata

在生产环境中,推荐参考 GCE 审计策略 配置。

审计存储后端

审计存储后端支持两种方式

  • 日志,配置 --audit-log-path 开启,格式为
  1. 2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="RequestReceived" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\""as="<self>"asgroups="<lookup>"namespace="kube-system"uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager"response="<deferred>"
  2. 2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="ResponseComplete" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\""as="<self>"asgroups="<lookup>"namespace="kube-system"uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager"response="200"
  • webhook,配置 --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig --audit-webhook-mode=batch 开启,其中 audit-webhook-mode 支持 batch 和 blocking 两种格式,而 webhook 配置文件格式为
  1. # clusters refers to the remote service.
  2. clusters:
  3. - name: name-of-remote-audit-service
  4. cluster:
  5. certificate-authority: /path/to/ca.pem # CA for verifying the remote service.
  6. server: https://audit.example.com/audit # URL of remote service to query. Must use 'https'.
  7. # users refers to the API server's webhook configuration.
  8. users:
  9. - name: name-of-api-server
  10. user:
  11. client-certificate: /path/to/cert.pem # cert for the webhook plugin to use
  12. client-key: /path/to/key.pem # key matching the cert
  13. # kubeconfig files require a context. Provide one for the API server.
  14. current-context: webhook
  15. contexts:
  16. - context:
  17. cluster: name-of-remote-audit-service
  18. user: name-of-api-sever
  19. name: webhook

所有的事件以 JSON 格式 POST 给 webhook server,如

  1. {
  2. "kind": "EventList",
  3. "apiVersion": "audit.k8s.io/v1alpha1",
  4. "items": [
  5. {
  6. "metadata": {
  7. "creationTimestamp": null
  8. },
  9. "level": "Metadata",
  10. "timestamp": "2017-06-15T23:07:40Z",
  11. "auditID": "4faf711a-9094-400f-a876-d9188ceda548",
  12. "stage": "ResponseComplete",
  13. "requestURI": "/apis/rbac.authorization.k8s.io/v1beta1/namespaces/kube-public/rolebindings/system:controller:bootstrap-signer",
  14. "verb": "get",
  15. "user": {
  16. "username": "system:apiserver",
  17. "uid": "97a62906-e4d7-4048-8eda-4f0fb6ff8f1e",
  18. "groups": [
  19. "system:masters"
  20. ]
  21. },
  22. "sourceIPs": [
  23. "127.0.0.1"
  24. ],
  25. "objectRef": {
  26. "resource": "rolebindings",
  27. "namespace": "kube-public",
  28. "name": "system:controller:bootstrap-signer",
  29. "apiVersion": "rbac.authorization.k8s.io/v1beta1"
  30. },
  31. "responseStatus": {
  32. "metadata": {},
  33. "code": 200
  34. }
  35. }
  36. ]
  37. }