3、控制器 - 3.10、PodSecurityPolicy - 《Kubernetes》

PodSecurityPolicy是集群级别的Pod安全策略，自动为集群中的Pod和Volume设置Security Context。

Admission Controller（准入控制器）拦截对 kube-apiserver 的请求，拦截发生在请求的对象被持久化之前，但是在请求被验证和授权之后。这样我们就可以查看请求对象的来源，并验证需要的内容是否正确。通过将它们添加到 kube-apiserver 的—enable-admission-plugins参数中来启用准入控制器。所以如果我们要使用PSP，我们就需要在kube-apiserver中添加起参数，如下：

--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,PodSecurityPolicy

其他插件是kubernetes官方推荐的插件。

然后重启kube-apiserver：

# systemctl daemon-reload
# systemctl restart kube-apiserver

这时候就已经启动了PSP控制器，如果我们现在创建Pod试试：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent

然后我们kubectl get pod的时候发现并没有pod。我们再查看deploy的状态，如下：

# kubectl get deployments.
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
nginx                  0/1     0            0           117s
# kubectl get replicasets.
NAME                              DESIRED   CURRENT   READY   AGE
nginx-55fc968d9                   1         0         0       2m17s

我们看到replicaset和deploy都创建成功了，但是replicaset并没有创建pod，这是因为我们集群现在缺少安全策略，所以创建新的Pod不会成功，这时我们就需要使用ServiceAccount。

正常情况下，我们并不会直接创建Pod，都是通过其他控制器比如Deployment、StatefulSet等来创建Pod。我们现在要使用PSP，需要配置kube-controller-manager来为其包含的每个控制器单独使用ServiceAccount，我们可以通过以下参数来添加，如下：

--use-service-account-credentials=true

然后重启controller-manager：

# systemctl daemon-reload
# systemctl restart kube-controller-manager

然后kubenetes就会自动生成如下一些SA，这些SA就指定了哪个控制器可以解析哪些策略：

# kubectl get serviceaccount -n kube-system | egrep -o '[A-Za-z0-9-]+-controller'
attachdetach-controller
certificate-controller
clusterrole-aggregation-controller
cronjob-controller
daemon-set-controller
deployment-controller
disruption-controller
endpoint-controller
expand-controller
job-controller
namespace-controller
node-controller
pv-protection-controller
pvc-protection-controller
replicaset-controller
replication-controller
resourcequota-controller
service-account-controller
service-controller
statefulset-controller
traefik-ingress-controller
ttl-controller

PSP提供一种声明式的方式，用于表达运行的用户和ServiceAccount在我们集群中创建的内容。其主要的策略有：

控制项	说明
privileged	运行特权容器
defaultAddCapabilities	可添加到容器的Capabilities
requiredDropCapabilities	会从容器中删除的Capabilities
volumes	控制容器可以使用哪些volume
hostNetwork	host网络
hostPorts	允许的host端口列表
hostPID	使用host PID namespace
hostIPC	使用host IPC namespace
seLinux	SELinux Context
runAsUser	user ID
supplementalGroups	允许的补充用户组
fsGroup	volume FSGroup
readOnlyRootFilesystem	只读根文件系统

在上面的示例中，我们需要两个策略：
1、提供限制访问的默认策略，保证使用特权设置无法创建Pod；
2、提升许可策略，允许将特权设置用于某些Pod，比如允许在特定命名空间下创建Pod；

首先，创建一个默认策略：
psp-restrictive.yaml

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: restrictive
spec:
  privileged: false
  hostNetwork: false
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  hostPID: false
  hostIPC: false
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - 'configMap'
  - 'downwardAPI'
  - 'emptyDir'
  - 'persistentVolumeClaim'
  - 'secret'
  - 'projected'
  allowedCapabilities:
  - '*'

然后直接创建这个PSP对象：

# kubectl apply -f psp-restrictive.yaml 
podsecuritypolicy.policy/restrictive created
# kubectl get psp
NAME          PRIV    CAPS   SELINUX    RUNASUSER   FSGROUP    SUPGROUP   READONLYROOTFS   VOLUMES
restrictive   false   *      RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            configMap,downwardAPI,emptyDir,persistentVolumeClaim,secret,projected

其次，创建一个提升策略，用于那些需要提升权限的Pod，比如kube-proxy，它就需要hostNetwork权限：
psp-permissive.yaml

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: permissive
spec:
  privileged: true
  hostNetwork: true
  hostIPC: true
  hostPID: true
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  hostPorts:
  - min: 0
    max: 65535
  volumes:
  - '*'

然后创建这个PSP对象：

# kubectl apply -f psp-permissive.yaml 
podsecuritypolicy.policy/permissive created
# kubectl get psp
NAME          PRIV    CAPS   SELINUX    RUNASUSER   FSGROUP    SUPGROUP   READONLYROOTFS   VOLUMES
permissive    true           RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            *
restrictive   false   *      RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            configMap,downwardAPI,emptyDir,persistentVolumeClaim,secret,projected

但是仅仅部署了这两个策略是不够的，我们RBAC进行授权，不然我们的Pod还是不能创建成功。RBAC确定一个ServiceAccount可以使用的策略，如果使用ClusterRoleBinding可以为ServiceAccount提供限制性策略（restrictive）的访问，如果使用RoleBinding可以为SeriveAccount提供虚空策略的访问。

首先创建允许使用restrictive策略的ClusterRole，然后再创建一个ClusterRoleBinding将所有控制器的ServiceAccount进行绑定：
psp-restrictive-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: psp-restrictive
rules:
- apiGroups:
  - extensions
  resources:
  - podsecruritypolicies
  resourceNames:
  - restrictive
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: psp-default
subjects:
- kind: Group
  name: system:serviceaccounts
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: psp-restrictive
  apiGroup: rbac.authorization.k8s.io

然后创建RBAC资源对象：

# kubectl apply -f  psp-restrictive-rbac.yaml 
clusterrole.rbac.authorization.k8s.io/psp-restrictive created
clusterrolebinding.rbac.authorization.k8s.io/psp-default created

然后我们可以看到外面刚开始创建的Pod现在可以创建了：

# kubectl get pod
NAME                                    READY   STATUS    RESTARTS   AGE
nginx-55fc968d9-qn2vr                   1/1     Running   0          17m

但是如果我们现在给这个Deployment清单加一个hostNetwork=true这个特权，观察Pod是否能够创：
nginx-deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
      hostNetwork: true

然后我们再次创建这个Deployment：

# kubectl apply -f nginx-deploy.yaml 
deployment.apps/nginx created
# kubectl get pod
NAME                                    READY   STATUS    RESTARTS   AGE
# kubectl get deployments.
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
nginx                  0/1     0            0           12s
# kubectl get replicasets.
NAME                              DESIRED   CURRENT   READY   AGE
nginx-5cd65fd4c6                  1         0         0       18s

我们可以看到Pod并未被创建，我们describe一个replicaset，发现如下日志：

# kubectl describe rs nginx-5cd65fd4c6
......
Events:
  Type     Reason        Age                   From                   Message
  ----     ------        ----                  ----                   -------
  Warning  FailedCreate  41s (x16 over 3m24s)  replicaset-controller  Error creating: pods "nginx-5cd65fd4c6-" is forbidden: unable to validate against any pod security policy: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used]

提示我们hostNetwork不允许被使用。

但是在某些情况下，我们需要在某个命名空间下使用特权，这时候我们就可以创建一个允许使用特权的ClusterRole，但是这里我们为特定ServiceAccount设置RoleBinding，如下：
psp-permissive-rbac.yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-permissive
rules:
- apiGroups:
  - extensions
  resources:
  - podsecuritypolicies
  resourceNames:
  - permissive
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: psp-permissive
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp-permissive
subjects:
- kind: ServiceAccount
  name: daemon-set-controller
  namespace: kube-system
- kind: ServiceAccount
  name: replicaset-controller
  namespace: kube-system
- kind: ServiceAccount
  name: job-controller
  namespace: kube-system

上面定义了对kube-system中的daemonset，replicaset，job拥有特权创建Pod。
然后我们创建RBAC资源清单：

# kubectl apply -f psp-permissive-rbac.yaml 
clusterrole.rbac.authorization.k8s.io/psp-permissive created
rolebinding.rbac.authorization.k8s.io/psp-permissive created

现在我们定义一个测试Deployment：


apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
      hostNetwork: true

然后创建资源对象：

# kubectl apply -f nginx-deploy.yaml 
deployment.apps/nginx created
[root@ecs-5704-0003 kubernetes]# kubectl get pod -n kube-system 
NAME                                   READY   STATUS    RESTARTS   AGE
......
nginx-5cd65fd4c6-7pn9z                 1/1     Running   0          4s

然后我们可以看到Pod可以正常被创建。

另外还有一种特殊的需求，就是在某个命令空间下只有某个应用可以使用特权，那么针对这类应用就需要单独创建一个SA，然后和permissive策略镜像RoleBinding了，如下：
（1）、创建可以使用特权的SA

# kubectl create serviceaccount specialsa
serviceaccount/specialsa created

（2）、创建RoleBinding
specialsa-psp.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: specialsa-psp-permissive
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp-permissive
subjects:
- kind: ServiceAccount
  name: specialsa
  namespace: default

然后创建上面的RoleBinding对象：

# kubectl apply -f specialsa-psp.yaml 
rolebinding.rbac.authorization.k8s.io/specialsa-psp-permissive created

然后创建一个测试的Deployment：
ng-deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-hostnetwork-deploy
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
      hostNetwork: true
      serviceAccount: specialsa  # 注意这里使用的sa的权限绑定

然后创建资源对象：

# kubectl apply -f ng-deploy.yaml 
deployment.apps/nginx-hostnetwork-deploy created
# kubectl get pod
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-hostnetwork-deploy-7b65cf7bbd-g5wl5   1/1     Running   0          2s

然后可以发现在default命名空间下可以创建拥有特权的Pod了。