20221017-Cilium Operator Releated[Cilium Identity] - 图1

    • 1.FAQ：Identity Management in the Cluster ```properties 1.在Slack上看到一个question：https://cilium.slack.com/archives/C1MATJ5U5/p1644325759701449 其中描述了一段内容是关于Kubernetes中的Pod对象和cilium的identity的对象。其中identity是有cilium-operator来负责回收的，而这个默认的回收时间是15min。 我们可以通过：https://docs.cilium.io/en/v1.9/cmdref/cilium-operator-generic/来查看：[cilium 为1.11.1版本] root@bpf1:~# kubectl get crd NAME CREATED AT ciliumclusterwidenetworkpolicies.cilium.io 2022-01-24T07:31:34Z ciliumendpoints.cilium.io 2022-01-24T07:31:33Z ciliumexternalworkloads.cilium.io 2022-01-24T07:31:33Z ciliumidentities.cilium.io 2022-01-24T07:31:33Z ciliumnetworkpolicies.cilium.io 2022-01-24T07:31:34Z ciliumnodes.cilium.io 2022-01-24T07:31:33Z root@bpf1:~#

    root@bpf1:~# kubectl -nkube-system exec -it cilium-operator-7c5c674b74-8wncv — cilium-operator-generic —help | grep identity-gc-interval —identity-gc-interval duration GC interval for security identities (default 15m0s)

    我们可以看到默认是15m0s的一个回收周期。 root@bpf1:~# kubectl explain ciliumidentities KIND: CiliumIdentity VERSION: cilium.io/v2

    DESCRIPTION: CiliumIdentity is a CRD that represents an identity managed by Cilium. It is intended as a backing store for identity allocation, acting as the global coordination backend, and can be used in place of a KVStore (such as etcd). The name of the CRD is the numeric identity and the labels on the CRD object are the the kubernetes sourced labels seen by cilium. This is currently the only label source possible when running under kubernetes. Non-kubernetes labels are filtered but all labels, from all sources, are places in the SecurityLabels field. These also include the source and are used to define the identity. The labels under metav1.ObjectMeta can be used when searching for CiliumIdentity instances that include particular labels. This can be done with invocations such as: kubectl get ciliumid -l ‘foo=bar’ Each node using a ciliumidentity updates the status field with it’s name and a timestamp when it first allocates or uses an identity, and periodically after that. It deletes its entry when no longer using this identity. cilium-operator uses the list of nodes in status to reference count users of this identity, and to expire stale usage.

    2. - [x] **2.Cilium Identity Overview**
    3. ```properties
    4. Identity：[简单说来：一个CIlium Identity在一个Cluster 中唯一切标识一组enpoints（pods）,此对象由cilium-operator来控制负责创建和回收]
    5. An identity is identified by Labels and is given a cluster wide unique identifier. The endpoint is assigned the identity which matches the endpoint’s Security Relevant Labels, i.e. all endpoints which share the same set of Security Relevant Labels will share the same identity. This concept allows to scale policy enforcement to a massive number of endpoints as many individual endpoints will typically share the same set of security Labels as applications are scaled.
    7. What is an Identity?
    8. The identity of an endpoint is derived based on the Labels associated with the pod or container which are derived to the endpoint. When a pod or container is started, Cilium will create an endpoint based on the event received by the container runtime to represent the pod or container on the network. As a next step, Cilium will resolve the identity of the endpoint created. Whenever the Labels of the pod or container change, the identity is reconfirmed and automatically modified as required.
    9. 我们可以通过在cilium中查看：
    10. 我们的环境中
    11. root@bpf1:~# kubectl get pods -o wide 
    12. NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
    13. indti-74cddb88cd-7qd7h   1/1     Running   0          18h   10.0.2.27    bpf1   <none>           <none>
    14. indti-74cddb88cd-ffc2w   1/1     Running   0          18h   10.0.2.204   bpf1   <none>           <none>
    15. indti-74cddb88cd-rwbpv   1/1     Running   0          18h   10.0.2.88    bpf1   <none>           <none>
    16. root@bpf1:~# 
    17. root@bpf1:/home/cilium# cilium endpoint list 
    18. ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6   IPv4         STATUS   
    19.            ENFORCEMENT        ENFORCEMENT                                                                                                                   
    20. 850        Disabled           Disabled          59511      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system          10.0.2.182   ready   
    21.                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
    22.                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns                                                          
    23.                                                            k8s:io.kubernetes.pod.namespace=kube-system                                                              
    24.                                                            k8s:k8s-app=kube-dns                                                                                     
    25. [--------------------------------------------------------------------------identity:3829 -------------------------------------------------------------------]
    26. 933        Disabled           Disabled          3829       k8s:app=indti                                                                       10.0.2.88    ready   
    27.                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default                                   
    28.                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
    29.                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
    30.                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
    31. 1329       Disabled           Disabled          3829       k8s:app=indti                                                                       10.0.2.27    ready   
    32.                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default                                   
    33.                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
    34.                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
    35.                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
    36. 2139       Disabled           Disabled          3829       k8s:app=indti                                                                       10.0.2.204   ready   
    37.                                                            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default                                   
    38.                                                            k8s:io.cilium.k8s.policy.cluster=default                                                                 
    39.                                                            k8s:io.cilium.k8s.policy.serviceaccount=default                                                          
    40.                                                            k8s:io.kubernetes.pod.namespace=default                                                                  
    41. [--------------------------------------------------------------------------identity:3829 -------------------------------------------------------------------]
    42. 2478       Disabled           Disabled          4          reserved:health                                                                     10.0.2.241   ready   
    43. 3945       Disabled           Disabled          1          k8s:node-role.kubernetes.io/control-plane                                                        ready   
    44.                                                            k8s:node-role.kubernetes.io/master                                                                       
    45.                                                            k8s:node.kubernetes.io/exclude-from-external-load-balancers                                              
    46.                                                            reserved:host                                                                                            
    47. root@bpf1:/home/cilium# 
    49. root@bpf1:/home/cilium# cilium identity list 
    50. ID      LABELS
    51. 1       reserved:host
    52.         reserved:kube-apiserver
    53. 2       reserved:world
    54. 3       reserved:unmanaged
    55. 4       reserved:health
    56. 5       reserved:init
    57. 6       reserved:remote-node
    58. 7       reserved:kube-apiserver
    59.         reserved:remote-node
    60. |# identity:3829:]
    61. |3829    k8s:app=indti
    62. |        k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
    63. |        k8s:io.cilium.k8s.policy.cluster=default
    64. |        k8s:io.cilium.k8s.policy.serviceaccount=default
    65. |        k8s:io.kubernetes.pod.namespace=default
    66. |# identity:3829:]
    67. 59511   k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
    68.         k8s:io.cilium.k8s.policy.cluster=default
    69.         k8s:io.cilium.k8s.policy.serviceaccount=coredns
    70.         k8s:io.kubernetes.pod.namespace=kube-system
    71.         k8s:k8s-app=kube-dns
    72. root@bpf1:/home/cilium# 
    74. 2.同时我们还可以观察到还有几个没有什么(pod)对象与其对应的cilium identity的对象ID：这里在官网中有做介绍：https://docs.cilium.io/en/v1.11/concepts/terminology/#special-identities
    75. 参考下图：

    86053ded09ffc3efd096c9e555bab29.png

    • 3.Cilium Identity 设计缘由 ```properties https://docs.cilium.io/en/v1.11/concepts/security/identity/ Container management systems such as Kubernetes deploy a networking model which assigns an individual IP address to each pod (group of containers). This ensures simplicity in architecture, avoids unnecessary network address translation (NAT) and provides each individual container with a full range of port numbers to use. The logical consequence of this model is that depending on the size of the cluster and total number of pods, the networking layer has to manage a large number of IP addresses. // 解读：Kubernetes{CNI}基本设计原理。为每一个Pod分配一个独立的ip地址，这样每一个地址就可以拥有全量的端口，创建不同的套接字。

    Traditionally security enforcement architectures have been based on IP address filters. Let’s walk through a simple example: If all pods with the label role=frontend should be allowed to initiate connections to all pods with the label role=backend then each cluster node which runs at least one pod with the label role=backend must have a corresponding filter installed which allows all IP addresses of all role=frontend pods to initiate a connection to the IP addresses of all local role=backend pods. All other connection requests should be denied. This could look like this: If the destination address is 10.1.1.2 then allow the connection only if the source address is one of the following [10.1.2.2,10.1.2.3,20.4.9.1]. //解读：network policy的基本原理：其中分为两种pod，一种是前端pod，一种是后端pod，为了安全起见，我们可以设置策略允许role=fronted 访问 role=backend pod。

    Every time a new pod with the label role=frontend or role=backend is either started or stopped, the rules on every cluster node which run any such pods must be updated by either adding or removing the corresponding IP address from the list of allowed IP addresses. In large distributed applications, this could imply updating thousands of cluster nodes multiple times per second depending on the churn rate of deployed pods. Worse, the starting of new role=frontend pods must be delayed until all servers running role=backend pods have been updated with the new security rules as otherwise connection attempts from the new pod could be mistakenly dropped. This makes it difficult to scale efficiently. //传统的设计原理： 传统的设计实际上根据添加对应的列表中添加或是删除ip地址。在规模较小，pod变化不大的情况下，这种设计当然也是可行的，但是随着规模的扩大，那么在单位时间内的pod创建和删除就会变得非常多，那么我们此时去维护这样的一个list的时候，就可能需要进行大量的规则的变更，此时是涉及到众多节点的问题。 此时：需要等所有的role=backend 的pod都知道这个list更新完毕以后，才能正式的启动，否则可能会由于没有更新完而使得新的role=fronted的pod过来的流量而被丢弃，因为此时规则还没有更新完毕。这样就使得集群的规模会受到一定的限制。

    In order to avoid these complications which can limit scalability and flexibility, Cilium entirely separates security from network addressing. Instead, security is based on the identity of a pod, which is derived through labels. This identity can be shared between pods. This means that when the first role=frontend pod is started, Cilium assigns an identity to that pod which is then allowed to initiate connections to the identity of the role=backend pod. The subsequent start of additional role=frontend pods only requires to resolve this identity via a key-value store, no action has to be performed on any of the cluster nodes hosting role=backend pods. The starting of a new pod must only be delayed until the identity of the pod has been resolved which is a much simpler operation than updating the security rules on all other cluster nodes. // 解读：为了避免传统的设计模式，在cilium中引入 cilium idetity以后：我们的policy将对cilium indentity作为一个整体来进行操作，而不再是单纯的考察ip地址了。这样就变成了：拥有role=fronted的cilium identity的pod和拥有role =backend的cilium identity的pod进行通信，那么此时只需要解析到对应的cilium即可。这样从这个角度上来看，维护的难易度就简单多了。 如下图所示：

    1. ![](https://cdn.nlark.com/yuque/0/2022/png/5357487/1644655563734-72b7c53e-742e-4a25-a8b2-eea25be77847.png#clientId=ubc86d66a-7ec0-4&crop=0&crop=0&crop=1&crop=1&from=paste&id=u443958e0&margin=%5Bobject%20Object%5D&originHeight=632&originWidth=1346&originalType=url&ratio=1&rotation=0&showTitle=false&status=done&style=none&taskId=u86698373-68be-432d-90af-533598cbde0&title=)
    2. ```properties
    3. 我们是以cilium identity来作为基本操作单位，所以只需要维护这个identity即可。