一、docker

1.1 docker概述

1.1.1 docker是什么

Docker 是一个开源的容器引擎，开发者可以打包他们的应用以及依赖包到一个可移植的镜像中，然后发布到任何 Linux 机器上，也可以实现虚拟化。容器是完全使用沙盒机制，相互之间不会有任何接口（类似 iPhone 的 app）。几乎没有性能开销，可以很容易地在机器和数据中心中运行。最重要的是，他们不依赖于任何语言、框架或操作系统。Docker 从 17.03 版本之后分为 CE（Community Edition: 社区版）和 EE（Enterprise Edition: 企业版），我们用社区版就可以
了
Docker核心技术:
Namespace: 实现Container的进程、网络、消息、文件系统和主机名的隔离
Cgroup: 实现对资源的配合和度量

1.1.2 docker优缺点

优点:
1. 快、小
比虚拟机小，比虚拟机快，管理操作(启动，停止，开始，重启等等) 都是以秒或毫秒为单位。Docker 启动一个容器实例时间很短，一两秒就可以启动一个实例。
2. 敏捷
像虚拟机一样敏捷，而且会更便宜，在 bare metal(裸机)上布署像点个按钮一样简单。
3. 灵活
将应用和系统“容器化”，不添加额外的操作系统，
4. 轻量
你会拥有足够的“操作系统”，仅需添加或减小镜像即可。在一台服务器上可以布署 100~1000 个Containers 容器。
5. 便宜
开源的，免费的，低成本的。由现代 Linux 内核支持并驱动。轻量的 Container 必定可以在一个物
理机上开启更多“容器”，注定比 VMs 要便宜。
6. 对于在笔记本电脑，数据中心的虚拟机，以及任何的云上，运行相同的没有变化的应用程序，IT
的发布速度更快。
7. 开发人员并不关心具体哪个 Linux 操作系统
使用 Docker，开发人员可以根据所有依赖关系构建相应的软件，针对他们所选择的操作系统。
然后，在部署时一切是完全一样的，因为一切都是基于 Docker Image 的运行。
8.Google，微软，亚马逊，IBM 等都支持 Docker。
9.Docker 支持 Unix/Linux 操作系统，也支持 Windows 或 Mac
缺点:
1. Docker 部署应用的时候，并不包含数据。日志，数据库等通常应放在 Docker 容器外。一个容器的镜像通常都很小，不适合存大量数据，存储可以通过外部挂载的方式使用。比如使用：NFS、ceph、glusterfs、ipsan，MFS 等
一句话：docker 只用于计算，存储交给别人。
2. 所有容器共用 linux kernel 资源，资源能否实现最大限度利用，所以在安全上也会存在漏洞

1.2 安装docker并配置镜像加速器

1.2.1 安装前初始化(选择执行)

注: 因为个体机器不一样，可以选择执行下面的命令
关闭selinux

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

配置yum源为阿里源

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
mv /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.backup
yum -y install wget yum-utils
yum-config-manager --add-repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/repo/epel-7.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum clean all && yum repolist

安装基础软件包和docker

yum -y install wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel openssh-server socat ipvsadm conntrack
yum -y install docker-ce
systemctl enable docker --now
docker version
systemctl status docker

1.2.2 开启包转发功能和修改内核参数

开启内核模块(临时生效，重启失效)

modprobe br_netfilter

开启内核模块（永久生效，选择执行）

tee /etc/rc.sysinit << 'EOF'
#!/bin/bash
for file in /etc/sysconfig/modules/*.modules; do
[ -x $file ] && $file
done
EOF
tee /etc/sysconfig/modules/br_netfilter.modules << 'EOF'
modprobe br_netfilter
EOF
chmod 755 /etc/sysconfig/modules/br_netfilter.modules

开启转发

tee /etc/sysctl.d/docker.conf << 'EOF'
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl -p /etc/sysctl.d/docker.conf

重启docker

systemctl restart docker

1.2.3设置镜像加速并部署应用测试

配置镜像加速

tee /etc/docker/daemon.json << 'EOF'
{
"registry-mirrors":["https://rsbud4vc.mirror.aliyuncs.com","https://registry.dockercn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hubmirror.c.163.com","http://qtid6917.mirror.aliyuncs.com","https://rncxm540.mirror.aliyuncs.com","https://e9yneuy4.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload && systemctl restart docker

拉取镜像测试

docker pull centos
docker images

1.3 通过docker部署应用

下载镜像到本地并查看

docker pull centos
docker images

传镜像包并解压

# docker load -i <包名>
docker load -i /opt/nginx.tar.gz

把镜像包打成离线包

# docker save -o <保存路径/镜像包名> <镜像名>
docker save -o centos.tar.gz centos

查看本地所有镜像并删除镜像

# 修改镜像tag
# docker tag <旧镜像名>:<旧tag> <新镜像名>:<tag>
docker tag centos:latest centos:v1
docker images
# docker rmi -f <镜像名字>
docker rmi -f centos:v1

基于镜像启动容器

# docker run -d -p <本机端口>:<容器端口> <镜像名>
# -d 在后台运行,-p  映射端口
docker run -d -p 80:80 nginx
# 查看所有正在运行的容器
docker ps
netstat -anutp | grep 80

访问端口测试

查看容器所有信息

#docker inspect <容器id|容器的名字>
docker inspect 794e04ecd9c9

停止容器，并查看

# 停止容器
# docker stop <容器id|容器的名字>
docker stop 794e04ecd9c9
# 查看所有，包括停止的容器
docker ps -a 
# 查看容器日志
# docker logs <容器id|容器的名字>
docker logs 794e04ecd9c9
# 启动已经停止的容器
# docker start <容器id|容器的名字>

1.4 基于dockerfile构建镜像

Dockerfile语法介绍
FROM：基础镜像，必须是可以下载下来的镜像，以什么镜像为基础

MAINTAINER： 指定镜像的作者信息

RUN： 指定在当前镜像构建过程中运行的命令，包含两种模式
#shell 模式
RUN
RUN yum install wget -y
#exec 模式
RUN [“executable”,”param1”,”param2”]
RUN [“/bin/bash”,”-c”,”echo hello”]

EXPOSE：用来声明端口的
帮助镜像使用者理解这个镜像服务的守护端口，以方便配置映射。在运行时使用随机端口映射时，也就是 docker run -P ，会自动随机映射 EXPOSE 的端口。
格式： EXPOSE <端口 1> [<端口 2>…]

CMD：为启动的容器指定默认要运行的程序，程序运行结束，容器也就结束。CMD 指令指定的程
序可被 docker run 命令行参数中指定要运行的程序所覆盖。如果 Dockerfile 中如果存在多个
CMD 指令，仅最后一个生效。
exec 模式： CMD[“executable”,”param1”,”param2”]
shell 模式 ： CMD command param1 param2

ENTRYPOINT： 类似于 CMD 指令，但其不会被 docker run 的命令行参数指定的指令所覆盖，而且这些命令行参数会被当作参数送给 ENTRYPOINT 指令指定的程序。如果 Dockerfile 中如果存在多个 ENTRYPOINT 指令，仅最后一个生效。如果运行 docker run 时使用了 —entrypoint选项，将覆盖 CMD 指令指定的程序。如果 Dockerfile 中如果存在多个 ENTRYPOINT 指令，仅最后一个生效。
格式: ENTRYPOINT [““,”“,”“,…]
可以搭配 CMD 命令使用：一般是变参才会使用 CMD ，这里的 CMD 等于是在给 ENTRYPOINT 传参
示例： 假设已通过 Dockerfile 构建了 nginx:test 镜像：

FROM nginx
ENTRYPOINT ["nginx", "-c"] # 定参
CMD ["/etc/nginx/nginx.conf"] # 变参

构建的镜像是 nginx:test
1、不传参运行
docker run nginx:test
容器内会默认运行以下命令，启动主进程。
nginx -c /etc/nginx/nginx.conf
2、传参运行
docker run nginx:test -c /etc/nginx/new.conf
容器内会默认运行以下命令，启动主进程(/etc/nginx/new.conf:假设容器内已有此文件)
nginx -c /etc/nginx/new.conf

COPY: 复制指令，从目录中复制文件或者目录到容器里指定路径。
COPY [—chown=:] <源路径 1>… <目标路径>
COPY [—chown=:] [“<源路径 1>”,… “<目标路径>”]
[—chown=:]：可选参数，用户改变复制到容器内文件的拥有者和属组。
<源路径>：源文件或者源目录，这里可以是通配符表达式，其通配符规则要满足 Go 的
filepath.Match 规则。

ADD：指令和 COPY 的使用格式一致（同样需求下，官方推荐使用 COPY）。功能也类似，不同之
处如下：
ADD 的优点：在执行 <源文件> 为 tar 压缩文件的话，压缩格式为 gzip, bzip2 以及 xz 的情
况下，会自动复制并解压到 <目标路径>。
ADD 的缺点：在不解压的前提下，无法复制 tar 压缩文件。会令镜像构建缓存失效，从而可能会
令镜像构建变得比较缓慢。具体是否使用，可以根据是否需要自动解压来决定

ENV：设置环境变量，定义了环境变量，那么在后续的指令中，就可以使用这个环境变量。
格式：
ENV
ENV = =…
以下示例设置 NODE_VERSION = 7.2.0 ， 在后续的指令中可以通过 $NODE_VERSION 引用: ENV NODE_VERSION 7.2.0

ARG ：构建参数，与 ENV 作用一至。不过作用域不一样。ARG 设置的环境变量仅对
Dockerfile 内有效，也就是说只有 docker build 的过程中有效，构建好的镜像内不存在此环境变量。
构建命令 docker build 中可以用 —build-arg <参数名>=<值> 来覆盖。
格式：
ARG <参数名>[=<默认值>]

VOLUME：定义匿名数据卷。在启动容器时忘记挂载数据卷，会自动挂载到匿名卷。
避免重要的数据，因容器重启而丢失，这是非常致命的。 避免容器不断变大。
格式：
VOLUME [“<路径 1>”, “<路径 2>”…]
VOLUME <路径>
在启动容器 docker run 的时候,我们可以通过 -v 参数修改挂载点

HEALTHCHECK：用于指定某个程序或者指令来监控docker容器服务的运行状态。
格式：
HEALTHCHECK [选项] CMD <命令>：设置检查容器健康状况的命令
HEALTHCHECK NONE：如果基础镜像有健康检查指令，使用这行可以屏蔽掉其健康检查指令
HEALTHCHECK [选项] CMD <命令> : 这边 CMD 后面跟随的命令使用，可以参考 CMD 的用法
编写一个简单的dockerfile文件

mkdir dockerfile && cd dockerfile
tee Dockerfile << 'EOF'
FROM centos
MAINTAINER yin
RUN rm -rf /etc/yum.repos.d/*
COPY Centos-vault-8.5.2111.repo /etc/yum.repos.d/
RUN yum install wget -y
RUN yum install nginx -y
COPY index.html /usr/share/nginx/html/
EXPOSE 80
ENTRYPOINT ["/usr/sbin/nginx","-g","daemon off;"]
EOF

编写首页内容, 并导入repo文件

tee index.html << 'EOF'
<html>
<head>
 <title>page added to dockerfile</title>
</head>
<body>
 <h1>welcome to docker, I am yin </h1>
</body>
</html>
EOF

基于Dockerfile来构建镜像并启动

# docker build -t <镜像名>:<tag> <Dockerfile路径>
docker build -t yin/nginx:v1 .
docker images
# --name html2 给容器起个名字，-p 80 容器端口，物理机端口会随机映射
docker run -d -p 80 --name html2 yin/nginx:v1
docker ps | grep html

访问测试

1.5 docker 容器资源配额控制之 cpu

Docker 通过 cgroup 来控制容器使用的资源配额，资源配额包括 CPU、内存、磁盘 IO，以免当多个容器运行时，防止某容器把所有的硬件都占用了。（比如一台被黑的容器，有可能把所有的资源都占用）
cgroup :
cgroup 是 Control Groups 的缩写，是 Linux 内核提供的一种可以限制、记录、隔离进程组所使
用的物理资源(如 cpu、memory、磁盘 IO 等等) ，被 LXC、docker 等很多项目用于实现进程资
源控制。
LXC:
LXC 是 Linux Container 的简写。可以提供轻量级的虚拟化，以便隔离进程和资源。

1.5.1 指定 docker 容器可以使用的 cpu 份额

cpu配额的参数: -c, —cpu-shares int
CPU shares (relative weight) 在创建容器时指定容器所使用的 CPU 份额值。cpu-shares 的值不能保证可以获得 1 个 vcpu 或者多少 GHz 的 CPU 资源，仅仅只是一个弹性的加权值。 默认每个 docker 容器的 cpu 份额值都是 1024。在同一个 CPU 核心上，同时运行多个容器时，容器的 cpu 加权的效果才能体现出来。
例： 两个容器 A、B 的 cpu 份额分别为 1000 和 500，结果会怎么样？
情况 1：A 和 B 正常运行，占用同一个 CPU，在 cpu 进行时间片分配的时候，容器 A 比容器 B 多一倍的机会获得 CPU 的时间片。
情况 2：分配的结果取决于当时其他容器的运行状态。比如容器 A 的进程一直是空闲的，那么容器 B是可以获取比容器 A 更多的 CPU 时间片的； 比如主机上只运行了一个容器，即使它的 cpu 份额只 有 50，它也可以独占整个主机的 cpu 资源。cgroups 只在多个容器同时争抢同一个 cpu 资源时，cpu 配额才会生效。因此，无法单纯根据某个容器的 cpu 份额来确定有多少 cpu 资源分配给它，资源分配结果取决于同时运行的其他容器的 cpu 分配和容器中进程运行情况。

运行一个容器并分配CPU使用份额， 并查看参数

docker run -it -c 512 centos cat /sys/fs/cgroup/cpu/cpu.shares

总结
通过-c 设置的 cpu share 并不是 CPU 资源的绝对数量，而是一个相对的权重值。某个容器最终能分配到的 CPU 资源取决于它的 cpu share 占所有容器 cpu share 总和的比例。通过 cpu share 可以设置容器使用 CPU 的优先级。
比如在 host 中启动了两个容器：
docker run —name “containerA” -c 1024 ubuntu
docker run —name “containerB” -c 512 ubuntu
containerA 的 cpu share 1024，是 containerB 的两倍。当两个容器都需要 CPU 资源时，containerA 可以得到的 CPU 是 container_B 的两倍。需要注意的是，这种按权重分配 CPU 只会发生在 CPU 资源紧张的情况
下。如果 containerA 处于空闲状态，为了充分利用 CPU 资源，containerB 也可以分配到全部可用的 CPU。

1.5.2 cpu核心控制

指定cpu核心参数：—cpuset 可以绑定 CPU
对多核 CPU 的服务器，docker 还可以控制容器运行限定使用哪些 cpu 内核和内存节点，即使用—
cpuset-cpus 和—cpuset-mems 参数。对具有 NUMA 拓扑（具有多 CPU、多内存节点）的服务
器尤其有用，可以对需要高性能计算的容器进行性能最优的配置。如果服务器只有一个内存节点，则
—cpuset-mems 的配置基本上不会有明显效果
扩展： 服务器架构一般分： SMP、NUMA、MPP 体系结构介绍,从系统架构来看，目前的商用服务器大体可以分为三类：
(1) 即对称多处理器结构(SMP ： Symmetric Multi-Processor) 例： x86 服务器，双路服务器。
主板上有两个物理 cpu
(2) 非一致存储访问结构 (NUMA ： Non-Uniform Memory Access) 例： IBM 小型机
pSeries 690
(3) 海量并行处理结构 (MPP ： Massive ParallelProcessing) 。 例： 大型机 Z14

taskset 命令
taskset 设定 cpu 亲和力，taskset 能够将一个或多个进程绑定到一个或多个处理器上运行
-c, —cpu-list 以列表格式显示和指定 CPU
-p, —pid 在已经存在的 pid 上操作
例: 设置只在 cup ID 是 1 和 2 的 cpu 上运行 sshd 进程程序

ps -ef | grep sshd
taskset -cp 1,2 1002        # 设置
taskset -cp 1002      #  查询验证

扩展: 为什么把进程绑定到 cpu 上，运行效率就高？
注：当 cpu 数量很多时，确实需要绑定进程到 cpu 上，这样可以减少 cpu 上下文切换的开销，节约时间。
扩展: 什么是 cpu 上下文切换？
我们都知道，Linux 是一个多任务操作系统，它支持远大于 CPU 数量的任务同时运行。当然这些任务实际上并不是真的在同时运行，而是因为系统在很短的时间内，将 CPU 轮流分配给它们，造成多任务同时运行的错觉。而在每个任务运行前，CPU 都需要知道任务从哪里加载、又从哪里开始运行，也就是说，需要系统 事先帮它设置好 CPU 寄存器和程序计数器(Program Counter，PC)。 CPU 寄存器，是 CPU 内置的容量小、但速度极快的内存。而程序计数器，则是用来存储 CPU 正在执行的指令位置、或者即将执行的下一条指令位置。它们都是 CPU 在运行任何任务前，必须的依赖环境，因此也被叫做 CPU 上下文。
而这些保存下来的上下文，会存储在系统内核中，并在任务重新调度执行时再次加载进来。这样就能保证任务原来的状态不受影响，让任务看起来还是连续运行。根据任务的不同，CPU 的上下文切换可以分为不同的场景，也就是进程上下文切换、线程上下文切换、中断上下文切换。

创建一个容器并指定cpu核数

docker run -it --cpuset-cpus 0-2 centos bash
cat /sys/fs/cgroup/cpuset/cpuset.cpus

1.5.3 cpu配额参数混合使用

在上面这些参数中，cpu-shares 控制只发生在容器竞争同一个 cpu 的时间片时有效。如果通过 cpuset-cpus 指定容器 A 使用 cpu 0，容器 B 只是用 cpu1，在主机上只有这两个容器使用对应内核的情况，它们各自占用全部的内核资源，cpu-shares 没有明显效果。如果容器 A 和容器 B 配置上 cpuset-cpus 值并都绑定到同一个 cpu 上，然后同时抢占 cpu 资源，就可 以看出效果了。

stress 命令
概述：linux 系统压力测试软件 stress。stress 可以测试 Linux 系统 cpu/memory/IO/disk 的 负载。
下载页：http://people.seas.harvard.edu/~apw/stress/
或者使用epel下载

yum -y install epel-release
yum -y install stress

参数解释:
-v 显示版本号
-q 不显示运行信息
-n 显示已完成的指令情况
-t —timeout N 指定运行 N 秒后停止
—backoff N 等待 N 微妙后开始运行
-c 产生 n 个进程 ：每个进程都反复不停的计算随机数的平方根，测试 cpu
-i 产生 n 个进程 ：每个进程反复调用 sync()，sync()用于将内存上的内容写到硬盘上，测试磁盘 io
-m —vm n 产生 n 个进程,每个进程不断调用内存分配 malloc（）和内存释放 free（）函数，测试内存
—vm-bytes B 指定 malloc 时内存的字节数 （默认 256MB）
—vm-hang N 指定在 free 栈的秒数
-d —hadd n 产生 n 个执行 write 和 unlink 函数的进程
-hadd-bytes B 指定写的字节数
—hadd-noclean 不 unlink
注：时间单位可以为秒 s，分 m，小时 h，天 d，年 y，文件大小单位可以为 K，M，G
例： 产生2个cpu进程，2个io进程，10秒后停止运行

stress -c 2 -i 2 -t 10
top

运行2个容器测试
例: 创建两个容器实例:docker10 和 docker20。 让 docker10 和 docker20 只运行在 cpu0 和 cpu1 上，最终测试一下 docker10 和 docker20 使用 cpu 的百分比

docker run -itd --name docker10 --cpuset-cpus 0,1 --cpu-shares 512 centos:centos7.9.2009 /bin/bash
docker run -itd --name docker20 --cpuset-cpus 0,1 --cpu-shares 1024 centos:centos7.9.2009 /bin/bash

进入docker10 执行命令,并查看是否在cpu0,1上运行

docker exec -it docker10 /bin/bash
yum install -y epel-release
yum install stress -y
stress -c 2 -v -t 10m

进入docker20 执行命令

docker exec -it docker20 /bin/bash
yum install -y epel-release
yum install stress -y
stress -c 2 -v -t 10m

1.6 docker 容器实例运行结束后自动释放资源

—rm 参数
作用：当容器命令运行结束后，自动删除容器，自动释放资源
应用场景：在某些环境下，可能需要大量的新建 docker 实例，然后仅仅运行几秒钟或几分钟，然后
就彻底删除。 如运行单元测试或测试弹性云计算
使用展示

 docker run -it --rm --name yin centos sleep 5
 docker ps -a | grep mk

等5秒之后在查看

docker ps | grep mk

1.7 docker容器资源配额控制之内存

Docker 提供参数-m, —memory=””限制容器的内存使用量
执行命令测试

docker run -it -m 128m centos
cat /sys/fs/cgroup/memory/memory.limit_in_bytes

和之前CPU命令整合使用举例

docker run -it --cpuset-cpus 0,1 -m 128m centos

1.8 docker 容器资源配额控制之 IO

Docker提供参数:
—device-write-bps value #限制此设备上的写速度（bytes per second），单位可以是 kb、mb 或者 gb。
—device-read-bps value #限制此设备上的读速度（bytes per second），单位可以是 kb、 mb 或者 gb。
执行命令测试
注：dd 参数：
direct：读写数据采用直接 IO 方式，不走缓存。直接从内存写硬盘上。
nonblock：读写数据采用非阻塞 IO 方式，优先写 dd 命令的数据

docker run -it -v /var/www/html/:/var/www/html --device /dev/sda:/dev/sda --device-write-bps /dev/sda:1mb centos /bin/bash
time dd if=/dev/sda of=/var/www/html/test.out bs=1M count=50 oflag=direct,nonblock

1.9 docker 容器的数据管理

1.9.1 docker容器的数据卷

什么是数据卷？
数据卷是经过特殊设计的目录，可以绕过联合文件系统（UFS），为一个或者多个容器提供访问，数据卷设计的目的，在于数据的永久性，它完全独立于容器的生存周期，因此，docker 不会在容器删除时删除其挂载的数据卷，也不会存在类似的垃圾收集机制，对容器引用的数据卷进行处理，同一个数据卷可以只支持多个容器的访问。
数据卷的特点：
（1）数据卷在容器启动时初始化，如果容器使用的镜像在挂载点包含了数据，这些数据会被拷贝到新初始化的数据卷中
（2）数据卷可以在容器之间共享和重用
（3）可以对数据卷里的内容直接进行修改
（4）数据卷的变化不会影像镜像的更新
（5）卷会一直存在，即使挂载数据卷的容器已经被删除
举例说明:

mkdir /datavolume
docker run --name volume -v /datavolume:/data -itd centos /bin/bash

注：datavolume 为宿主机目录，data 为 docker 启动的 volume 容器的目录 这样在宿主机的/datavolume 目录下创建的数据就会同步到容器的/data 目录下
给容器挂载卷只读权限:

docker run --name volume1 -v /datavolume1:/data:ro -itd centos /bin/bash

1.9.2 使用 dockerfile 构建包含数据卷的镜像

编写dockerfile文件测试

FROM centos
VOLUME ["/datavolume3","/datavolume6"]
CMD /bin/bash

执行dockerfile并启动验证

docker build -t volume .
docker run --name volume-dubble -it volume

查看挂载的位置

 docker inspect volume-dubble

1.10 使用 registry 搭建 docker 私有仓库

1.10.1 初始化操作参考1.2.1和1.2.2

1.10.2 安装registry

将registry镜像包转递到服务器上并解压
registry 镜像官方下载地址：https://hub.docker.com/_/registry?tab=tags

docker load -i registry.tar
docker images

运行私有仓库镜像

docker run -d --name registry -p 5000:5000 -v /opt/registry:/var/lib/registry registry:latest
docker ps 
netstat -anutp | grep 5000
curl http://127.0.0.1:5000/v2/_catalog

默认情况下，Registry 存放镜像的目录是/var/lib/registry 目录下，这样如果容器被删除，则存放于容器中的镜像也会丢失，所以我们一般情况下会指定本地物理机一个目录如/opt/registry 挂载到容器的/var/lib/registry 下。使用-v 参数，指定本地持久的路径。
在其他机器配置私有仓库地址

# 配置docker配置文件, 指定docker加速节点为私有仓库地址
#  "insecure-registries":["192.168.1.64:5000"]
vim /etc/docker/daemon.json

systemctl daemon-reload && systemctl restart docker

其他机器拉取镜像测试

docker pull busybox
docker images

修改镜像名并推送镜像

docker tag busybox:latest 192.168.1.66:5000/busybox:latest
docker images
docker push 192.168.1.66:5000/busybox

在私有仓库registry上查看

curl 127.0.0.1:5000/v2/_catalog

在其他机器删除刚才上传的镜像，下载测试

docker rmi -f 192.168.1.66:5000/busybox
docker pull 192.168.1.66:5000/busybox
docker run 192.168.1.64:5000/busybox:latest echo "hello"

1.11 使用 Harbor 搭建 Docker 私有仓库

1.11.1 Harbor介绍

Docker 容器应用的开发和运行离不开可靠的镜像管理，虽然 Docker 官方也提供了公共的镜像仓库，但是从安全和效率等方面考虑，部署我们私有环境内的 Registry 也是非常必要的。Harbor 是由VMware 公司开源的企业级的 Docker Registry 管理项目，它包括权限管理(RBAC)、LDAP、日志审核、管理界面、自我注册、镜像复制和中文支持等功能。
官网地址：https://github.com/goharbor/harbor

1.11.2 为Harbor自签发证书

mkdir /opt/ssl -p && cd /data/ssl/
yum -y install openssl
openssl genrsa -out ca.key 3072
#生成一个 3072 位的 key，也就是私钥

生成一个数字证书 ca.pem，3650 表示证书的有效时间是 10 年，按箭头提示填写即可，没有箭头标注的为空：

openssl req -new -x509 -days 3650 -key ca.key -out ca.pem

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ‘.’, the field will be left blank.
——-
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:guangdong
Locality Name (eg, city) [Default City]:shenzhen
Organization Name (eg, company) [Default Company Ltd]:yin
Organizational Unit Name (eg, section) []:CA
Common Name (eg, your name or your server’s hostname) []:k8s-slave2
Email Address []:yin@163.com

openssl genrsa -out harbor.key 3072
# #生成一个 3072 位的 key，也就是私钥
openssl req -new -key harbor.key -out harbor.csr

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ‘.’, the field will be left blank.
——-
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:guangdong
Locality Name (eg, city) [Default City]:shenzhen
Organization Name (eg, company) [Default Company Ltd]:yin
Organizational Unit Name (eg, section) []:CA
Common Name (eg, your name or your server’s hostname) []:k8s-slave2
Email Address []:yin@163.com

Please enter the following ‘extra’ attributes
to be sent with your certificate request
A challenge password []:123456
An optional company name []:123456

openssl x509 -req -in harbor.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out harbor.pem -days 3650
# 签发证书
openssl x509 -noout -text -in harbor.pem

1.11.3 安装harbor

安装准备工作

mkdir /opt/install -p &&  cd /opt/install/
ll /opt/ssl/

将Harbor配置压缩包，解压到instll目录下

tar -xf harbor-offline-installer-v2.3.0-rc3.tgz

修改解压后配置文件

cd harbor 
cp harbor.yml.tmpl harbor.yml
vim harbor.yml

修改配置文件：
hostname: k8s-slave2
#修改 hostname，跟上面签发的证书域名保持一致
#协议用 https
certificate: /opt/ssl/harbor.pem
private_key: /opt/ssl/harbor.key
邮件和 ldap 不需要配置，在 harbor 的 web 界面可以配置
其他配置采用默认即可
修改之后保存退出
注：harbor 默认的账号密码：admin/Harbor12345
安装docker-compose
注： docker-compose 项目是 Docker 官方的开源项目，负责实现对 Docker 容器集群的快速编排。Docker-Compose 的工程配置文件默认为 docker-compose.yml，Docker-Compose 运行目录下的必要有一个 docker-compose.yml。docker-compose 可以管理多个 docker 实例。
下载地址: https://github.com/docker/compose
下载之后传到服务器

mv docker-compose-Linux-x86_64.64 /usr/local/bin/docker-compose
chmod a+x /usr/local/bin/docker-compose
docker-compose version

解压Harbor镜像，然后启动

docker load -i docker-harbor-2-3-0.tar.gz
cd /opt/install/harbor
./install.sh
cd /opt/install/harbor/
docker-compose start        # 启动
docker-compose stop         # 停止

注: 镜像不要停止，因为停止后，因为docker-compose有启动顺序问题，可能会起不来
访问ip，出现下图则成功:

账号: admin 密码:Harbor12345

1.11.4 上传镜像到Harbor仓库

创建项目名字，并测试传递镜像

创建成功:

去测试机器修改docker配置文件

vim /etc/docker/daemon.json

 systemctl daemon-reload && systemctl restart docker
 systemctl status docker

登录到仓库, 并传递镜像

docker login 192.168.1.66
docker tag busybox:latest 192.168.1.66/test/busybox:v1
docker images
docker push 192.168.1.66/test/busybox:v1

下图为传递成功:

1.11.5 从Harbor仓库下载镜像

删除之前的镜像,然后拉取一次

docker rmi -f 192.168.1.66/test/busybox:v1
docker pull 192.168.1.66/test/busybox:v1
docker images

启动测试

docker run --name test-harbor 192.168.1.66/test/busybox:v1 echo "hello"

1.12 创建docker静态化

1.12.1 docker网络模式介绍

host 模式，使用—net=host 指定。
container 模式，使用—net=container:NAME_or_ID 指定。
none 模式，使用—net=none 指定。
bridge 模式，使用—net=bridge 指定，默认就是 bridge 模式。
container 模式是指，创建新容器的时候，通过—net container 参数，指定其和已经存在的某个容器共享一个 Network Namespace。如下图所示，右方黄色新创建的 container，其网卡共享左边容器。因此就不会拥有自己独立的 IP，而是共享左边容器的 IP 172.17.0.2,端口范围等 网络资源，两个容器的进程通过 lo 网卡设备通信。

默认选择 bridge 的情况下，容器启动后会通过 DHCP 获取一个地址，这可能不是我们想要的，在centos7 系统上， docker 环境下可以使用 pipework 脚本对容器分配固定 IP（这个 IP 可以是和物理机同网段 IP）
注：Pipework 有个缺陷，容器重启后 IP 设置会自动消失，需要重新设置。
Docker 网络 host 模式是指共享宿主机的网络
Docker 网络 none 模式是指创建的容器没有网络地址，只有 lo 网卡

1.12.2 配置桥接网络

桥接本地物理网络的目的,是为了局域网内用户方便访问 docker 实例中服务,不要需要各种端口映射即可访问服务。 但是这样做,又违背了 docker 容器的安全隔离的原则,工作中辩证的选择.
安装桥接设备br0

yum -y install bridge-utils

备份网卡配置文件并修改配置

cd /etc/sysconfig/network-scripts/
cp ifcfg-ens33 /opt/
tee ifcfg-ens33 << EOF
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
DEVICE=ens33
ONBOOT=yes
BRIDGE="br0"
EOF

生成桥设备br0的配置文件，并重启网卡服务

tee ifcfg-br0 << EOF
DEVICE="br0"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Bridge"
BOOTPROTO=none
IPADDR=192.168.1.63
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=192.168.1.1
EOF
systemctl restart network
ip addr

扩展：配置桥接 br0 的好处？
（1）真实主机中安装的虚拟主机，需要和外界主机进行通讯的时候，数据需要通过真实主机的网卡进行传输，但是虚拟主机内核无法对真实主机的网卡进行控制，一般情况下需要虚拟主机先将数据包发送给真实主机的内核，再由真实主机内核将该数据通过真实物理网卡发送出去，该过程可以用 NAT（网络地址转换）实现该功能，但是数据传数度较慢。
（2）可以由真实主机的内核虚拟出来一个接口 br0，同时这个也是一个对外的虚拟网卡设备，通过该接口可以将虚拟主机网卡和真实主机网卡直接连接起来，进行正常的数据通讯，提升数据传输效率。该过程就是桥接。

1.12.3 使用pipework配置静态ip

1. 下载pipework

方法 1：直接下载 pipework zip 包，把 pipework-master.zip 上传到 Linux 中
https://github.com/jpetazzo/pipework

（2）使用 git 获得：
git 下载链接：https://github.com/jpetazzo/pipework.git

git clone https://github.com/jpetazzo/pipework.git

1. 设置pipework命令并重启docker ```bash unzip pipework-master.zip

cp ./pipework-master/pipework /usr/bin/

systemctl restart docker

---
<a name="Na6gY"></a>
### 1.12.4 创建4种网络模式的容器
以none模式创建一个docker容器
```bash
docker pull centos
docker run -itd --name none --net=none --privileged=true centos bash
docker ps

扩展：
—privileged=true #允许开启特权功能
privileged [ˈprɪvəlɪdʒd] 有特权的
在 docker 0.6 版以后，privileged 被引入 docker。使用该参数，container 内的 root 拥有真正的 root 权限。否则，container 内的 root 只是外部物理机的一个普通用户权限。使用 privileged 启动的容器，可以看到很多 host 上的设备，并且可以执行 mount。甚至允许你在docker 容器中启动 docker 容器。不启用 privileged，容器中 root 用户不能执行 mount。
用 pipework 给容器分配地址
pipework 语法：pipework 网桥名 容器实例 ID 分配给容器的 IP/掩码@网关

pipework br0 42ace1da75c9 192.168.1.166/24@192.168.1.2
docker inspect 42ace1da75c9

进入容器测试网络:

docker exec -it 42ace1da75c9 /bin/bash
ip addr

创建共享宿主机网络的容器并测试

docker run --name host -itd --net=host --privileged=true centos bash
docker exec -it host /bin/bash
ip addr

创建共享容器网络模式的容器

docker run --name container2 --net=container:none -itd --privileged=true centos bash
docker exec -it container2 /bin/bash
ip addr

创建桥接网络模式的容器

 docker run --name bridge -itd --privileged=true centos bash
 docker exec -it bridge bash
 ip addr

1.13 docker的数据卷容器创建、备份和还原

1.13.1 数据卷容器介绍

什么是数据卷容器？
一个容器挂载了数据卷，其他容器通过挂载这个容器实现数据共享，挂载数据卷的容器，就叫做数据
卷容器，挂载数据卷容器的方法：
docker run —volumes-from [container name]

1.13.2 创建数据卷容器并用容器挂载

注: 需要先创建数据卷镜像，参考1.9.2
创建数据卷容器并创建测试文件

 docker run --name data-volume -itd volume bash
 docker exec -it data-volume /bin/bash
 touch datavolume6/lucky.txt

创建一个容器挂载数据卷容器挂载的文件

docker run --name data-volume2 --volumes-from data-volume -itd centos /bin/bash
docker exec -it data-volume2 /bin/bash
cd /datavolume6 && ls

注意：生产容器做持久化，还用-v 参数，这样数据才不会丢失，多个容器可以挂载同一个物理机目录，做到数据同步

1.13.3 docker数据卷的备份和还原

备份方法
docker run —volumes-from [container name] -v $(pwd):/backup centos tar czvf /backup/backup.tar [container data volume]
—volumes-from : 挂载一个要备份的容器。
-v ：指定保存数据的路径。
tar cvf ：是容器运行时执行的命令，执行的是一个压缩文件 tar 命令，将要备份容器中的目录压缩到指定的目录下。

docker run --volumes-from data-volume2 -v /root/backup:/backup --name datavolume-copy centos tar zcvf /backup/data-volume2.tar.gz /datavolume6

数据还原
docker run —volumes-from [container name] -v $(pwd):/backup centos tar xzvf /backup/backup.tar.gz [container data volume]

docker run --volumes-from data-volume2 -v /root/backup/:/backup centos tar zxvf /backup/data-volume2.tar.gz -C /

二、Containerd 容器概述

2.1 Containerd 容器介绍

2.1.1 什么是Containerd

官方文档:
https://containerd.io
containerd 是容器虚拟化技术，从 docker 中剥离出来，形成开放容器接口（OCI）标准的一部分。containerd 是真正管控容器的一个进程，执行容器的时候用的是 runc。
docker 对容器的管理和操作基本都是通过 containerd 完成的。Containerd 是一个工业级标准的容器运行时，它强调简单性、健壮性和可移植性。Containerd 可以在宿主机中管理完整的容器生命周期：容器镜像的传输和存储、容器的执行和管理、容器的网络管理等。
什么是容器运行时？
Docker、containerd、podman 都可以称为容器运行时
什么是 OCI？
Linux 基金会于 2015 年 6 月成立 OCI（Open Container Initiative）组织，旨在围绕容器格式和运行时制定一个开放的工业化标准，目前主要有两个标准文档：容器运行时标准 （runtime spec）和容器镜像标准（image spec）。制定容器格式标准的宗旨就是不受上层结构的绑定，如特定的客户端、编排栈等，同时也不受特定的供应商或项目的绑定，即不限于某种特定操作系统、硬件、CPU 架构、公有云等。

2.1.2 Containerd和docker区别

containerd 是从 Docker 中分离出来的一个项目，可以作为一个底层容器运行时，现在它成了 Kubernete 容器运行时更好的选择。K8s1.20 之后版本官方建议用 containerd 作为容器引擎，K3S 中默认使用 containerd 作为容器运行时，我们只要安装好 K3S 就可以使用它了。
K8S 为什么要放弃使用 Docker 作为容器运行时，而使用 containerd 呢？
如果你使用 Docker 作为 K8S 容器运行时的话，kubelet 需要先要通过 dockershim 去调用Docker，再通过 Docker 去调用 containerd。如果你使用 containerd 作为 K8S 容器运行时的话，由于 containerd 内置了 CRI 插件，kubelet 可以直接调用 containerd。使用 containerd 不仅性能提高了（调用链变短了），而且资源占用也会变小（Docker 不是一个纯粹的容器运行时，具有大量其他功能）。

2.1.3 Containerd 容器组件

Containerd 是一个容器运行时组件，它原本是 Docker 平台的一部分，虽然小巧，但在整个系统里起到很关键的作用。Containerd 被设计成一种解耦合并且很容易与其它工具集成的组件。
containerd：守护进程，高性能容器运行时。
ctr：containerd 的命令行客户端。
runc：运行容器的命令行工具。

2.1.4 docker、containerd、docker-shim、runC 关系

containerd 囊括了单机运行一个容器时所需要的一切：
为了能够支持多种 OCI Runtime，containerd 内部使用 containerd-shim，每启动一个容器都会创建一个新的 containerd-shim 进程，指定容器 ID，Bundle 目录，运行时的二进制（比如 runc）。
1、docker：docker 本身而言，包括了 docker client 和 dockerd，是一个客户端工具，用来把用户的请求发送给 docker daemon(dockerd)。dockerd： dockerd 是对容器相关操作的最上层封装，直接面向操作用户。docker daemon，一般也会被称为 docker engine。dockerd 启动时会启动 containerd 子进程。
2、containerd：是一个工业级标准的容器运行时，它强调简单性、健壮性和可移植性，几乎囊括了单机运行一个容器运行时所需要的一切：执行，分发，监控，网络，构建，日志等。主要作用是：
1）、管理容器的生命周期(从创建容器到销毁容器)
2）、拉取/推送容器镜像
3）、存储管理(管理镜像及容器数据的存储)
4）、调用 runC 运行容器(与 runC 等容器运行时交互)
5）、管理容器网络接口及网络，dockerd 实际真实调用的还是 containerd 的 api 接口，containerd 是 dockerd 和 runC 之间的一个中间交流组件。
3、containerd-shim：一个真实运行容器的载体，为了能够支持多种 OCI Runtime，containerd 内部使用 containerd-shim，每启动一个容器都会起一个新的 containerd-shim 的进程。它通过指定三个参数：容器 ID、bundle 目录（对应某个容器生成的目录，一般位于：/var/run/docker/containerd/containerID）
4、runC：是一个轻量级的工具，用来运行容器的，我们可以不用通过 docker 引擎，直接运行容器。
他们之间的调用关系如下图：

docker 镜像和 containerd 镜像通用，但组织方式和存放目录不同，导致 docker 与 ctr 命令不通用，各自管理自己的镜像和容器。

2.2 Containerd 的安装和配置

2.2.1 安装

yum -y install containerd jq

jq 是什么？
jq 可以对 json 数据进行分片、过滤、映射和转换，和 sed、awk、grep 等命令一样，都可以让你轻松地把玩文本。它能轻松地把你拥有的数据转换成你期望的格式，而且需要写的程序通常也比你期望的更加简短

2.2.2 初始化配置

 containerd config default > /etc/containerd/config.toml
 systemctl enable containerd --now
 vim /etc/containerd/config.toml

sandbox_image = “k8s.gcr.io/pause:3.2” 替换成 registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.2
配置镜像加速: 先找到关键词: registry.mirrors 在下面添加
[plugins.”io.containerd.grpc.v1.cri”.registry.mirrors.”docker.io”]
endpoint = [“https://registry.docker-cn.com“]

# 上传文件并解压
docker load -i pause.tar.gz
systemctl daemon-reload && systemctl restart containerd
systemctl statud containerd

2.2.3 Containerd 应用-和镜像和容器相关命令练习

查看镜像

ctr image ls

下载镜像，必须是全路径

ctr image pull docker.io/library/busybox:latest

创建容器

ctr run -d docker.io/library/busybox:latest busybox-v1

查看容器在宿主机PID

ctr task ls

进入容器

ctr task exec --exec-id 2917 -t busybox-v1 sh

删除容器

ctr task rm -f busybox-v1

与宿主机共享目录

 ctr run --mount type=bind,src=/tmp,dst=/host,options=rbind:rw -d docker.io/library/busybox:latest busybox-v6

压缩镜像

 ctr images export busybox.tar.gz docker.io/library/busybox:latest

导入镜像

 ctr images import busybox.tar.gz

ctr 和 docker 命令比较

三、 Podman 容器概述

3.1 Podman 容器介绍

3.1.1 什么是podman

Docker 是很流行的容器技术，它在运行的时候有一个守护进程，需要把服务启动起来，才能通过CLI 管理容器，镜像，守护进程负责处理很多的事情，所以就可能有单点故障风险，当 Docker 服务程序
挂了，依赖它启动的容器就都不能使用了。
Podman 原来是 CRI-O 项目的一部分，后来被分离成一个单独的项目叫 libpod。Podman 的使用体验和 Docker 类似，不同的是 Podman 没有 daemon。以前使用 Docker CLI 的时候，Docker CLI 会通过 gRPC API 去跟 Docker Engine 说「我要启动一个容器」，然后 Docker Engine 才会通过 OCI Container runtime（默认是 runc）来启动一个容器。这就意味着容器的进程不可能是 Docker CLI 的子进程，而是 Docker Engine 的子进程。
3.1.2 podman特点
Podman 无需守护进程，可以用来管理容器，镜像，以下是它的一些特点：
无需安装 Docker，安装 Podman 后就可以进行管理
Podman 的命令与 Docker 几乎相同
Docker 下的镜像 Podman 也可以使用
Podman 存储它的镜像和容器与 Docker 的位置不同（即通过 Podman 载入镜像后，用 Docker查看镜像是看不到的）
Podman 控制下的容器可以由 root 用户运行，也可以由非特权用户运行
Podan 可以利用 RUNC 直接与容器，镜像或者内核等进行交互，而不是通过守护进程，是使用用户 命名空间来模拟容器中的 root，无需连接到具有 root 权限的套接字保证容器的体系安全

使用 Podman 之后我们不需要管理和 Docker 守护进程一样的守护进程，Podman 也同样支持Docker 命令，他们的镜像也是兼容的
Podman 官网地址：https://podman.io/
Podman 项目地址：https://github.com/containers/libpod

3.2 podman 容器的安装和应用

3.2.1 安装podman

yum -y install podman
podman version
podman info

3.2.2 podman 应用

1. 搜索镜像

   podman search busybox

   拉取镜像

   podman pull busybox

   查看镜像列表

   podman images

   给镜像打标签

   podman tag busybox busybox:v1

   删除镜像

   podman rmi -f busybox:v1

   创建容器

   podman run busybox /bin/echo "Hello,Welcome to xuegod"

   创建容器并进入

   podman run -it busybox sh

   查看运行容器

   podman ps

   删除容器

   podman rm -f 8dca6db6aa1f

   podman部署应用 ```bash podman pull docker.io/library/nginx

podman images

podman run —name nginx-v1 —restart=always -itd -p 9090:80 docker.io/library/nginx

curl localhost:9090

---
<a name="c7KAx"></a>
# 四、kubernetes 深度解读-架构-组件-资源对象
<a name="k6p3P"></a>
## 4.1 Kubernetes 概述
官网： <br />kubernetes 的 github 地址： https://github.com/kubernetes/kubernetes <br />kubernetes 官方站点： <br />https://kubernetes.io/ <br />https://kubernetes.io/zh/ <br />https://kubernetes.io/docs/<br />Kubernetes 起源<br />Kubernetes 单词起源于希腊语, 是“舵手”或者“领航员、飞行员”的意思。 来源于 Google 的 Borg 项目<br />Borg 是谷歌内部的一个容器编排工具，谷歌业务 90%以上都在 Borg 上运行，Borg 在谷歌内部已经使用了大概 15 年。 K8S 是在 Borg 的基础上开发出来的轻量级容器编排工具。K8S 的根基非常牢固，是站在巨人的肩膀上发展起来的项目。开源之后，迅速称霸容器编排技术领域
---
<a name="eUZzQ"></a>
## 4.2 Kubernetes 架构
单节点和高可用架构： <br />k8s 的物理架构是 master/node 模式： <br />k8s 集群至少需要一个主节点(Master)和多个工作节点(Worker)，主节点是集群的控制节点，负责整个集群的管理和控制，主节点主要用于暴露 API，调度部署和节点的管理。工作节点主要是运行容器 的。 <br />单 master 节点架构如下：<br />![image.png](https://cdn.nlark.com/yuque/0/2022/png/27422002/1650901239372-e7c4cc8b-f9d0-4a14-bf25-688bc7bdcaa0.png#clientId=u8dcb8bc6-ba77-4&crop=0&crop=0&crop=1&crop=1&from=paste&height=656&id=u490b68ca&margin=%5Bobject%20Object%5D&name=image.png&originHeight=656&originWidth=1245&originalType=binary&ratio=1&rotation=0&showTitle=false&size=143789&status=done&style=none&taskId=u17896dc5-a4a6-4777-9bee-bf7526fbc6e&title=&width=1245)<br />多 master 节点架构如下:<br />![image.png](https://cdn.nlark.com/yuque/0/2022/png/27422002/1650901212533-9e14c06d-081c-48f5-a0ea-84ff16d50a48.png#clientId=u8dcb8bc6-ba77-4&crop=0&crop=0&crop=1&crop=1&from=paste&height=484&id=ube4aea97&margin=%5Bobject%20Object%5D&name=image.png&originHeight=484&originWidth=1159&originalType=binary&ratio=1&rotation=0&showTitle=false&size=78613&status=done&style=none&taskId=u676d410f-d4bb-4542-b7ef-f554cb4913b&title=&width=1159)
---
<a name="mAFwt"></a>
## 4.2 Kubernetes 组件
kubectl：管理 k8s 的命令行工具，可以操作 k8s 中的资源对象。<br />etcd： 是一个高可用的键值数据库，存储 k8s 的资源状态信息和网络信息的，etcd 中的数据变更是通过 api server 进行的<br />apiserver: 提供 k8s api，是整个系统的对外接口，提供资源操作的唯一入口，供客户端和其它组件调用，提供了 k8s 各类资源对象（pod,deployment,Service 等）的增删改查，是整个系统的数据总 线和数据中心，并提供认证、授权、访问控制、API 注册和发现等机制，并将操作对象持久化到 etcd 中。相当于“营业厅”。<br />scheduler：负责 k8s 集群中 pod 的调度的 ， scheduler 通过与 apiserver 交互监听到创建 Pod 副本的信息后，它会检索所有符合该 Pod 要求的工作节点列表，开始执行 Pod 调度逻辑。调度成功后将 Pod 绑定到目标节点上，相当于“调度室”。<br />controller-manager：作为集群内部的管理控制中心，负责集群内的 Node、Pod 副本、服务端点（Endpoint）、命名空间（Namespace）、服务账号（ServiceAccount）、资源定额（ResourceQuota）的管理，当某个 Node 意外宕机时，Controller Manager 会及时发现并执行自动 化修复流程，确保集群始终处于预期的工作状态。<br />kubelet： 每个 Node 节点上的 kubelet 定期就会调用 API Server 的 REST 接口报告自身状态，API Server 接收这些信息后，将节点状态信息更新到 etcd 中。kubelet 也通过 API Server 监听 Pod 信息，从而对 Node 机器上的 POD 进行管理：如创建、删除、更新 Pod<br />kube-proxy：提供网络代理和负载均衡，是实现 service 的通信与负载均衡机制的重要组件，kube-proxy 负责为 Pod 创建代理服务，从 apiserver 获取所有 service 信息，并根据 service 信息创建代理服务，实现 service 到 Pod 的请求路由和转发，从而实现 K8s 层级的虚拟转发网络，将到 service 的请求转发到后端的 pod 上。<br />Calico：Calico 是一个纯三层的网络插件，calico 的 bgp 模式类似于 flannel 的 host-gw，calico在 kubernetes 中可提供网络功能和网络策略<br />Cordns：k8s1.11 之前使用的是 kube dns，1.11 之后才有 coredns，coredns 是一个 DNS 服务器，能够为 Kubernetes services 提供 DNS 记录
---
<a name="ykXwc"></a>
## 4.3 Kubernetes 容器编排工具的优势
灵活部署 <br />kubernetes 支持在多种平台部署，可在私有云，公有云，混合云，openstack、VMware vSphere，VMware Workstation，虚拟机，物理机等环境部署。 <br />安全高效，拥有完善的认证授权机制，自带审计功能 <br />可以对多用户做细化的授权管理（如 rbac 授权），达到相互之间的操作完全隔离，互不影响，而且自身带有审计功能，可以对操作过程进行实时的日志记录，出现问题可以方便排查。 <br />负载均衡 <br />支持四层、七层负载均衡，可用于多种场景。 <br />可扩展性强 <br />拥有强大的集群扩展能力，可以根据业务规模自动增加和缩减主机节点的数量，确保服务可以承受大量并发带来的压力，保证业务稳定运行。 <br />根据节点资源的使用情况对 pod 进行合理的调度可以按照用户需要调度 pod，例如保证 Pod 只在资源足够的节点上运行，会尝试把同一功能的 pod分散在不同的节点上，还会尝试平衡不同节点的资源使用率等。 <br />拥有完善的灾备预警方案拥有多种灾备解决方案，支持备份和容灾，出现故障可以达到秒级切换，保证线上业务不受影响
---
<a name="mHSok"></a>
## 4.4 Kubernetes 功能详解
多租户网络隔离 <br />Kubernetes 支持多种网络插件，如 flannel，calico，canel 等，每个插件都有独特的性能，可以分别适用于多种场景，我们可以利用 calico 的 network policy（网络策略）解决 k8s 中的网络隔离，对于多租户场景，可以每一个用户一个名称空间（namespace），然后对这个名称空间设置网络隔离。 <br />高可用，高可扩展性 <br />kubernetes 中支持多种高可用解决方案，如 keepalive+nginx，keepalived+haproxy 等，可以使访问流量分发到不同的主机节点，减轻节点压力，如果某个节点出现故障，可以实现秒级切换，达到高可用，保证业务不中断。 <br />用户数据的持久化存储 <br />kubernetes 支持多种持久化存储解决方案，保证数据可以得到很好的备份，降低用户数据丢失的风险，kubernetes 中可使用的存储方案如下： <br />本地存储：emptyDir，hostPath <br />网络连接类存储： <br />SAN（存储局域网络）：iSCSI <br />NAS（网络附加存储）：nfs，cifs <br />分布式存储：glusterfs，ceph-块级别的存储，cephfs-文件系统级别的存储<br />云存储：Azure Disk 等 <br />拥有多种更新回滚策略 <br />通过管理 kubernetes 的控制器和 service 等，可以实现灰度发布，蓝绿部署，金丝雀发布等，达到业务的快速部署和回滚等操作，用以满足不同客户的不同升级需求。 <br />弹性伸缩 <br />根据访问的流量压力，可以实现 pod 的水平扩容和缩减，达到秒级扩容，让 pod 始终处于满足业务正常运行时所需的数量即可，避免了资源的浪费。 <br />节省资源，优化硬件资源的使用 <br />根据流量进行业务组件的扩缩容，可节省机器资源。
---
<a name="tHIqN"></a>
## 4.5 容器部署和传统部署对比分析
![image.png](https://cdn.nlark.com/yuque/0/2022/png/27422002/1650901947826-f8327955-0dd0-4827-92b7-a069386a672e.png#clientId=u8dcb8bc6-ba77-4&crop=0&crop=0&crop=1&crop=1&from=paste&height=474&id=u91d4d105&margin=%5Bobject%20Object%5D&name=image.png&originHeight=474&originWidth=1165&originalType=binary&ratio=1&rotation=0&showTitle=false&size=93530&status=done&style=none&taskId=u7a1c7b05-78fc-4833-a83a-f531f91ec4e&title=&width=1165)<br />1.传统部署时代： <br />早期，应用程序在物理服务器上运行。无法为物理服务器中的应用程序定义资源边界，这会导致资源分配问题。例如，如果在物理服务器上运行多个应用程序，则可能会出现一个应用程序占用大部分资源的情况，结果可能导致其他应用程序的性能下降。一种解决方案是在不同的物理服务器上运行每个应用程序，但是由于资源利用不足而无法扩展，并且许多物理服务器的维护成本也很高。 <br />2.虚拟化部署时代： <br />作为解决方案，引入了虚拟化功能，它允许你在单个物理服务器的 CPU 上运行多个虚拟机（VM）。虚拟化功能允许应用程序在 VM 之间隔离，并提供安全级别，因为一个应用程序的信息不能被另一应用程序自由地访问。因为虚拟化可以轻松地添加或更新应用程序、降低硬件成本等，所以虚拟化可以更好地利用物理服务器中的资源，并可以实现更好的可伸缩性。每个 VM 是一台完整的计算机，在虚拟化硬件之上运行所有组件，包括其自己的操作系统。<br />3.容器部署时代： <br />容器类似 VM，但是它们具有轻量级的隔离属性，可以在应用程序之间共享操作系统（OS）。因此，容器被认为是轻量级的。容器与 VM 类似，具有自己的文件系统、CPU、内存、进程空间等。由于它们与基础架构分离，因此可以跨云和 OS 分发进行移植。容器因具有许多优势而变得流行起来。下面列出了容器的一些好处：<br />1）敏捷应用程序的创建和部署： <br />与使用 VM 镜像相比，提高了容器镜像创建的简便性和效率.持续开发、集成和部署： <br />2）通过快速简单的回滚(由于镜像不可变性)，提供可靠且频繁的容器镜像构建和部署。 <br />3）关注开发与运维的分离： <br />在构建/发布时而不是在部署时创建应用程序容器镜像，从而将应用程序与基础架构分离。 <br />4）可观察性：不仅可以显示操作系统级别的信息和指标，还可以显示应用程序的运行状况和其他指标信号。 <br />5）跨开发、测试和生产的环境一致性：在便携式计算机上与在云中相同地运行。 <br />6）云和操作系统分发的可移植性：可在 Ubuntu、RHEL、CoreOS、本地、Google Kubernetes Engine 和其他任何地方运行。 <br />7）以应用程序为中心的管理：提高抽象级别，从在虚拟硬件上运行 OS 到在 OS 上运行应用程序。 <br />8）松散耦合、分布式、弹性、解放的微服务：应用程序被分解成较小的独立部分，并且可以动态部署和管理 - 而不是在一台大型单机上整体运行。 <br />9）资源隔离：可预测的应用程序性能。 <br />10）资源利用：高效率和高密度
---
<a name="yOo5M"></a>
## 4.6 Kubernetes 中的资源对象
最小调度单元 Pod <br />Pod 是 Kubernetes 中的最小调度单元，当指派容器时，容器实际上并不会指派到物理硬件上，容器会被分配到一个 Pod 里，一个 Pod 封装一个容器（也可以封装多个容器），Pod 里的容器共享存储、网络等。也就是说，应该把整个 pod 看作虚拟机，然后每个容器相当于运行在虚拟机的进程。所有容器都被统一安排和调度，并运行在共享的上下文中。对于具体应用而言，Pod 是它们的逻辑主机，Pod 包含业务相关的多个应用容器。<br />![image.png](https://cdn.nlark.com/yuque/0/2022/png/27422002/1650902244906-769fe8f3-2931-47b0-ae63-0cc5ffe6a105.png#clientId=u8dcb8bc6-ba77-4&crop=0&crop=0&crop=1&crop=1&from=paste&height=510&id=uaea6cfe2&margin=%5Bobject%20Object%5D&name=image.png&originHeight=510&originWidth=771&originalType=binary&ratio=1&rotation=0&showTitle=false&size=526602&status=done&style=none&taskId=u24c63005-2fb9-4b20-89bb-28ae07c275f&title=&width=771)<br />资源对象标签 label<br />label 是标签的意思，k8s 中的资源对象大都可以打上标签，如 Node、Pod、Service 等，一个资源可以绑定任意多个 label，k8s 通过 Label 可实现多维度的资源分组管理，后续可通过 Label Selector 查询和筛选拥有某些 Label 的资源对象，例如创建一个 Pod，给定一个 Label 是app=tomcat，那么 service 可以通过 label selector 选择拥有 app=tomcat 的 pod，和其相关联，也可通过 app=tomcat 删除拥有该标签的 Pod 资源。<br />![image.png](https://cdn.nlark.com/yuque/0/2022/png/27422002/1650902233518-849f2b79-c7d7-40a5-a19a-61aa58b7d509.png#clientId=u8dcb8bc6-ba77-4&crop=0&crop=0&crop=1&crop=1&from=paste&height=422&id=u010a3c0e&margin=%5Bobject%20Object%5D&name=image.png&originHeight=422&originWidth=1150&originalType=binary&ratio=1&rotation=0&showTitle=false&size=49179&status=done&style=none&taskId=u44c6706b-db20-4e40-a4de-2d7cacaa93a&title=&width=1150)<br />控制器 Deployment<br />Replicaset： <br />Kubernetes 中的副本控制器，管理 Pod，使 pod 副本的数量始终维持在预设的个数。 <br />Deployment： <br />Deployment 管理 Replicaset 和 Pod 的副本控制器，Deployment 可以管理多个 Replicaset，是比 Replicaset 更高级的控制器，也即是说在创建 Deployment 的时候，会自动创建 Replicaset，由Replicaset 再创建 Pod，Deployment 能对 Pod 扩容、缩容、滚动更新和回滚、维持 Pod 数量。<br />![image.png](https://cdn.nlark.com/yuque/0/2022/png/27422002/1650902223727-d61afa7c-b181-45aa-86da-3606a89dd044.png#clientId=u8dcb8bc6-ba77-4&crop=0&crop=0&crop=1&crop=1&from=paste&height=464&id=u0d06ac43&margin=%5Bobject%20Object%5D&name=image.png&originHeight=464&originWidth=711&originalType=binary&ratio=1&rotation=0&showTitle=false&size=31795&status=done&style=none&taskId=ua197f501-8a69-42e6-981c-689ea64d7e4&title=&width=711)<br />控制器 Statefulset<br />Deployment 控制器是用来管理无状态应用的：管理的所有 Pod 一模一样，提供同一个服务，也不考虑在哪台 Node 运行，可随意扩容和缩容。这种应用称为“无状态”，例如 Web 服务，Deployment 部署的 pod 数据卷是共享的，创建 3 个 pod 都是用的同一个数据卷，对外提供统一的服务。在实际的场景中，尤其是分布式应用，会部署多个实例，这些实例之间往往有依赖关系，例如主从关系、主备关系，这种应用称为“有状态”，例如 MySQL 主从、Etcd 集群。 <br />StatefulSet 控制器用于部署有状态应用，满足一些有状态场景的： <br />1）StatefulSet 管控的的每个 Pod 对象都有固定的主机名和专有存储卷，即便被重构后亦能保持不变 <br />2）Pod 有序的部署、扩容、删除和停止 <br />3）Pod 分配一个稳定的且唯一的网络标识 <br />4）Pod 分配一个独享的存储<br />控制器 Daemonset<br />DaemonSet 确保全部（或者一些）Node 上运行一个 Pod 的副本。当有 Node 加入集群时，也会为他们新增一个 Pod 。当有 Node 从集群移除时，这些 Pod 也会被回收。删除 DaemonSet 将会删除它创建的所有Pod。 <br />使用 DaemonSet 的一些典型用法： <br />1）运行集群存储 daemon，例如在每个 Node 上运行 glusterd、ceph。 <br />2）日志收集，比如 fluentd，logstash 等 <br />3）系统监控，比如 Prometheus Node Exporter，collectd，New Relic agent，Ganglia gmond 等 <br />4）系统程序，比如 kube-proxy, kube-dns, glusterd, ceph 等 <br />控制器 Job&CronJob<br />Job 负责批量处理任务：Job 创建一个或多个 Pod，并确保指定数量的 Pod 成功终止。Pod 成功完 成后，Job 将跟踪成功完成的情况。当达到指定的成功完成次数时，任务（即 Job）就完成了。删除 Job将清除其创建的 Pod。 一个简单的情况是创建一个 Job 对象，以便可靠地运行一个 Pod 来完成。如果第一个 Pod 发生故 障或被删除（例如，由于节点硬件故障或节点重启），则 Job 对象将启动一个新的 Pod。 <br />Cron Job ：一个 CronJob 对象就像 crontab 文件中的一行。它用 Cron 格式进行编写，并周期性地在给定的调度时间执行 Job。在每次调度运行时间内大概会创建一个 Job 对象。我们之所以说 大概 ，是因为在特定的环境下可能会创建两个 Job，或者一个 Job 都没创建。<br />Service<br />![image.png](https://cdn.nlark.com/yuque/0/2022/png/27422002/1650902507413-42864d68-0aee-4574-87d0-daaa89154c95.png#clientId=u8dcb8bc6-ba77-4&crop=0&crop=0&crop=1&crop=1&from=paste&height=315&id=u1d664edc&margin=%5Bobject%20Object%5D&name=image.png&originHeight=315&originWidth=961&originalType=binary&ratio=1&rotation=0&showTitle=false&size=38623&status=done&style=none&taskId=uc262fde4-101c-461e-8964-abf7481fe10&title=&width=961)<br />在 kubernetes 中，Pod 是有生命周期的，Pod 重启 IP 就会发生变化。如果我们的服务都是将Pod 的 IP 地址写死，Pod 的挂掉或者重启，和刚才重启的 pod 相关联的其他服务将会找不到它所关联的 Pod，为了解决这个问题，在 kubernetes 中定义了 service 资源对象，Service 定义了一个服务访问的入口，客户端通过这个入口即可访问服务背后的应用集群实例，service 是一组 Pod 的逻辑集合，这一组 Pod 能够被 Service 访问到，通常是通过 Label Selector 实现的。<br />Ingress Controller<br />Ingress 可以把进入到集群内部的请求转发到集群中的一些服务上，从而可以把服务映射到集群外部。Ingress 能把集群内 Service 配置成外网能够访问的 URL，流量负载均衡，提供基于域名访问的虚拟主机等。 <br />Ingress Controller 可以理解为控制器，它通过不断的跟 Kubernetes API 交互，实时获取后端Service、Pod 的变化，比如新增、删除等，结合 Ingress 定义的规则生成配置，然后动态更新上边的 Nginx 或者 trafik 负载均衡器，并刷新使配置生效，来达到服务自动发现的作用。
---
<a name="SJ3jd"></a>
# 五、使用 kubeadm 搭建生产环境的多 master 节点 k8s 高可用集群
<a name="Scc7W"></a>
## 5.1 kubeadm试用场景
Kubeadm：kubeadm 是官方提供的开源安装工具，是一个开源项目，团队投入大量精力在维护，Kubeadm 安装的 k8s，k8s 所有组件都是以 pod 运行的，可以做到故障自恢复，不需要人为写脚本维护。目前应该是比较方便和推荐的，简单易用。 <br />二进制：二进制安装的 k8s，所有组件都是通过 systemd 维护的，二进制安装对于理解 k8s 架构和原理更好，调试也比较方便。但是不适合经常需要大批量安装 k8s 需求。
---
<a name="Rd75S"></a>
## 5.2 初始化集群
参考docker初始化方法，文档1.2<br />添加阿里k8s源(所有机器都需要配置)
```bash
tee /etc/yum.repos.d/kubernetes.repo <<-'EOF'
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
EOF

修改docker配置文件，将内核驱动修改成systemd驱动

tee /etc/docker/daemon.json << 'EOF'
{
"registry-mirrors":["https://vh3bm52y.mirror.aliyuncs.com","https://registry.docker-cn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hub-mirror.c.163.com"],"exec-opts":["native.cgroupdriver=systemd"]
} 
EOF
systemctl daemon-reload && systemctl restart docker
systemctl status docker

关闭swap分区

swapoff -a
# 永久关闭,注释即可
vim /etc/fstab

安装k8s组件(所有机器执行)

yum install -y kubelet-1.23.1 kubeadm-1.23.1 kubectl-1.23.1
systemctl enable --now kubelet

5.3 使用nginx+keepalived实现 apiserver 高可用

yum 安装nginx和apiserver (所有master节点)

yum install nginx keepalived nginx-mod-stream -y

修改nginx配置文件

vim /etc/nginx/nginx.conf

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
   worker_connections 1024;
}
# 四层负载均衡，为两台 Master apiserver 组件提供负载均衡
stream {
   log_format main '$remote_addr $upstream_addr - [$time_local] $status 
   $upstream_bytes_sent';
   access_log /var/log/nginx/k8s-access.log main;
   upstream k8s-apiserver {
   server 192.168.1.63:6443; # Master1 APISERVER IP:PORT
   server 192.168.1.64:6443; # Master2 APISERVER IP:PORT
   }
   server {
      listen 16443; # 由于 nginx 与 master 节点复用，这个监听端口不能是 6443，否则会冲突
      proxy_pass k8s-apiserver;
   }
}
http {
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';
    access_log /var/log/nginx/access.log main;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    server {
        listen 80 default_server;
        server_name _;
        location / {
        }
    }
}

nginx -t

修改keepalived配置文件

vim /etc/keepalived/keepalived.conf

global_defs {
 notification_email { 
 acassen@firewall.loc 
 failover@firewall.loc 
 sysadmin@firewall.loc 
 } 
 notification_email_from Alexandre.Cassen@firewall.loc 
 smtp_server 127.0.0.1 
 smtp_connect_timeout 30 
 router_id NGINX_MASTER
} 
vrrp_script check_nginx {
 script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 { 
 state MASTER 
 interface ens33 # 修改为实际网卡名
 virtual_router_id 51 # VRRP 路由 ID 实例，每个实例是唯一的
 priority 100 # 优先级，备服务器设置 90
 advert_int 1 # 指定 VRRP 心跳包通告间隔时间，默认 1 秒
 authentication { 
 auth_type PASS 
 auth_pass 1111
 }
 # 虚拟 IP
 virtual_ipaddress { 
 192.168.1.199/24     # VIP
 } 
 track_script {
 check_nginx
 } 
}

global_defs {
 notification_email { 
 acassen@firewall.loc 
 failover@firewall.loc 
 sysadmin@firewall.loc 
 } 
 notification_email_from Alexandre.Cassen@firewall.loc 
 smtp_server 127.0.0.1 
 smtp_connect_timeout 30 
 router_id NGINX_BACKUP
}
vrrp_script check_nginx {
 script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 { 
 state BACKUP 
 interface ens33
 virtual_router_id 51 # VRRP 路由 ID 实例，每个实例是唯一的
 priority 90
 advert_int 1
 authentication { 
 auth_type PASS 
 auth_pass 1111 
 } 
 virtual_ipaddress { 
 192.168.1.199/24
 } 
 track_script {
 check_nginx
 } 
}

配置nginx探活脚本

 vim /etc/keepalived/check_nginx.sh

#!/bin/bash
#1、判断 Nginx 是否存活
counter=`ps -C nginx --no-header | wc -l`
if [ $counter -eq 0 ]; then
 #2、如果不存活则尝试启动 Nginx
 service nginx start
 sleep 2
 #3、等待 2 秒后再次获取一次 Nginx 状态
 counter=`ps -C nginx --no-header | wc -l`
 #4、再次进行判断，如 Nginx 还不存活则停止 Keepalived，让地址进行漂移
 if [ $counter -eq 0 ]; then
 service keepalived stop
 fi
fi

chmod +x /etc/keepalived/check_nginx.sh

启动并测试

systemctl enable nginx keepalived --now
ip addr

注: 可以停止nginx服务用来测试ip是否会漂移

5.4 初始化集群

主节点随便一台编写k8s配置文件

vim /opt/kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.23.1
controlPlaneEndpoint: 192.168.1.199:16443
imageRepository: registry.aliyuncs.com/google_containers
apiServer:
  certSANs:
  - 192.168.1.63
  - 192.168.1.64
  - 192.168.1.65
  - 192.168.1.66
  - 192.168.1.199
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.10.0.0/16
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

初始化集群

kubeadm init --config /opt/kubeadm-config.yaml --ignore-preflight-errors=SystemVerification
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

5.5 其他节点加入集群

复制密钥到其他主节点

# 其他主节点执行
cd /root && mkdir -p /etc/kubernetes/pki/etcd &&mkdir -p ~/.kube/
# 初始化主节点执行
scp /etc/kubernetes/pki/ca.crt k8s-master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/ca.key k8s-master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.key k8s-master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.pub k8s-master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.key k8s-master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.crt k8s-master2:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/pki/etcd/ca.key k8s-master2:/etc/kubernetes/pki/etcd/

复制主节点加入命令

  kubeadm join 192.168.1.199:16443 --token kxpyo6.ac4xjsaxarnrs1bz \
    --discovery-token-ca-cert-hash sha256:9d662cd9e9b56cdf874293e58d3c698fb4a5f8acbe872db591b6471820231f2b \
    --control-plane

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

子节点加入命令

kubeadm join 192.168.1.199:16443 --token kxpyo6.ac4xjsaxarnrs1bz \
    --discovery-token-ca-cert-hash sha256:9d662cd9e9b56cdf874293e58d3c698fb4a5f8acbe872db591b6471820231f2b

测试节点是否加入

kubectl get nodes

5.6 使用网络组件calico

vim calico.yaml

---
# Source: calico/templates/calico-config.yaml
# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Typha is disabled.
  typha_service_name: "none"
  # Configure the backend to use.
  calico_backend: "bird"
  # Configure the MTU to use for workload interfaces and tunnels.
  # By default, MTU is auto-detected, and explicitly setting this field should not be required.
  # You can override auto-detection by providing a non-zero value.
  veth_mtu: "0"
  # The CNI network configuration to install on each node. The special
  # values in this config will be automatically populated.
  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "calico",
          "log_level": "info",
          "log_file_path": "/var/log/calico/cni/cni.log",
          "datastore_type": "kubernetes",
          "nodename": "__KUBERNETES_NODE_NAME__",
          "mtu": __CNI_MTU__,
          "ipam": {
              "type": "calico-ipam"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        },
        {
          "type": "bandwidth",
          "capabilities": {"bandwidth": true}
        }
      ]
    }
---
# Source: calico/templates/kdd-crds.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: bgpconfigurations.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: BGPConfiguration
    listKind: BGPConfigurationList
    plural: bgpconfigurations
    singular: bgpconfiguration
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: BGPConfiguration contains the configuration for any BGP routing.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: BGPConfigurationSpec contains the values of the BGP configuration.
            properties:
              asNumber:
                description: 'ASNumber is the default AS number used by a node. [Default:
                  64512]'
                format: int32
                type: integer
              communities:
                description: Communities is a list of BGP community values and their
                  arbitrary names for tagging routes.
                items:
                  description: Community contains standard or large community value
                    and its name.
                  properties:
                    name:
                      description: Name given to community value.
                      type: string
                    value:
                      description: Value must be of format `aa:nn` or `aa:nn:mm`.
                        For standard community use `aa:nn` format, where `aa` and
                        `nn` are 16 bit number. For large community use `aa:nn:mm`
                        format, where `aa`, `nn` and `mm` are 32 bit number. Where,
                        `aa` is an AS Number, `nn` and `mm` are per-AS identifier.
                      pattern: ^(\d+):(\d+)$|^(\d+):(\d+):(\d+)$
                      type: string
                  type: object
                type: array
              listenPort:
                description: ListenPort is the port where BGP protocol should listen.
                  Defaults to 179
                maximum: 65535
                minimum: 1
                type: integer
              logSeverityScreen:
                description: 'LogSeverityScreen is the log severity above which logs
                  are sent to the stdout. [Default: INFO]'
                type: string
              nodeToNodeMeshEnabled:
                description: 'NodeToNodeMeshEnabled sets whether full node to node
                  BGP mesh is enabled. [Default: true]'
                type: boolean
              prefixAdvertisements:
                description: PrefixAdvertisements contains per-prefix advertisement
                  configuration.
                items:
                  description: PrefixAdvertisement configures advertisement properties
                    for the specified CIDR.
                  properties:
                    cidr:
                      description: CIDR for which properties should be advertised.
                      type: string
                    communities:
                      description: Communities can be list of either community names
                        already defined in `Specs.Communities` or community value
                        of format `aa:nn` or `aa:nn:mm`. For standard community use
                        `aa:nn` format, where `aa` and `nn` are 16 bit number. For
                        large community use `aa:nn:mm` format, where `aa`, `nn` and
                        `mm` are 32 bit number. Where,`aa` is an AS Number, `nn` and
                        `mm` are per-AS identifier.
                      items:
                        type: string
                      type: array
                  type: object
                type: array
              serviceClusterIPs:
                description: ServiceClusterIPs are the CIDR blocks from which service
                  cluster IPs are allocated. If specified, Calico will advertise these
                  blocks, as well as any cluster IPs within them.
                items:
                  description: ServiceClusterIPBlock represents a single allowed ClusterIP
                    CIDR block.
                  properties:
                    cidr:
                      type: string
                  type: object
                type: array
              serviceExternalIPs:
                description: ServiceExternalIPs are the CIDR blocks for Kubernetes
                  Service External IPs. Kubernetes Service ExternalIPs will only be
                  advertised if they are within one of these blocks.
                items:
                  description: ServiceExternalIPBlock represents a single allowed
                    External IP CIDR block.
                  properties:
                    cidr:
                      type: string
                  type: object
                type: array
              serviceLoadBalancerIPs:
                description: ServiceLoadBalancerIPs are the CIDR blocks for Kubernetes
                  Service LoadBalancer IPs. Kubernetes Service status.LoadBalancer.Ingress
                  IPs will only be advertised if they are within one of these blocks.
                items:
                  description: ServiceLoadBalancerIPBlock represents a single allowed
                    LoadBalancer IP CIDR block.
                  properties:
                    cidr:
                      type: string
                  type: object
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: bgppeers.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: BGPPeer
    listKind: BGPPeerList
    plural: bgppeers
    singular: bgppeer
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: BGPPeerSpec contains the specification for a BGPPeer resource.
            properties:
              asNumber:
                description: The AS Number of the peer.
                format: int32
                type: integer
              keepOriginalNextHop:
                description: Option to keep the original nexthop field when routes
                  are sent to a BGP Peer. Setting "true" configures the selected BGP
                  Peers node to use the "next hop keep;" instead of "next hop self;"(default)
                  in the specific branch of the Node on "bird.cfg".
                type: boolean
              node:
                description: The node name identifying the Calico node instance that
                  is targeted by this peer. If this is not set, and no nodeSelector
                  is specified, then this BGP peer selects all nodes in the cluster.
                type: string
              nodeSelector:
                description: Selector for the nodes that should have this peering.  When
                  this is set, the Node field must be empty.
                type: string
              password:
                description: Optional BGP password for the peerings generated by this
                  BGPPeer resource.
                properties:
                  secretKeyRef:
                    description: Selects a key of a secret in the node pod's namespace.
                    properties:
                      key:
                        description: The key of the secret to select from.  Must be
                          a valid secret key.
                        type: string
                      name:
                        description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                          TODO: Add other useful fields. apiVersion, kind, uid?'
                        type: string
                      optional:
                        description: Specify whether the Secret or its key must be
                          defined
                        type: boolean
                    required:
                    - key
                    type: object
                type: object
              peerIP:
                description: The IP address of the peer followed by an optional port
                  number to peer with. If port number is given, format should be `[<IPv6>]:port`
                  or `<IPv4>:<port>` for IPv4. If optional port number is not set,
                  and this peer IP and ASNumber belongs to a calico/node with ListenPort
                  set in BGPConfiguration, then we use that port to peer.
                type: string
              peerSelector:
                description: Selector for the remote nodes to peer with.  When this
                  is set, the PeerIP and ASNumber fields must be empty.  For each
                  peering between the local node and selected remote nodes, we configure
                  an IPv4 peering if both ends have NodeBGPSpec.IPv4Address specified,
                  and an IPv6 peering if both ends have NodeBGPSpec.IPv6Address specified.  The
                  remote AS number comes from the remote node's NodeBGPSpec.ASNumber,
                  or the global default if that is not set.
                type: string
              sourceAddress:
                description: Specifies whether and how to configure a source address
                  for the peerings generated by this BGPPeer resource.  Default value
                  "UseNodeIP" means to configure the node IP as the source address.  "None"
                  means not to configure a source address.
                type: string
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: blockaffinities.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: BlockAffinity
    listKind: BlockAffinityList
    plural: blockaffinities
    singular: blockaffinity
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: BlockAffinitySpec contains the specification for a BlockAffinity
              resource.
            properties:
              cidr:
                type: string
              deleted:
                description: Deleted indicates that this block affinity is being deleted.
                  This field is a string for compatibility with older releases that
                  mistakenly treat this field as a string.
                type: string
              node:
                type: string
              state:
                type: string
            required:
            - cidr
            - deleted
            - node
            - state
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: clusterinformations.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: ClusterInformation
    listKind: ClusterInformationList
    plural: clusterinformations
    singular: clusterinformation
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: ClusterInformation contains the cluster specific information.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: ClusterInformationSpec contains the values of describing
              the cluster.
            properties:
              calicoVersion:
                description: CalicoVersion is the version of Calico that the cluster
                  is running
                type: string
              clusterGUID:
                description: ClusterGUID is the GUID of the cluster
                type: string
              clusterType:
                description: ClusterType describes the type of the cluster
                type: string
              datastoreReady:
                description: DatastoreReady is used during significant datastore migrations
                  to signal to components such as Felix that it should wait before
                  accessing the datastore.
                type: boolean
              variant:
                description: Variant declares which variant of Calico should be active.
                type: string
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: felixconfigurations.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: FelixConfiguration
    listKind: FelixConfigurationList
    plural: felixconfigurations
    singular: felixconfiguration
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: Felix Configuration contains the configuration for Felix.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: FelixConfigurationSpec contains the values of the Felix configuration.
            properties:
              allowIPIPPacketsFromWorkloads:
                description: 'AllowIPIPPacketsFromWorkloads controls whether Felix
                  will add a rule to drop IPIP encapsulated traffic from workloads
                  [Default: false]'
                type: boolean
              allowVXLANPacketsFromWorkloads:
                description: 'AllowVXLANPacketsFromWorkloads controls whether Felix
                  will add a rule to drop VXLAN encapsulated traffic from workloads
                  [Default: false]'
                type: boolean
              awsSrcDstCheck:
                description: 'Set source-destination-check on AWS EC2 instances. Accepted
                  value must be one of "DoNothing", "Enabled" or "Disabled". [Default:
                  DoNothing]'
                enum:
                - DoNothing
                - Enable
                - Disable
                type: string
              bpfConnectTimeLoadBalancingEnabled:
                description: 'BPFConnectTimeLoadBalancingEnabled when in BPF mode,
                  controls whether Felix installs the connection-time load balancer.  The
                  connect-time load balancer is required for the host to be able to
                  reach Kubernetes services and it improves the performance of pod-to-service
                  connections.  The only reason to disable it is for debugging purposes.  [Default:
                  true]'
                type: boolean
              bpfDataIfacePattern:
                description: BPFDataIfacePattern is a regular expression that controls
                  which interfaces Felix should attach BPF programs to in order to
                  catch traffic to/from the network.  This needs to match the interfaces
                  that Calico workload traffic flows over as well as any interfaces
                  that handle incoming traffic to nodeports and services from outside
                  the cluster.  It should not match the workload interfaces (usually
                  named cali...).
                type: string
              bpfDisableUnprivileged:
                description: 'BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled
                  sysctl to disable unprivileged use of BPF.  This ensures that unprivileged
                  users cannot access Calico''s BPF maps and cannot insert their own
                  BPF programs to interfere with Calico''s. [Default: true]'
                type: boolean
              bpfEnabled:
                description: 'BPFEnabled, if enabled Felix will use the BPF dataplane.
                  [Default: false]'
                type: boolean
              bpfExternalServiceMode:
                description: 'BPFExternalServiceMode in BPF mode, controls how connections
                  from outside the cluster to services (node ports and cluster IPs)
                  are forwarded to remote workloads.  If set to "Tunnel" then both
                  request and response traffic is tunneled to the remote node.  If
                  set to "DSR", the request traffic is tunneled but the response traffic
                  is sent directly from the remote node.  In "DSR" mode, the remote
                  node appears to use the IP of the ingress node; this requires a
                  permissive L2 network.  [Default: Tunnel]'
                type: string
              bpfKubeProxyEndpointSlicesEnabled:
                description: BPFKubeProxyEndpointSlicesEnabled in BPF mode, controls
                  whether Felix's embedded kube-proxy accepts EndpointSlices or not.
                type: boolean
              bpfKubeProxyIptablesCleanupEnabled:
                description: 'BPFKubeProxyIptablesCleanupEnabled, if enabled in BPF
                  mode, Felix will proactively clean up the upstream Kubernetes kube-proxy''s
                  iptables chains.  Should only be enabled if kube-proxy is not running.  [Default:
                  true]'
                type: boolean
              bpfKubeProxyMinSyncPeriod:
                description: 'BPFKubeProxyMinSyncPeriod, in BPF mode, controls the
                  minimum time between updates to the dataplane for Felix''s embedded
                  kube-proxy.  Lower values give reduced set-up latency.  Higher values
                  reduce Felix CPU usage by batching up more work.  [Default: 1s]'
                type: string
              bpfLogLevel:
                description: 'BPFLogLevel controls the log level of the BPF programs
                  when in BPF dataplane mode.  One of "Off", "Info", or "Debug".  The
                  logs are emitted to the BPF trace pipe, accessible with the command
                  `tc exec bpf debug`. [Default: Off].'
                type: string
              chainInsertMode:
                description: 'ChainInsertMode controls whether Felix hooks the kernel''s
                  top-level iptables chains by inserting a rule at the top of the
                  chain or by appending a rule at the bottom. insert is the safe default
                  since it prevents Calico''s rules from being bypassed. If you switch
                  to append mode, be sure that the other rules in the chains signal
                  acceptance by falling through to the Calico rules, otherwise the
                  Calico policy will be bypassed. [Default: insert]'
                type: string
              dataplaneDriver:
                type: string
              debugDisableLogDropping:
                type: boolean
              debugMemoryProfilePath:
                type: string
              debugSimulateCalcGraphHangAfter:
                type: string
              debugSimulateDataplaneHangAfter:
                type: string
              defaultEndpointToHostAction:
                description: 'DefaultEndpointToHostAction controls what happens to
                  traffic that goes from a workload endpoint to the host itself (after
                  the traffic hits the endpoint egress policy). By default Calico
                  blocks traffic from workload endpoints to the host itself with an
                  iptables "DROP" action. If you want to allow some or all traffic
                  from endpoint to host, set this parameter to RETURN or ACCEPT. Use
                  RETURN if you have your own rules in the iptables "INPUT" chain;
                  Calico will insert its rules at the top of that chain, then "RETURN"
                  packets to the "INPUT" chain once it has completed processing workload
                  endpoint egress policy. Use ACCEPT to unconditionally accept packets
                  from workloads after processing workload endpoint egress policy.
                  [Default: Drop]'
                type: string
              deviceRouteProtocol:
                description: This defines the route protocol added to programmed device
                  routes, by default this will be RTPROT_BOOT when left blank.
                type: integer
              deviceRouteSourceAddress:
                description: This is the source address to use on programmed device
                  routes. By default the source address is left blank, leaving the
                  kernel to choose the source address used.
                type: string
              disableConntrackInvalidCheck:
                type: boolean
              endpointReportingDelay:
                type: string
              endpointReportingEnabled:
                type: boolean
              externalNodesList:
                description: ExternalNodesCIDRList is a list of CIDR's of external-non-calico-nodes
                  which may source tunnel traffic and have the tunneled traffic be
                  accepted at calico nodes.
                items:
                  type: string
                type: array
              failsafeInboundHostPorts:
                description: 'FailsafeInboundHostPorts is a comma-delimited list of
                  UDP/TCP ports that Felix will allow incoming traffic to host endpoints
                  on irrespective of the security policy. This is useful to avoid
                  accidentally cutting off a host with incorrect configuration. Each
                  port should be specified as tcp:<port-number> or udp:<port-number>.
                  For back-compatibility, if the protocol is not specified, it defaults
                  to "tcp". To disable all inbound host ports, use the value none.
                  The default value allows ssh access and DHCP. [Default: tcp:22,
                  udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]'
                items:
                  description: ProtoPort is combination of protocol and port, both
                    must be specified.
                  properties:
                    port:
                      type: integer
                    protocol:
                      type: string
                  required:
                  - port
                  - protocol
                  type: object
                type: array
              failsafeOutboundHostPorts:
                description: 'FailsafeOutboundHostPorts is a comma-delimited list
                  of UDP/TCP ports that Felix will allow outgoing traffic from host
                  endpoints to irrespective of the security policy. This is useful
                  to avoid accidentally cutting off a host with incorrect configuration.
                  Each port should be specified as tcp:<port-number> or udp:<port-number>.
                  For back-compatibility, if the protocol is not specified, it defaults
                  to "tcp". To disable all outbound host ports, use the value none.
                  The default value opens etcd''s standard ports to ensure that Felix
                  does not get cut off from etcd as well as allowing DHCP and DNS.
                  [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667,
                  udp:53, udp:67]'
                items:
                  description: ProtoPort is combination of protocol and port, both
                    must be specified.
                  properties:
                    port:
                      type: integer
                    protocol:
                      type: string
                  required:
                  - port
                  - protocol
                  type: object
                type: array
              featureDetectOverride:
                description: FeatureDetectOverride is used to override the feature
                  detection. Values are specified in a comma separated list with no
                  spaces, example; "SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=".
                  "true" or "false" will force the feature, empty or omitted values
                  are auto-detected.
                type: string
              genericXDPEnabled:
                description: 'GenericXDPEnabled enables Generic XDP so network cards
                  that don''t support XDP offload or driver modes can use XDP. This
                  is not recommended since it doesn''t provide better performance
                  than iptables. [Default: false]'
                type: boolean
              healthEnabled:
                type: boolean
              healthHost:
                type: string
              healthPort:
                type: integer
              interfaceExclude:
                description: 'InterfaceExclude is a comma-separated list of interfaces
                  that Felix should exclude when monitoring for host endpoints. The
                  default value ensures that Felix ignores Kubernetes'' IPVS dummy
                  interface, which is used internally by kube-proxy. If you want to
                  exclude multiple interface names using a single value, the list
                  supports regular expressions. For regular expressions you must wrap
                  the value with ''/''. For example having values ''/^kube/,veth1''
                  will exclude all interfaces that begin with ''kube'' and also the
                  interface ''veth1''. [Default: kube-ipvs0]'
                type: string
              interfacePrefix:
                description: 'InterfacePrefix is the interface name prefix that identifies
                  workload endpoints and so distinguishes them from host endpoint
                  interfaces. Note: in environments other than bare metal, the orchestrators
                  configure this appropriately. For example our Kubernetes and Docker
                  integrations set the ''cali'' value, and our OpenStack integration
                  sets the ''tap'' value. [Default: cali]'
                type: string
              interfaceRefreshInterval:
                description: InterfaceRefreshInterval is the period at which Felix
                  rescans local interfaces to verify their state. The rescan can be
                  disabled by setting the interval to 0.
                type: string
              ipipEnabled:
                type: boolean
              ipipMTU:
                description: 'IPIPMTU is the MTU to set on the tunnel device. See
                  Configuring MTU [Default: 1440]'
                type: integer
              ipsetsRefreshInterval:
                description: 'IpsetsRefreshInterval is the period at which Felix re-checks
                  all iptables state to ensure that no other process has accidentally
                  broken Calico''s rules. Set to 0 to disable iptables refresh. [Default:
                  90s]'
                type: string
              iptablesBackend:
                description: IptablesBackend specifies which backend of iptables will
                  be used. The default is legacy.
                type: string
              iptablesFilterAllowAction:
                type: string
              iptablesLockFilePath:
                description: 'IptablesLockFilePath is the location of the iptables
                  lock file. You may need to change this if the lock file is not in
                  its standard location (for example if you have mapped it into Felix''s
                  container at a different path). [Default: /run/xtables.lock]'
                type: string
              iptablesLockProbeInterval:
                description: 'IptablesLockProbeInterval is the time that Felix will
                  wait between attempts to acquire the iptables lock if it is not
                  available. Lower values make Felix more responsive when the lock
                  is contended, but use more CPU. [Default: 50ms]'
                type: string
              iptablesLockTimeout:
                description: 'IptablesLockTimeout is the time that Felix will wait
                  for the iptables lock, or 0, to disable. To use this feature, Felix
                  must share the iptables lock file with all other processes that
                  also take the lock. When running Felix inside a container, this
                  requires the /run directory of the host to be mounted into the calico/node
                  or calico/felix container. [Default: 0s disabled]'
                type: string
              iptablesMangleAllowAction:
                type: string
              iptablesMarkMask:
                description: 'IptablesMarkMask is the mask that Felix selects its
                  IPTables Mark bits from. Should be a 32 bit hexadecimal number with
                  at least 8 bits set, none of which clash with any other mark bits
                  in use on the system. [Default: 0xff000000]'
                format: int32
                type: integer
              iptablesNATOutgoingInterfaceFilter:
                type: string
              iptablesPostWriteCheckInterval:
                description: 'IptablesPostWriteCheckInterval is the period after Felix
                  has done a write to the dataplane that it schedules an extra read
                  back in order to check the write was not clobbered by another process.
                  This should only occur if another application on the system doesn''t
                  respect the iptables lock. [Default: 1s]'
                type: string
              iptablesRefreshInterval:
                description: 'IptablesRefreshInterval is the period at which Felix
                  re-checks the IP sets in the dataplane to ensure that no other process
                  has accidentally broken Calico''s rules. Set to 0 to disable IP
                  sets refresh. Note: the default for this value is lower than the
                  other refresh intervals as a workaround for a Linux kernel bug that
                  was fixed in kernel version 4.11. If you are using v4.11 or greater
                  you may want to set this to, a higher value to reduce Felix CPU
                  usage. [Default: 10s]'
                type: string
              ipv6Support:
                type: boolean
              kubeNodePortRanges:
                description: 'KubeNodePortRanges holds list of port ranges used for
                  service node ports. Only used if felix detects kube-proxy running
                  in ipvs mode. Felix uses these ranges to separate host and workload
                  traffic. [Default: 30000:32767].'
                items:
                  anyOf:
                  - type: integer
                  - type: string
                  pattern: ^.*
                  x-kubernetes-int-or-string: true
                type: array
              logFilePath:
                description: 'LogFilePath is the full path to the Felix log. Set to
                  none to disable file logging. [Default: /var/log/calico/felix.log]'
                type: string
              logPrefix:
                description: 'LogPrefix is the log prefix that Felix uses when rendering
                  LOG rules. [Default: calico-packet]'
                type: string
              logSeverityFile:
                description: 'LogSeverityFile is the log severity above which logs
                  are sent to the log file. [Default: Info]'
                type: string
              logSeverityScreen:
                description: 'LogSeverityScreen is the log severity above which logs
                  are sent to the stdout. [Default: Info]'
                type: string
              logSeveritySys:
                description: 'LogSeveritySys is the log severity above which logs
                  are sent to the syslog. Set to None for no logging to syslog. [Default:
                  Info]'
                type: string
              maxIpsetSize:
                type: integer
              metadataAddr:
                description: 'MetadataAddr is the IP address or domain name of the
                  server that can answer VM queries for cloud-init metadata. In OpenStack,
                  this corresponds to the machine running nova-api (or in Ubuntu,
                  nova-api-metadata). A value of none (case insensitive) means that
                  Felix should not set up any NAT rule for the metadata path. [Default:
                  127.0.0.1]'
                type: string
              metadataPort:
                description: 'MetadataPort is the port of the metadata server. This,
                  combined with global.MetadataAddr (if not ''None''), is used to
                  set up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort.
                  In most cases this should not need to be changed [Default: 8775].'
                type: integer
              mtuIfacePattern:
                description: MTUIfacePattern is a regular expression that controls
                  which interfaces Felix should scan in order to calculate the host's
                  MTU. This should not match workload interfaces (usually named cali...).
                type: string
              natOutgoingAddress:
                description: NATOutgoingAddress specifies an address to use when performing
                  source NAT for traffic in a natOutgoing pool that is leaving the
                  network. By default the address used is an address on the interface
                  the traffic is leaving on (ie it uses the iptables MASQUERADE target)
                type: string
              natPortRange:
                anyOf:
                - type: integer
                - type: string
                description: NATPortRange specifies the range of ports that is used
                  for port mapping when doing outgoing NAT. When unset the default
                  behavior of the network stack is used.
                pattern: ^.*
                x-kubernetes-int-or-string: true
              netlinkTimeout:
                type: string
              openstackRegion:
                description: 'OpenstackRegion is the name of the region that a particular
                  Felix belongs to. In a multi-region Calico/OpenStack deployment,
                  this must be configured somehow for each Felix (here in the datamodel,
                  or in felix.cfg or the environment on each compute node), and must
                  match the [calico] openstack_region value configured in neutron.conf
                  on each node. [Default: Empty]'
                type: string
              policySyncPathPrefix:
                description: 'PolicySyncPathPrefix is used to by Felix to communicate
                  policy changes to external services, like Application layer policy.
                  [Default: Empty]'
                type: string
              prometheusGoMetricsEnabled:
                description: 'PrometheusGoMetricsEnabled disables Go runtime metrics
                  collection, which the Prometheus client does by default, when set
                  to false. This reduces the number of metrics reported, reducing
                  Prometheus load. [Default: true]'
                type: boolean
              prometheusMetricsEnabled:
                description: 'PrometheusMetricsEnabled enables the Prometheus metrics
                  server in Felix if set to true. [Default: false]'
                type: boolean
              prometheusMetricsHost:
                description: 'PrometheusMetricsHost is the host that the Prometheus
                  metrics server should bind to. [Default: empty]'
                type: string
              prometheusMetricsPort:
                description: 'PrometheusMetricsPort is the TCP port that the Prometheus
                  metrics server should bind to. [Default: 9091]'
                type: integer
              prometheusProcessMetricsEnabled:
                description: 'PrometheusProcessMetricsEnabled disables process metrics
                  collection, which the Prometheus client does by default, when set
                  to false. This reduces the number of metrics reported, reducing
                  Prometheus load. [Default: true]'
                type: boolean
              removeExternalRoutes:
                description: Whether or not to remove device routes that have not
                  been programmed by Felix. Disabling this will allow external applications
                  to also add device routes. This is enabled by default which means
                  we will remove externally added routes.
                type: boolean
              reportingInterval:
                description: 'ReportingInterval is the interval at which Felix reports
                  its status into the datastore or 0 to disable. Must be non-zero
                  in OpenStack deployments. [Default: 30s]'
                type: string
              reportingTTL:
                description: 'ReportingTTL is the time-to-live setting for process-wide
                  status reports. [Default: 90s]'
                type: string
              routeRefreshInterval:
                description: 'RouteRefreshInterval is the period at which Felix re-checks
                  the routes in the dataplane to ensure that no other process has
                  accidentally broken Calico''s rules. Set to 0 to disable route refresh.
                  [Default: 90s]'
                type: string
              routeSource:
                description: 'RouteSource configures where Felix gets its routing
                  information. - WorkloadIPs: use workload endpoints to construct
                  routes. - CalicoIPAM: the default - use IPAM data to construct routes.'
                type: string
              routeTableRange:
                description: Calico programs additional Linux route tables for various
                  purposes.  RouteTableRange specifies the indices of the route tables
                  that Calico should use.
                properties:
                  max:
                    type: integer
                  min:
                    type: integer
                required:
                - max
                - min
                type: object
              serviceLoopPrevention:
                description: 'When service IP advertisement is enabled, prevent routing
                  loops to service IPs that are not in use, by dropping or rejecting
                  packets that do not get DNAT''d by kube-proxy. Unless set to "Disabled",
                  in which case such routing loops continue to be allowed. [Default:
                  Drop]'
                type: string
              sidecarAccelerationEnabled:
                description: 'SidecarAccelerationEnabled enables experimental sidecar
                  acceleration [Default: false]'
                type: boolean
              usageReportingEnabled:
                description: 'UsageReportingEnabled reports anonymous Calico version
                  number and cluster size to projectcalico.org. Logs warnings returned
                  by the usage server. For example, if a significant security vulnerability
                  has been discovered in the version of Calico being used. [Default:
                  true]'
                type: boolean
              usageReportingInitialDelay:
                description: 'UsageReportingInitialDelay controls the minimum delay
                  before Felix makes a report. [Default: 300s]'
                type: string
              usageReportingInterval:
                description: 'UsageReportingInterval controls the interval at which
                  Felix makes reports. [Default: 86400s]'
                type: string
              useInternalDataplaneDriver:
                type: boolean
              vxlanEnabled:
                type: boolean
              vxlanMTU:
                description: 'VXLANMTU is the MTU to set on the tunnel device. See
                  Configuring MTU [Default: 1440]'
                type: integer
              vxlanPort:
                type: integer
              vxlanVNI:
                type: integer
              wireguardEnabled:
                description: 'WireguardEnabled controls whether Wireguard is enabled.
                  [Default: false]'
                type: boolean
              wireguardInterfaceName:
                description: 'WireguardInterfaceName specifies the name to use for
                  the Wireguard interface. [Default: wg.calico]'
                type: string
              wireguardListeningPort:
                description: 'WireguardListeningPort controls the listening port used
                  by Wireguard. [Default: 51820]'
                type: integer
              wireguardMTU:
                description: 'WireguardMTU controls the MTU on the Wireguard interface.
                  See Configuring MTU [Default: 1420]'
                type: integer
              wireguardRoutingRulePriority:
                description: 'WireguardRoutingRulePriority controls the priority value
                  to use for the Wireguard routing rule. [Default: 99]'
                type: integer
              xdpEnabled:
                description: 'XDPEnabled enables XDP acceleration for suitable untracked
                  incoming deny rules. [Default: true]'
                type: boolean
              xdpRefreshInterval:
                description: 'XDPRefreshInterval is the period at which Felix re-checks
                  all XDP state to ensure that no other process has accidentally broken
                  Calico''s BPF maps or attached programs. Set to 0 to disable XDP
                  refresh. [Default: 90s]'
                type: string
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: globalnetworkpolicies.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: GlobalNetworkPolicy
    listKind: GlobalNetworkPolicyList
    plural: globalnetworkpolicies
    singular: globalnetworkpolicy
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            properties:
              applyOnForward:
                description: ApplyOnForward indicates to apply the rules in this policy
                  on forward traffic.
                type: boolean
              doNotTrack:
                description: DoNotTrack indicates whether packets matched by the rules
                  in this policy should go through the data plane's connection tracking,
                  such as Linux conntrack.  If True, the rules in this policy are
                  applied before any data plane connection tracking, and packets allowed
                  by this policy are marked as not to be tracked.
                type: boolean
              egress:
                description: The ordered set of egress rules.  Each rule contains
                  a set of packet match criteria and a corresponding action to apply.
                items:
                  description: "A Rule encapsulates a set of match criteria and an
                    action.  Both selector-based security Policy and security Profiles
                    reference rules - separated out as a list of rules for both ingress
                    and egress packet matching. \n Each positive match criteria has
                    a negated version, prefixed with \"Not\". All the match criteria
                    within a rule must be satisfied for a packet to match. A single
                    rule can contain the positive and negative version of a match
                    and both must be satisfied for the rule to match."
                  properties:
                    action:
                      type: string
                    destination:
                      description: Destination contains the match criteria that apply
                        to destination entity.
                      properties:
                        namespaceSelector:
                          description: "NamespaceSelector is an optional field that
                            contains a selector expression. Only traffic that originates
                            from (or terminates at) endpoints within the selected
                            namespaces will be matched. When both NamespaceSelector
                            and Selector are defined on the same rule, then only workload
                            endpoints that are matched by both selectors will be selected
                            by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                            implies that the Selector is limited to selecting only
                            workload endpoints in the same namespace as the NetworkPolicy.
                            \n For NetworkPolicy, `global()` NamespaceSelector implies
                            that the Selector is limited to selecting only GlobalNetworkSet
                            or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                            NamespaceSelector implies the Selector applies to workload
                            endpoints across all namespaces."
                          type: string
                        nets:
                          description: Nets is an optional field that restricts the
                            rule to only apply to traffic that originates from (or
                            terminates at) IP addresses in any of the given subnets.
                          items:
                            type: string
                          type: array
                        notNets:
                          description: NotNets is the negated version of the Nets
                            field.
                          items:
                            type: string
                          type: array
                        notPorts:
                          description: NotPorts is the negated version of the Ports
                            field. Since only some protocols have ports, if any ports
                            are specified it requires the Protocol match in the Rule
                            to be set to "TCP" or "UDP".
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          description: NotSelector is the negated version of the Selector
                            field.  See Selector field for subtleties with negated
                            selectors.
                          type: string
                        ports:
                          description: "Ports is an optional field that restricts
                            the rule to only apply to traffic that has a source (destination)
                            port that matches one of these ranges/values. This value
                            is a list of integers or strings that represent ranges
                            of ports. \n Since only some protocols have ports, if
                            any ports are specified it requires the Protocol match
                            in the Rule to be set to \"TCP\" or \"UDP\"."
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          description: "Selector is an optional field that contains
                            a selector expression (see Policy for sample syntax).
                            \ Only traffic that originates from (terminates at) endpoints
                            matching the selector will be matched. \n Note that: in
                            addition to the negated version of the Selector (see NotSelector
                            below), the selector expression syntax itself supports
                            negation.  The two types of negation are subtly different.
                            One negates the set of matched endpoints, the other negates
                            the whole match: \n \tSelector = \"!has(my_label)\" matches
                            packets that are from other Calico-controlled \tendpoints
                            that do not have the label \"my_label\". \n \tNotSelector
                            = \"has(my_label)\" matches packets that are not from
                            Calico-controlled \tendpoints that do have the label \"my_label\".
                            \n The effect is that the latter will accept packets from
                            non-Calico sources whereas the former is limited to packets
                            from Calico-controlled endpoints."
                          type: string
                        serviceAccounts:
                          description: ServiceAccounts is an optional field that restricts
                            the rule to only apply to traffic that originates from
                            (or terminates at) a pod running as a matching service
                            account.
                          properties:
                            names:
                              description: Names is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account whose name is in the list.
                              items:
                                type: string
                              type: array
                            selector:
                              description: Selector is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account that matches the given label selector. If
                                both Names and Selector are specified then they are
                                AND'ed.
                              type: string
                          type: object
                      type: object
                    http:
                      description: HTTP contains match criteria that apply to HTTP
                        requests.
                      properties:
                        methods:
                          description: Methods is an optional field that restricts
                            the rule to apply only to HTTP requests that use one of
                            the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                            methods are OR'd together.
                          items:
                            type: string
                          type: array
                        paths:
                          description: 'Paths is an optional field that restricts
                            the rule to apply to HTTP requests that use one of the
                            listed HTTP Paths. Multiple paths are OR''d together.
                            e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                            ONLY specify either a `exact` or a `prefix` match. The
                            validator will check for it.'
                          items:
                            description: 'HTTPPath specifies an HTTP path to match.
                              It may be either of the form: exact: <path>: which matches
                              the path exactly or prefix: <path-prefix>: which matches
                              the path prefix'
                            properties:
                              exact:
                                type: string
                              prefix:
                                type: string
                            type: object
                          type: array
                      type: object
                    icmp:
                      description: ICMP is an optional field that restricts the rule
                        to apply to a specific type and code of ICMP traffic.  This
                        should only be specified if the Protocol field is set to "ICMP"
                        or "ICMPv6".
                      properties:
                        code:
                          description: Match on a specific ICMP code.  If specified,
                            the Type value must also be specified. This is a technical
                            limitation imposed by the kernel's iptables firewall,
                            which Calico uses to enforce the rule.
                          type: integer
                        type:
                          description: Match on a specific ICMP type.  For example
                            a value of 8 refers to ICMP Echo Request (i.e. pings).
                          type: integer
                      type: object
                    ipVersion:
                      description: IPVersion is an optional field that restricts the
                        rule to only match a specific IP version.
                      type: integer
                    metadata:
                      description: Metadata contains additional information for this
                        rule
                      properties:
                        annotations:
                          additionalProperties:
                            type: string
                          description: Annotations is a set of key value pairs that
                            give extra information about the rule
                          type: object
                      type: object
                    notICMP:
                      description: NotICMP is the negated version of the ICMP field.
                      properties:
                        code:
                          description: Match on a specific ICMP code.  If specified,
                            the Type value must also be specified. This is a technical
                            limitation imposed by the kernel's iptables firewall,
                            which Calico uses to enforce the rule.
                          type: integer
                        type:
                          description: Match on a specific ICMP type.  For example
                            a value of 8 refers to ICMP Echo Request (i.e. pings).
                          type: integer
                      type: object
                    notProtocol:
                      anyOf:
                      - type: integer
                      - type: string
                      description: NotProtocol is the negated version of the Protocol
                        field.
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    protocol:
                      anyOf:
                      - type: integer
                      - type: string
                      description: "Protocol is an optional field that restricts the
                        rule to only apply to traffic of a specific IP protocol. Required
                        if any of the EntityRules contain Ports (because ports only
                        apply to certain protocols). \n Must be one of these string
                        values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                        \"UDPLite\" or an integer in the range 1-255."
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    source:
                      description: Source contains the match criteria that apply to
                        source entity.
                      properties:
                        namespaceSelector:
                          description: "NamespaceSelector is an optional field that
                            contains a selector expression. Only traffic that originates
                            from (or terminates at) endpoints within the selected
                            namespaces will be matched. When both NamespaceSelector
                            and Selector are defined on the same rule, then only workload
                            endpoints that are matched by both selectors will be selected
                            by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                            implies that the Selector is limited to selecting only
                            workload endpoints in the same namespace as the NetworkPolicy.
                            \n For NetworkPolicy, `global()` NamespaceSelector implies
                            that the Selector is limited to selecting only GlobalNetworkSet
                            or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                            NamespaceSelector implies the Selector applies to workload
                            endpoints across all namespaces."
                          type: string
                        nets:
                          description: Nets is an optional field that restricts the
                            rule to only apply to traffic that originates from (or
                            terminates at) IP addresses in any of the given subnets.
                          items:
                            type: string
                          type: array
                        notNets:
                          description: NotNets is the negated version of the Nets
                            field.
                          items:
                            type: string
                          type: array
                        notPorts:
                          description: NotPorts is the negated version of the Ports
                            field. Since only some protocols have ports, if any ports
                            are specified it requires the Protocol match in the Rule
                            to be set to "TCP" or "UDP".
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          description: NotSelector is the negated version of the Selector
                            field.  See Selector field for subtleties with negated
                            selectors.
                          type: string
                        ports:
                          description: "Ports is an optional field that restricts
                            the rule to only apply to traffic that has a source (destination)
                            port that matches one of these ranges/values. This value
                            is a list of integers or strings that represent ranges
                            of ports. \n Since only some protocols have ports, if
                            any ports are specified it requires the Protocol match
                            in the Rule to be set to \"TCP\" or \"UDP\"."
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          description: "Selector is an optional field that contains
                            a selector expression (see Policy for sample syntax).
                            \ Only traffic that originates from (terminates at) endpoints
                            matching the selector will be matched. \n Note that: in
                            addition to the negated version of the Selector (see NotSelector
                            below), the selector expression syntax itself supports
                            negation.  The two types of negation are subtly different.
                            One negates the set of matched endpoints, the other negates
                            the whole match: \n \tSelector = \"!has(my_label)\" matches
                            packets that are from other Calico-controlled \tendpoints
                            that do not have the label \"my_label\". \n \tNotSelector
                            = \"has(my_label)\" matches packets that are not from
                            Calico-controlled \tendpoints that do have the label \"my_label\".
                            \n The effect is that the latter will accept packets from
                            non-Calico sources whereas the former is limited to packets
                            from Calico-controlled endpoints."
                          type: string
                        serviceAccounts:
                          description: ServiceAccounts is an optional field that restricts
                            the rule to only apply to traffic that originates from
                            (or terminates at) a pod running as a matching service
                            account.
                          properties:
                            names:
                              description: Names is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account whose name is in the list.
                              items:
                                type: string
                              type: array
                            selector:
                              description: Selector is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account that matches the given label selector. If
                                both Names and Selector are specified then they are
                                AND'ed.
                              type: string
                          type: object
                      type: object
                  required:
                  - action
                  type: object
                type: array
              ingress:
                description: The ordered set of ingress rules.  Each rule contains
                  a set of packet match criteria and a corresponding action to apply.
                items:
                  description: "A Rule encapsulates a set of match criteria and an
                    action.  Both selector-based security Policy and security Profiles
                    reference rules - separated out as a list of rules for both ingress
                    and egress packet matching. \n Each positive match criteria has
                    a negated version, prefixed with \"Not\". All the match criteria
                    within a rule must be satisfied for a packet to match. A single
                    rule can contain the positive and negative version of a match
                    and both must be satisfied for the rule to match."
                  properties:
                    action:
                      type: string
                    destination:
                      description: Destination contains the match criteria that apply
                        to destination entity.
                      properties:
                        namespaceSelector:
                          description: "NamespaceSelector is an optional field that
                            contains a selector expression. Only traffic that originates
                            from (or terminates at) endpoints within the selected
                            namespaces will be matched. When both NamespaceSelector
                            and Selector are defined on the same rule, then only workload
                            endpoints that are matched by both selectors will be selected
                            by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                            implies that the Selector is limited to selecting only
                            workload endpoints in the same namespace as the NetworkPolicy.
                            \n For NetworkPolicy, `global()` NamespaceSelector implies
                            that the Selector is limited to selecting only GlobalNetworkSet
                            or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                            NamespaceSelector implies the Selector applies to workload
                            endpoints across all namespaces."
                          type: string
                        nets:
                          description: Nets is an optional field that restricts the
                            rule to only apply to traffic that originates from (or
                            terminates at) IP addresses in any of the given subnets.
                          items:
                            type: string
                          type: array
                        notNets:
                          description: NotNets is the negated version of the Nets
                            field.
                          items:
                            type: string
                          type: array
                        notPorts:
                          description: NotPorts is the negated version of the Ports
                            field. Since only some protocols have ports, if any ports
                            are specified it requires the Protocol match in the Rule
                            to be set to "TCP" or "UDP".
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          description: NotSelector is the negated version of the Selector
                            field.  See Selector field for subtleties with negated
                            selectors.
                          type: string
                        ports:
                          description: "Ports is an optional field that restricts
                            the rule to only apply to traffic that has a source (destination)
                            port that matches one of these ranges/values. This value
                            is a list of integers or strings that represent ranges
                            of ports. \n Since only some protocols have ports, if
                            any ports are specified it requires the Protocol match
                            in the Rule to be set to \"TCP\" or \"UDP\"."
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          description: "Selector is an optional field that contains
                            a selector expression (see Policy for sample syntax).
                            \ Only traffic that originates from (terminates at) endpoints
                            matching the selector will be matched. \n Note that: in
                            addition to the negated version of the Selector (see NotSelector
                            below), the selector expression syntax itself supports
                            negation.  The two types of negation are subtly different.
                            One negates the set of matched endpoints, the other negates
                            the whole match: \n \tSelector = \"!has(my_label)\" matches
                            packets that are from other Calico-controlled \tendpoints
                            that do not have the label \"my_label\". \n \tNotSelector
                            = \"has(my_label)\" matches packets that are not from
                            Calico-controlled \tendpoints that do have the label \"my_label\".
                            \n The effect is that the latter will accept packets from
                            non-Calico sources whereas the former is limited to packets
                            from Calico-controlled endpoints."
                          type: string
                        serviceAccounts:
                          description: ServiceAccounts is an optional field that restricts
                            the rule to only apply to traffic that originates from
                            (or terminates at) a pod running as a matching service
                            account.
                          properties:
                            names:
                              description: Names is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account whose name is in the list.
                              items:
                                type: string
                              type: array
                            selector:
                              description: Selector is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account that matches the given label selector. If
                                both Names and Selector are specified then they are
                                AND'ed.
                              type: string
                          type: object
                      type: object
                    http:
                      description: HTTP contains match criteria that apply to HTTP
                        requests.
                      properties:
                        methods:
                          description: Methods is an optional field that restricts
                            the rule to apply only to HTTP requests that use one of
                            the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                            methods are OR'd together.
                          items:
                            type: string
                          type: array
                        paths:
                          description: 'Paths is an optional field that restricts
                            the rule to apply to HTTP requests that use one of the
                            listed HTTP Paths. Multiple paths are OR''d together.
                            e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                            ONLY specify either a `exact` or a `prefix` match. The
                            validator will check for it.'
                          items:
                            description: 'HTTPPath specifies an HTTP path to match.
                              It may be either of the form: exact: <path>: which matches
                              the path exactly or prefix: <path-prefix>: which matches
                              the path prefix'
                            properties:
                              exact:
                                type: string
                              prefix:
                                type: string
                            type: object
                          type: array
                      type: object
                    icmp:
                      description: ICMP is an optional field that restricts the rule
                        to apply to a specific type and code of ICMP traffic.  This
                        should only be specified if the Protocol field is set to "ICMP"
                        or "ICMPv6".
                      properties:
                        code:
                          description: Match on a specific ICMP code.  If specified,
                            the Type value must also be specified. This is a technical
                            limitation imposed by the kernel's iptables firewall,
                            which Calico uses to enforce the rule.
                          type: integer
                        type:
                          description: Match on a specific ICMP type.  For example
                            a value of 8 refers to ICMP Echo Request (i.e. pings).
                          type: integer
                      type: object
                    ipVersion:
                      description: IPVersion is an optional field that restricts the
                        rule to only match a specific IP version.
                      type: integer
                    metadata:
                      description: Metadata contains additional information for this
                        rule
                      properties:
                        annotations:
                          additionalProperties:
                            type: string
                          description: Annotations is a set of key value pairs that
                            give extra information about the rule
                          type: object
                      type: object
                    notICMP:
                      description: NotICMP is the negated version of the ICMP field.
                      properties:
                        code:
                          description: Match on a specific ICMP code.  If specified,
                            the Type value must also be specified. This is a technical
                            limitation imposed by the kernel's iptables firewall,
                            which Calico uses to enforce the rule.
                          type: integer
                        type:
                          description: Match on a specific ICMP type.  For example
                            a value of 8 refers to ICMP Echo Request (i.e. pings).
                          type: integer
                      type: object
                    notProtocol:
                      anyOf:
                      - type: integer
                      - type: string
                      description: NotProtocol is the negated version of the Protocol
                        field.
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    protocol:
                      anyOf:
                      - type: integer
                      - type: string
                      description: "Protocol is an optional field that restricts the
                        rule to only apply to traffic of a specific IP protocol. Required
                        if any of the EntityRules contain Ports (because ports only
                        apply to certain protocols). \n Must be one of these string
                        values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                        \"UDPLite\" or an integer in the range 1-255."
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    source:
                      description: Source contains the match criteria that apply to
                        source entity.
                      properties:
                        namespaceSelector:
                          description: "NamespaceSelector is an optional field that
                            contains a selector expression. Only traffic that originates
                            from (or terminates at) endpoints within the selected
                            namespaces will be matched. When both NamespaceSelector
                            and Selector are defined on the same rule, then only workload
                            endpoints that are matched by both selectors will be selected
                            by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                            implies that the Selector is limited to selecting only
                            workload endpoints in the same namespace as the NetworkPolicy.
                            \n For NetworkPolicy, `global()` NamespaceSelector implies
                            that the Selector is limited to selecting only GlobalNetworkSet
                            or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                            NamespaceSelector implies the Selector applies to workload
                            endpoints across all namespaces."
                          type: string
                        nets:
                          description: Nets is an optional field that restricts the
                            rule to only apply to traffic that originates from (or
                            terminates at) IP addresses in any of the given subnets.
                          items:
                            type: string
                          type: array
                        notNets:
                          description: NotNets is the negated version of the Nets
                            field.
                          items:
                            type: string
                          type: array
                        notPorts:
                          description: NotPorts is the negated version of the Ports
                            field. Since only some protocols have ports, if any ports
                            are specified it requires the Protocol match in the Rule
                            to be set to "TCP" or "UDP".
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          description: NotSelector is the negated version of the Selector
                            field.  See Selector field for subtleties with negated
                            selectors.
                          type: string
                        ports:
                          description: "Ports is an optional field that restricts
                            the rule to only apply to traffic that has a source (destination)
                            port that matches one of these ranges/values. This value
                            is a list of integers or strings that represent ranges
                            of ports. \n Since only some protocols have ports, if
                            any ports are specified it requires the Protocol match
                            in the Rule to be set to \"TCP\" or \"UDP\"."
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          description: "Selector is an optional field that contains
                            a selector expression (see Policy for sample syntax).
                            \ Only traffic that originates from (terminates at) endpoints
                            matching the selector will be matched. \n Note that: in
                            addition to the negated version of the Selector (see NotSelector
                            below), the selector expression syntax itself supports
                            negation.  The two types of negation are subtly different.
                            One negates the set of matched endpoints, the other negates
                            the whole match: \n \tSelector = \"!has(my_label)\" matches
                            packets that are from other Calico-controlled \tendpoints
                            that do not have the label \"my_label\". \n \tNotSelector
                            = \"has(my_label)\" matches packets that are not from
                            Calico-controlled \tendpoints that do have the label \"my_label\".
                            \n The effect is that the latter will accept packets from
                            non-Calico sources whereas the former is limited to packets
                            from Calico-controlled endpoints."
                          type: string
                        serviceAccounts:
                          description: ServiceAccounts is an optional field that restricts
                            the rule to only apply to traffic that originates from
                            (or terminates at) a pod running as a matching service
                            account.
                          properties:
                            names:
                              description: Names is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account whose name is in the list.
                              items:
                                type: string
                              type: array
                            selector:
                              description: Selector is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account that matches the given label selector. If
                                both Names and Selector are specified then they are
                                AND'ed.
                              type: string
                          type: object
                      type: object
                  required:
                  - action
                  type: object
                type: array
              namespaceSelector:
                description: NamespaceSelector is an optional field for an expression
                  used to select a pod based on namespaces.
                type: string
              order:
                description: Order is an optional field that specifies the order in
                  which the policy is applied. Policies with higher "order" are applied
                  after those with lower order.  If the order is omitted, it may be
                  considered to be "infinite" - i.e. the policy will be applied last.  Policies
                  with identical order will be applied in alphanumerical order based
                  on the Policy "Name".
                type: number
              preDNAT:
                description: PreDNAT indicates to apply the rules in this policy before
                  any DNAT.
                type: boolean
              selector:
                description: "The selector is an expression used to pick pick out
                  the endpoints that the policy should be applied to. \n Selector
                  expressions follow this syntax: \n \tlabel == \"string_literal\"
                  \ ->  comparison, e.g. my_label == \"foo bar\" \tlabel != \"string_literal\"
                  \  ->  not equal; also matches if label is not present \tlabel in
                  { \"a\", \"b\", \"c\", ... }  ->  true if the value of label X is
                  one of \"a\", \"b\", \"c\" \tlabel not in { \"a\", \"b\", \"c\",
                  ... }  ->  true if the value of label X is not one of \"a\", \"b\",
                  \"c\" \thas(label_name)  -> True if that label is present \t! expr
                  -> negation of expr \texpr && expr  -> Short-circuit and \texpr
                  || expr  -> Short-circuit or \t( expr ) -> parens for grouping \tall()
                  or the empty selector -> matches all endpoints. \n Label names are
                  allowed to contain alphanumerics, -, _ and /. String literals are
                  more permissive but they do not support escape characters. \n Examples
                  (with made-up labels): \n \ttype == \"webserver\" && deployment
                  == \"prod\" \ttype in {\"frontend\", \"backend\"} \tdeployment !=
                  \"dev\" \t! has(label_name)"
                type: string
              serviceAccountSelector:
                description: ServiceAccountSelector is an optional field for an expression
                  used to select a pod based on service accounts.
                type: string
              types:
                description: "Types indicates whether this policy applies to ingress,
                  or to egress, or to both.  When not explicitly specified (and so
                  the value on creation is empty or nil), Calico defaults Types according
                  to what Ingress and Egress rules are present in the policy.  The
                  default is: \n - [ PolicyTypeIngress ], if there are no Egress rules
                  (including the case where there are   also no Ingress rules) \n
                  - [ PolicyTypeEgress ], if there are Egress rules but no Ingress
                  rules \n - [ PolicyTypeIngress, PolicyTypeEgress ], if there are
                  both Ingress and Egress rules. \n When the policy is read back again,
                  Types will always be one of these values, never empty or nil."
                items:
                  description: PolicyType enumerates the possible values of the PolicySpec
                    Types field.
                  type: string
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: globalnetworksets.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: GlobalNetworkSet
    listKind: GlobalNetworkSetList
    plural: globalnetworksets
    singular: globalnetworkset
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: GlobalNetworkSet contains a set of arbitrary IP sub-networks/CIDRs
          that share labels to allow rules to refer to them via selectors.  The labels
          of GlobalNetworkSet are not namespaced.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: GlobalNetworkSetSpec contains the specification for a NetworkSet
              resource.
            properties:
              nets:
                description: The list of IP networks that belong to this set.
                items:
                  type: string
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: hostendpoints.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: HostEndpoint
    listKind: HostEndpointList
    plural: hostendpoints
    singular: hostendpoint
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: HostEndpointSpec contains the specification for a HostEndpoint
              resource.
            properties:
              expectedIPs:
                description: "The expected IP addresses (IPv4 and IPv6) of the endpoint.
                  If \"InterfaceName\" is not present, Calico will look for an interface
                  matching any of the IPs in the list and apply policy to that. Note:
                  \tWhen using the selector match criteria in an ingress or egress
                  security Policy \tor Profile, Calico converts the selector into
                  a set of IP addresses. For host \tendpoints, the ExpectedIPs field
                  is used for that purpose. (If only the interface \tname is specified,
                  Calico does not learn the IPs of the interface for use in match
                  \tcriteria.)"
                items:
                  type: string
                type: array
              interfaceName:
                description: "Either \"*\", or the name of a specific Linux interface
                  to apply policy to; or empty.  \"*\" indicates that this HostEndpoint
                  governs all traffic to, from or through the default network namespace
                  of the host named by the \"Node\" field; entering and leaving that
                  namespace via any interface, including those from/to non-host-networked
                  local workloads. \n If InterfaceName is not \"*\", this HostEndpoint
                  only governs traffic that enters or leaves the host through the
                  specific interface named by InterfaceName, or - when InterfaceName
                  is empty - through the specific interface that has one of the IPs
                  in ExpectedIPs. Therefore, when InterfaceName is empty, at least
                  one expected IP must be specified.  Only external interfaces (such
                  as \"eth0\") are supported here; it isn't possible for a HostEndpoint
                  to protect traffic through a specific local workload interface.
                  \n Note: Only some kinds of policy are implemented for \"*\" HostEndpoints;
                  initially just pre-DNAT policy.  Please check Calico documentation
                  for the latest position."
                type: string
              node:
                description: The node name identifying the Calico node instance.
                type: string
              ports:
                description: Ports contains the endpoint's named ports, which may
                  be referenced in security policy rules.
                items:
                  properties:
                    name:
                      type: string
                    port:
                      type: integer
                    protocol:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                  required:
                  - name
                  - port
                  - protocol
                  type: object
                type: array
              profiles:
                description: A list of identifiers of security Profile objects that
                  apply to this endpoint. Each profile is applied in the order that
                  they appear in this list.  Profile rules are applied after the selector-based
                  security policy.
                items:
                  type: string
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ipamblocks.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: IPAMBlock
    listKind: IPAMBlockList
    plural: ipamblocks
    singular: ipamblock
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: IPAMBlockSpec contains the specification for an IPAMBlock
              resource.
            properties:
              affinity:
                type: string
              allocations:
                items:
                  type: integer
                  # TODO: This nullable is manually added in. We should update controller-gen
                  # to handle []*int properly itself.
                  nullable: true
                type: array
              attributes:
                items:
                  properties:
                    handle_id:
                      type: string
                    secondary:
                      additionalProperties:
                        type: string
                      type: object
                  type: object
                type: array
              cidr:
                type: string
              deleted:
                type: boolean
              strictAffinity:
                type: boolean
              unallocated:
                items:
                  type: integer
                type: array
            required:
            - allocations
            - attributes
            - cidr
            - strictAffinity
            - unallocated
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ipamconfigs.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: IPAMConfig
    listKind: IPAMConfigList
    plural: ipamconfigs
    singular: ipamconfig
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: IPAMConfigSpec contains the specification for an IPAMConfig
              resource.
            properties:
              autoAllocateBlocks:
                type: boolean
              maxBlocksPerHost:
                description: MaxBlocksPerHost, if non-zero, is the max number of blocks
                  that can be affine to each host.
                type: integer
              strictAffinity:
                type: boolean
            required:
            - autoAllocateBlocks
            - strictAffinity
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ipamhandles.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: IPAMHandle
    listKind: IPAMHandleList
    plural: ipamhandles
    singular: ipamhandle
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: IPAMHandleSpec contains the specification for an IPAMHandle
              resource.
            properties:
              block:
                additionalProperties:
                  type: integer
                type: object
              deleted:
                type: boolean
              handleID:
                type: string
            required:
            - block
            - handleID
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ippools.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: IPPool
    listKind: IPPoolList
    plural: ippools
    singular: ippool
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: IPPoolSpec contains the specification for an IPPool resource.
            properties:
              blockSize:
                description: The block size to use for IP address assignments from
                  this pool. Defaults to 26 for IPv4 and 112 for IPv6.
                type: integer
              cidr:
                description: The pool CIDR.
                type: string
              disabled:
                description: When disabled is true, Calico IPAM will not assign addresses
                  from this pool.
                type: boolean
              ipip:
                description: 'Deprecated: this field is only used for APIv1 backwards
                  compatibility. Setting this field is not allowed, this field is
                  for internal use only.'
                properties:
                  enabled:
                    description: When enabled is true, ipip tunneling will be used
                      to deliver packets to destinations within this pool.
                    type: boolean
                  mode:
                    description: The IPIP mode.  This can be one of "always" or "cross-subnet".  A
                      mode of "always" will also use IPIP tunneling for routing to
                      destination IP addresses within this pool.  A mode of "cross-subnet"
                      will only use IPIP tunneling when the destination node is on
                      a different subnet to the originating node.  The default value
                      (if not specified) is "always".
                    type: string
                type: object
              ipipMode:
                description: Contains configuration for IPIP tunneling for this pool.
                  If not specified, then this is defaulted to "Never" (i.e. IPIP tunneling
                  is disabled).
                type: string
              nat-outgoing:
                description: 'Deprecated: this field is only used for APIv1 backwards
                  compatibility. Setting this field is not allowed, this field is
                  for internal use only.'
                type: boolean
              natOutgoing:
                description: When nat-outgoing is true, packets sent from Calico networked
                  containers in this pool to destinations outside of this pool will
                  be masqueraded.
                type: boolean
              nodeSelector:
                description: Allows IPPool to allocate for a specific node by label
                  selector.
                type: string
              vxlanMode:
                description: Contains configuration for VXLAN tunneling for this pool.
                  If not specified, then this is defaulted to "Never" (i.e. VXLAN
                  tunneling is disabled).
                type: string
            required:
            - cidr
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: kubecontrollersconfigurations.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: KubeControllersConfiguration
    listKind: KubeControllersConfigurationList
    plural: kubecontrollersconfigurations
    singular: kubecontrollersconfiguration
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: KubeControllersConfigurationSpec contains the values of the
              Kubernetes controllers configuration.
            properties:
              controllers:
                description: Controllers enables and configures individual Kubernetes
                  controllers
                properties:
                  namespace:
                    description: Namespace enables and configures the namespace controller.
                      Enabled by default, set to nil to disable.
                    properties:
                      reconcilerPeriod:
                        description: 'ReconcilerPeriod is the period to perform reconciliation
                          with the Calico datastore. [Default: 5m]'
                        type: string
                    type: object
                  node:
                    description: Node enables and configures the node controller.
                      Enabled by default, set to nil to disable.
                    properties:
                      hostEndpoint:
                        description: HostEndpoint controls syncing nodes to host endpoints.
                          Disabled by default, set to nil to disable.
                        properties:
                          autoCreate:
                            description: 'AutoCreate enables automatic creation of
                              host endpoints for every node. [Default: Disabled]'
                            type: string
                        type: object
                      reconcilerPeriod:
                        description: 'ReconcilerPeriod is the period to perform reconciliation
                          with the Calico datastore. [Default: 5m]'
                        type: string
                      syncLabels:
                        description: 'SyncLabels controls whether to copy Kubernetes
                          node labels to Calico nodes. [Default: Enabled]'
                        type: string
                    type: object
                  policy:
                    description: Policy enables and configures the policy controller.
                      Enabled by default, set to nil to disable.
                    properties:
                      reconcilerPeriod:
                        description: 'ReconcilerPeriod is the period to perform reconciliation
                          with the Calico datastore. [Default: 5m]'
                        type: string
                    type: object
                  serviceAccount:
                    description: ServiceAccount enables and configures the service
                      account controller. Enabled by default, set to nil to disable.
                    properties:
                      reconcilerPeriod:
                        description: 'ReconcilerPeriod is the period to perform reconciliation
                          with the Calico datastore. [Default: 5m]'
                        type: string
                    type: object
                  workloadEndpoint:
                    description: WorkloadEndpoint enables and configures the workload
                      endpoint controller. Enabled by default, set to nil to disable.
                    properties:
                      reconcilerPeriod:
                        description: 'ReconcilerPeriod is the period to perform reconciliation
                          with the Calico datastore. [Default: 5m]'
                        type: string
                    type: object
                type: object
              etcdV3CompactionPeriod:
                description: 'EtcdV3CompactionPeriod is the period between etcdv3
                  compaction requests. Set to 0 to disable. [Default: 10m]'
                type: string
              healthChecks:
                description: 'HealthChecks enables or disables support for health
                  checks [Default: Enabled]'
                type: string
              logSeverityScreen:
                description: 'LogSeverityScreen is the log severity above which logs
                  are sent to the stdout. [Default: Info]'
                type: string
              prometheusMetricsPort:
                description: 'PrometheusMetricsPort is the TCP port that the Prometheus
                  metrics server should bind to. Set to 0 to disable. [Default: 9094]'
                type: integer
            required:
            - controllers
            type: object
          status:
            description: KubeControllersConfigurationStatus represents the status
              of the configuration. It's useful for admins to be able to see the actual
              config that was applied, which can be modified by environment variables
              on the kube-controllers process.
            properties:
              environmentVars:
                additionalProperties:
                  type: string
                description: EnvironmentVars contains the environment variables on
                  the kube-controllers that influenced the RunningConfig.
                type: object
              runningConfig:
                description: RunningConfig contains the effective config that is running
                  in the kube-controllers pod, after merging the API resource with
                  any environment variables.
                properties:
                  controllers:
                    description: Controllers enables and configures individual Kubernetes
                      controllers
                    properties:
                      namespace:
                        description: Namespace enables and configures the namespace
                          controller. Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform
                              reconciliation with the Calico datastore. [Default:
                              5m]'
                            type: string
                        type: object
                      node:
                        description: Node enables and configures the node controller.
                          Enabled by default, set to nil to disable.
                        properties:
                          hostEndpoint:
                            description: HostEndpoint controls syncing nodes to host
                              endpoints. Disabled by default, set to nil to disable.
                            properties:
                              autoCreate:
                                description: 'AutoCreate enables automatic creation
                                  of host endpoints for every node. [Default: Disabled]'
                                type: string
                            type: object
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform
                              reconciliation with the Calico datastore. [Default:
                              5m]'
                            type: string
                          syncLabels:
                            description: 'SyncLabels controls whether to copy Kubernetes
                              node labels to Calico nodes. [Default: Enabled]'
                            type: string
                        type: object
                      policy:
                        description: Policy enables and configures the policy controller.
                          Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform
                              reconciliation with the Calico datastore. [Default:
                              5m]'
                            type: string
                        type: object
                      serviceAccount:
                        description: ServiceAccount enables and configures the service
                          account controller. Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform
                              reconciliation with the Calico datastore. [Default:
                              5m]'
                            type: string
                        type: object
                      workloadEndpoint:
                        description: WorkloadEndpoint enables and configures the workload
                          endpoint controller. Enabled by default, set to nil to disable.
                        properties:
                          reconcilerPeriod:
                            description: 'ReconcilerPeriod is the period to perform
                              reconciliation with the Calico datastore. [Default:
                              5m]'
                            type: string
                        type: object
                    type: object
                  etcdV3CompactionPeriod:
                    description: 'EtcdV3CompactionPeriod is the period between etcdv3
                      compaction requests. Set to 0 to disable. [Default: 10m]'
                    type: string
                  healthChecks:
                    description: 'HealthChecks enables or disables support for health
                      checks [Default: Enabled]'
                    type: string
                  logSeverityScreen:
                    description: 'LogSeverityScreen is the log severity above which
                      logs are sent to the stdout. [Default: Info]'
                    type: string
                  prometheusMetricsPort:
                    description: 'PrometheusMetricsPort is the TCP port that the Prometheus
                      metrics server should bind to. Set to 0 to disable. [Default:
                      9094]'
                    type: integer
                required:
                - controllers
                type: object
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: networkpolicies.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: NetworkPolicy
    listKind: NetworkPolicyList
    plural: networkpolicies
    singular: networkpolicy
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            properties:
              egress:
                description: The ordered set of egress rules.  Each rule contains
                  a set of packet match criteria and a corresponding action to apply.
                items:
                  description: "A Rule encapsulates a set of match criteria and an
                    action.  Both selector-based security Policy and security Profiles
                    reference rules - separated out as a list of rules for both ingress
                    and egress packet matching. \n Each positive match criteria has
                    a negated version, prefixed with \"Not\". All the match criteria
                    within a rule must be satisfied for a packet to match. A single
                    rule can contain the positive and negative version of a match
                    and both must be satisfied for the rule to match."
                  properties:
                    action:
                      type: string
                    destination:
                      description: Destination contains the match criteria that apply
                        to destination entity.
                      properties:
                        namespaceSelector:
                          description: "NamespaceSelector is an optional field that
                            contains a selector expression. Only traffic that originates
                            from (or terminates at) endpoints within the selected
                            namespaces will be matched. When both NamespaceSelector
                            and Selector are defined on the same rule, then only workload
                            endpoints that are matched by both selectors will be selected
                            by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                            implies that the Selector is limited to selecting only
                            workload endpoints in the same namespace as the NetworkPolicy.
                            \n For NetworkPolicy, `global()` NamespaceSelector implies
                            that the Selector is limited to selecting only GlobalNetworkSet
                            or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                            NamespaceSelector implies the Selector applies to workload
                            endpoints across all namespaces."
                          type: string
                        nets:
                          description: Nets is an optional field that restricts the
                            rule to only apply to traffic that originates from (or
                            terminates at) IP addresses in any of the given subnets.
                          items:
                            type: string
                          type: array
                        notNets:
                          description: NotNets is the negated version of the Nets
                            field.
                          items:
                            type: string
                          type: array
                        notPorts:
                          description: NotPorts is the negated version of the Ports
                            field. Since only some protocols have ports, if any ports
                            are specified it requires the Protocol match in the Rule
                            to be set to "TCP" or "UDP".
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          description: NotSelector is the negated version of the Selector
                            field.  See Selector field for subtleties with negated
                            selectors.
                          type: string
                        ports:
                          description: "Ports is an optional field that restricts
                            the rule to only apply to traffic that has a source (destination)
                            port that matches one of these ranges/values. This value
                            is a list of integers or strings that represent ranges
                            of ports. \n Since only some protocols have ports, if
                            any ports are specified it requires the Protocol match
                            in the Rule to be set to \"TCP\" or \"UDP\"."
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          description: "Selector is an optional field that contains
                            a selector expression (see Policy for sample syntax).
                            \ Only traffic that originates from (terminates at) endpoints
                            matching the selector will be matched. \n Note that: in
                            addition to the negated version of the Selector (see NotSelector
                            below), the selector expression syntax itself supports
                            negation.  The two types of negation are subtly different.
                            One negates the set of matched endpoints, the other negates
                            the whole match: \n \tSelector = \"!has(my_label)\" matches
                            packets that are from other Calico-controlled \tendpoints
                            that do not have the label \"my_label\". \n \tNotSelector
                            = \"has(my_label)\" matches packets that are not from
                            Calico-controlled \tendpoints that do have the label \"my_label\".
                            \n The effect is that the latter will accept packets from
                            non-Calico sources whereas the former is limited to packets
                            from Calico-controlled endpoints."
                          type: string
                        serviceAccounts:
                          description: ServiceAccounts is an optional field that restricts
                            the rule to only apply to traffic that originates from
                            (or terminates at) a pod running as a matching service
                            account.
                          properties:
                            names:
                              description: Names is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account whose name is in the list.
                              items:
                                type: string
                              type: array
                            selector:
                              description: Selector is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account that matches the given label selector. If
                                both Names and Selector are specified then they are
                                AND'ed.
                              type: string
                          type: object
                      type: object
                    http:
                      description: HTTP contains match criteria that apply to HTTP
                        requests.
                      properties:
                        methods:
                          description: Methods is an optional field that restricts
                            the rule to apply only to HTTP requests that use one of
                            the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                            methods are OR'd together.
                          items:
                            type: string
                          type: array
                        paths:
                          description: 'Paths is an optional field that restricts
                            the rule to apply to HTTP requests that use one of the
                            listed HTTP Paths. Multiple paths are OR''d together.
                            e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                            ONLY specify either a `exact` or a `prefix` match. The
                            validator will check for it.'
                          items:
                            description: 'HTTPPath specifies an HTTP path to match.
                              It may be either of the form: exact: <path>: which matches
                              the path exactly or prefix: <path-prefix>: which matches
                              the path prefix'
                            properties:
                              exact:
                                type: string
                              prefix:
                                type: string
                            type: object
                          type: array
                      type: object
                    icmp:
                      description: ICMP is an optional field that restricts the rule
                        to apply to a specific type and code of ICMP traffic.  This
                        should only be specified if the Protocol field is set to "ICMP"
                        or "ICMPv6".
                      properties:
                        code:
                          description: Match on a specific ICMP code.  If specified,
                            the Type value must also be specified. This is a technical
                            limitation imposed by the kernel's iptables firewall,
                            which Calico uses to enforce the rule.
                          type: integer
                        type:
                          description: Match on a specific ICMP type.  For example
                            a value of 8 refers to ICMP Echo Request (i.e. pings).
                          type: integer
                      type: object
                    ipVersion:
                      description: IPVersion is an optional field that restricts the
                        rule to only match a specific IP version.
                      type: integer
                    metadata:
                      description: Metadata contains additional information for this
                        rule
                      properties:
                        annotations:
                          additionalProperties:
                            type: string
                          description: Annotations is a set of key value pairs that
                            give extra information about the rule
                          type: object
                      type: object
                    notICMP:
                      description: NotICMP is the negated version of the ICMP field.
                      properties:
                        code:
                          description: Match on a specific ICMP code.  If specified,
                            the Type value must also be specified. This is a technical
                            limitation imposed by the kernel's iptables firewall,
                            which Calico uses to enforce the rule.
                          type: integer
                        type:
                          description: Match on a specific ICMP type.  For example
                            a value of 8 refers to ICMP Echo Request (i.e. pings).
                          type: integer
                      type: object
                    notProtocol:
                      anyOf:
                      - type: integer
                      - type: string
                      description: NotProtocol is the negated version of the Protocol
                        field.
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    protocol:
                      anyOf:
                      - type: integer
                      - type: string
                      description: "Protocol is an optional field that restricts the
                        rule to only apply to traffic of a specific IP protocol. Required
                        if any of the EntityRules contain Ports (because ports only
                        apply to certain protocols). \n Must be one of these string
                        values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                        \"UDPLite\" or an integer in the range 1-255."
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    source:
                      description: Source contains the match criteria that apply to
                        source entity.
                      properties:
                        namespaceSelector:
                          description: "NamespaceSelector is an optional field that
                            contains a selector expression. Only traffic that originates
                            from (or terminates at) endpoints within the selected
                            namespaces will be matched. When both NamespaceSelector
                            and Selector are defined on the same rule, then only workload
                            endpoints that are matched by both selectors will be selected
                            by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                            implies that the Selector is limited to selecting only
                            workload endpoints in the same namespace as the NetworkPolicy.
                            \n For NetworkPolicy, `global()` NamespaceSelector implies
                            that the Selector is limited to selecting only GlobalNetworkSet
                            or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                            NamespaceSelector implies the Selector applies to workload
                            endpoints across all namespaces."
                          type: string
                        nets:
                          description: Nets is an optional field that restricts the
                            rule to only apply to traffic that originates from (or
                            terminates at) IP addresses in any of the given subnets.
                          items:
                            type: string
                          type: array
                        notNets:
                          description: NotNets is the negated version of the Nets
                            field.
                          items:
                            type: string
                          type: array
                        notPorts:
                          description: NotPorts is the negated version of the Ports
                            field. Since only some protocols have ports, if any ports
                            are specified it requires the Protocol match in the Rule
                            to be set to "TCP" or "UDP".
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          description: NotSelector is the negated version of the Selector
                            field.  See Selector field for subtleties with negated
                            selectors.
                          type: string
                        ports:
                          description: "Ports is an optional field that restricts
                            the rule to only apply to traffic that has a source (destination)
                            port that matches one of these ranges/values. This value
                            is a list of integers or strings that represent ranges
                            of ports. \n Since only some protocols have ports, if
                            any ports are specified it requires the Protocol match
                            in the Rule to be set to \"TCP\" or \"UDP\"."
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          description: "Selector is an optional field that contains
                            a selector expression (see Policy for sample syntax).
                            \ Only traffic that originates from (terminates at) endpoints
                            matching the selector will be matched. \n Note that: in
                            addition to the negated version of the Selector (see NotSelector
                            below), the selector expression syntax itself supports
                            negation.  The two types of negation are subtly different.
                            One negates the set of matched endpoints, the other negates
                            the whole match: \n \tSelector = \"!has(my_label)\" matches
                            packets that are from other Calico-controlled \tendpoints
                            that do not have the label \"my_label\". \n \tNotSelector
                            = \"has(my_label)\" matches packets that are not from
                            Calico-controlled \tendpoints that do have the label \"my_label\".
                            \n The effect is that the latter will accept packets from
                            non-Calico sources whereas the former is limited to packets
                            from Calico-controlled endpoints."
                          type: string
                        serviceAccounts:
                          description: ServiceAccounts is an optional field that restricts
                            the rule to only apply to traffic that originates from
                            (or terminates at) a pod running as a matching service
                            account.
                          properties:
                            names:
                              description: Names is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account whose name is in the list.
                              items:
                                type: string
                              type: array
                            selector:
                              description: Selector is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account that matches the given label selector. If
                                both Names and Selector are specified then they are
                                AND'ed.
                              type: string
                          type: object
                      type: object
                  required:
                  - action
                  type: object
                type: array
              ingress:
                description: The ordered set of ingress rules.  Each rule contains
                  a set of packet match criteria and a corresponding action to apply.
                items:
                  description: "A Rule encapsulates a set of match criteria and an
                    action.  Both selector-based security Policy and security Profiles
                    reference rules - separated out as a list of rules for both ingress
                    and egress packet matching. \n Each positive match criteria has
                    a negated version, prefixed with \"Not\". All the match criteria
                    within a rule must be satisfied for a packet to match. A single
                    rule can contain the positive and negative version of a match
                    and both must be satisfied for the rule to match."
                  properties:
                    action:
                      type: string
                    destination:
                      description: Destination contains the match criteria that apply
                        to destination entity.
                      properties:
                        namespaceSelector:
                          description: "NamespaceSelector is an optional field that
                            contains a selector expression. Only traffic that originates
                            from (or terminates at) endpoints within the selected
                            namespaces will be matched. When both NamespaceSelector
                            and Selector are defined on the same rule, then only workload
                            endpoints that are matched by both selectors will be selected
                            by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                            implies that the Selector is limited to selecting only
                            workload endpoints in the same namespace as the NetworkPolicy.
                            \n For NetworkPolicy, `global()` NamespaceSelector implies
                            that the Selector is limited to selecting only GlobalNetworkSet
                            or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                            NamespaceSelector implies the Selector applies to workload
                            endpoints across all namespaces."
                          type: string
                        nets:
                          description: Nets is an optional field that restricts the
                            rule to only apply to traffic that originates from (or
                            terminates at) IP addresses in any of the given subnets.
                          items:
                            type: string
                          type: array
                        notNets:
                          description: NotNets is the negated version of the Nets
                            field.
                          items:
                            type: string
                          type: array
                        notPorts:
                          description: NotPorts is the negated version of the Ports
                            field. Since only some protocols have ports, if any ports
                            are specified it requires the Protocol match in the Rule
                            to be set to "TCP" or "UDP".
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          description: NotSelector is the negated version of the Selector
                            field.  See Selector field for subtleties with negated
                            selectors.
                          type: string
                        ports:
                          description: "Ports is an optional field that restricts
                            the rule to only apply to traffic that has a source (destination)
                            port that matches one of these ranges/values. This value
                            is a list of integers or strings that represent ranges
                            of ports. \n Since only some protocols have ports, if
                            any ports are specified it requires the Protocol match
                            in the Rule to be set to \"TCP\" or \"UDP\"."
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          description: "Selector is an optional field that contains
                            a selector expression (see Policy for sample syntax).
                            \ Only traffic that originates from (terminates at) endpoints
                            matching the selector will be matched. \n Note that: in
                            addition to the negated version of the Selector (see NotSelector
                            below), the selector expression syntax itself supports
                            negation.  The two types of negation are subtly different.
                            One negates the set of matched endpoints, the other negates
                            the whole match: \n \tSelector = \"!has(my_label)\" matches
                            packets that are from other Calico-controlled \tendpoints
                            that do not have the label \"my_label\". \n \tNotSelector
                            = \"has(my_label)\" matches packets that are not from
                            Calico-controlled \tendpoints that do have the label \"my_label\".
                            \n The effect is that the latter will accept packets from
                            non-Calico sources whereas the former is limited to packets
                            from Calico-controlled endpoints."
                          type: string
                        serviceAccounts:
                          description: ServiceAccounts is an optional field that restricts
                            the rule to only apply to traffic that originates from
                            (or terminates at) a pod running as a matching service
                            account.
                          properties:
                            names:
                              description: Names is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account whose name is in the list.
                              items:
                                type: string
                              type: array
                            selector:
                              description: Selector is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account that matches the given label selector. If
                                both Names and Selector are specified then they are
                                AND'ed.
                              type: string
                          type: object
                      type: object
                    http:
                      description: HTTP contains match criteria that apply to HTTP
                        requests.
                      properties:
                        methods:
                          description: Methods is an optional field that restricts
                            the rule to apply only to HTTP requests that use one of
                            the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple
                            methods are OR'd together.
                          items:
                            type: string
                          type: array
                        paths:
                          description: 'Paths is an optional field that restricts
                            the rule to apply to HTTP requests that use one of the
                            listed HTTP Paths. Multiple paths are OR''d together.
                            e.g: - exact: /foo - prefix: /bar NOTE: Each entry may
                            ONLY specify either a `exact` or a `prefix` match. The
                            validator will check for it.'
                          items:
                            description: 'HTTPPath specifies an HTTP path to match.
                              It may be either of the form: exact: <path>: which matches
                              the path exactly or prefix: <path-prefix>: which matches
                              the path prefix'
                            properties:
                              exact:
                                type: string
                              prefix:
                                type: string
                            type: object
                          type: array
                      type: object
                    icmp:
                      description: ICMP is an optional field that restricts the rule
                        to apply to a specific type and code of ICMP traffic.  This
                        should only be specified if the Protocol field is set to "ICMP"
                        or "ICMPv6".
                      properties:
                        code:
                          description: Match on a specific ICMP code.  If specified,
                            the Type value must also be specified. This is a technical
                            limitation imposed by the kernel's iptables firewall,
                            which Calico uses to enforce the rule.
                          type: integer
                        type:
                          description: Match on a specific ICMP type.  For example
                            a value of 8 refers to ICMP Echo Request (i.e. pings).
                          type: integer
                      type: object
                    ipVersion:
                      description: IPVersion is an optional field that restricts the
                        rule to only match a specific IP version.
                      type: integer
                    metadata:
                      description: Metadata contains additional information for this
                        rule
                      properties:
                        annotations:
                          additionalProperties:
                            type: string
                          description: Annotations is a set of key value pairs that
                            give extra information about the rule
                          type: object
                      type: object
                    notICMP:
                      description: NotICMP is the negated version of the ICMP field.
                      properties:
                        code:
                          description: Match on a specific ICMP code.  If specified,
                            the Type value must also be specified. This is a technical
                            limitation imposed by the kernel's iptables firewall,
                            which Calico uses to enforce the rule.
                          type: integer
                        type:
                          description: Match on a specific ICMP type.  For example
                            a value of 8 refers to ICMP Echo Request (i.e. pings).
                          type: integer
                      type: object
                    notProtocol:
                      anyOf:
                      - type: integer
                      - type: string
                      description: NotProtocol is the negated version of the Protocol
                        field.
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    protocol:
                      anyOf:
                      - type: integer
                      - type: string
                      description: "Protocol is an optional field that restricts the
                        rule to only apply to traffic of a specific IP protocol. Required
                        if any of the EntityRules contain Ports (because ports only
                        apply to certain protocols). \n Must be one of these string
                        values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\",
                        \"UDPLite\" or an integer in the range 1-255."
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    source:
                      description: Source contains the match criteria that apply to
                        source entity.
                      properties:
                        namespaceSelector:
                          description: "NamespaceSelector is an optional field that
                            contains a selector expression. Only traffic that originates
                            from (or terminates at) endpoints within the selected
                            namespaces will be matched. When both NamespaceSelector
                            and Selector are defined on the same rule, then only workload
                            endpoints that are matched by both selectors will be selected
                            by the rule. \n For NetworkPolicy, an empty NamespaceSelector
                            implies that the Selector is limited to selecting only
                            workload endpoints in the same namespace as the NetworkPolicy.
                            \n For NetworkPolicy, `global()` NamespaceSelector implies
                            that the Selector is limited to selecting only GlobalNetworkSet
                            or HostEndpoint. \n For GlobalNetworkPolicy, an empty
                            NamespaceSelector implies the Selector applies to workload
                            endpoints across all namespaces."
                          type: string
                        nets:
                          description: Nets is an optional field that restricts the
                            rule to only apply to traffic that originates from (or
                            terminates at) IP addresses in any of the given subnets.
                          items:
                            type: string
                          type: array
                        notNets:
                          description: NotNets is the negated version of the Nets
                            field.
                          items:
                            type: string
                          type: array
                        notPorts:
                          description: NotPorts is the negated version of the Ports
                            field. Since only some protocols have ports, if any ports
                            are specified it requires the Protocol match in the Rule
                            to be set to "TCP" or "UDP".
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          description: NotSelector is the negated version of the Selector
                            field.  See Selector field for subtleties with negated
                            selectors.
                          type: string
                        ports:
                          description: "Ports is an optional field that restricts
                            the rule to only apply to traffic that has a source (destination)
                            port that matches one of these ranges/values. This value
                            is a list of integers or strings that represent ranges
                            of ports. \n Since only some protocols have ports, if
                            any ports are specified it requires the Protocol match
                            in the Rule to be set to \"TCP\" or \"UDP\"."
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          description: "Selector is an optional field that contains
                            a selector expression (see Policy for sample syntax).
                            \ Only traffic that originates from (terminates at) endpoints
                            matching the selector will be matched. \n Note that: in
                            addition to the negated version of the Selector (see NotSelector
                            below), the selector expression syntax itself supports
                            negation.  The two types of negation are subtly different.
                            One negates the set of matched endpoints, the other negates
                            the whole match: \n \tSelector = \"!has(my_label)\" matches
                            packets that are from other Calico-controlled \tendpoints
                            that do not have the label \"my_label\". \n \tNotSelector
                            = \"has(my_label)\" matches packets that are not from
                            Calico-controlled \tendpoints that do have the label \"my_label\".
                            \n The effect is that the latter will accept packets from
                            non-Calico sources whereas the former is limited to packets
                            from Calico-controlled endpoints."
                          type: string
                        serviceAccounts:
                          description: ServiceAccounts is an optional field that restricts
                            the rule to only apply to traffic that originates from
                            (or terminates at) a pod running as a matching service
                            account.
                          properties:
                            names:
                              description: Names is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account whose name is in the list.
                              items:
                                type: string
                              type: array
                            selector:
                              description: Selector is an optional field that restricts
                                the rule to only apply to traffic that originates
                                from (or terminates at) a pod running as a service
                                account that matches the given label selector. If
                                both Names and Selector are specified then they are
                                AND'ed.
                              type: string
                          type: object
                      type: object
                  required:
                  - action
                  type: object
                type: array
              order:
                description: Order is an optional field that specifies the order in
                  which the policy is applied. Policies with higher "order" are applied
                  after those with lower order.  If the order is omitted, it may be
                  considered to be "infinite" - i.e. the policy will be applied last.  Policies
                  with identical order will be applied in alphanumerical order based
                  on the Policy "Name".
                type: number
              selector:
                description: "The selector is an expression used to pick pick out
                  the endpoints that the policy should be applied to. \n Selector
                  expressions follow this syntax: \n \tlabel == \"string_literal\"
                  \ ->  comparison, e.g. my_label == \"foo bar\" \tlabel != \"string_literal\"
                  \  ->  not equal; also matches if label is not present \tlabel in
                  { \"a\", \"b\", \"c\", ... }  ->  true if the value of label X is
                  one of \"a\", \"b\", \"c\" \tlabel not in { \"a\", \"b\", \"c\",
                  ... }  ->  true if the value of label X is not one of \"a\", \"b\",
                  \"c\" \thas(label_name)  -> True if that label is present \t! expr
                  -> negation of expr \texpr && expr  -> Short-circuit and \texpr
                  || expr  -> Short-circuit or \t( expr ) -> parens for grouping \tall()
                  or the empty selector -> matches all endpoints. \n Label names are
                  allowed to contain alphanumerics, -, _ and /. String literals are
                  more permissive but they do not support escape characters. \n Examples
                  (with made-up labels): \n \ttype == \"webserver\" && deployment
                  == \"prod\" \ttype in {\"frontend\", \"backend\"} \tdeployment !=
                  \"dev\" \t! has(label_name)"
                type: string
              serviceAccountSelector:
                description: ServiceAccountSelector is an optional field for an expression
                  used to select a pod based on service accounts.
                type: string
              types:
                description: "Types indicates whether this policy applies to ingress,
                  or to egress, or to both.  When not explicitly specified (and so
                  the value on creation is empty or nil), Calico defaults Types according
                  to what Ingress and Egress are present in the policy.  The default
                  is: \n - [ PolicyTypeIngress ], if there are no Egress rules (including
                  the case where there are   also no Ingress rules) \n - [ PolicyTypeEgress
                  ], if there are Egress rules but no Ingress rules \n - [ PolicyTypeIngress,
                  PolicyTypeEgress ], if there are both Ingress and Egress rules.
                  \n When the policy is read back again, Types will always be one
                  of these values, never empty or nil."
                items:
                  description: PolicyType enumerates the possible values of the PolicySpec
                    Types field.
                  type: string
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: networksets.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: NetworkSet
    listKind: NetworkSetList
    plural: networksets
    singular: networkset
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: NetworkSet is the Namespaced-equivalent of the GlobalNetworkSet.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: NetworkSetSpec contains the specification for a NetworkSet
              resource.
            properties:
              nets:
                description: The list of IP networks that belong to this set.
                items:
                  type: string
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
---
# Source: calico/templates/calico-kube-controllers-rbac.yaml
# Include a clusterrole for the kube-controllers component,
# and bind it to the calico-kube-controllers serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
rules:
  # Nodes are watched to monitor for deletions.
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - watch
      - list
      - get
  # Pods are queried to check for existence.
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
  # IPAM resources are manipulated when nodes are deleted.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
    verbs:
      - list
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
      - ipamblocks
      - ipamhandles
    verbs:
      - get
      - list
      - create
      - update
      - delete
      - watch
  # kube-controllers manages hostendpoints.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - hostendpoints
    verbs:
      - get
      - list
      - create
      - update
      - delete
  # Needs access to update clusterinformations.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - clusterinformations
    verbs:
      - get
      - create
      - update
  # KubeControllersConfiguration is where it gets its config
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - kubecontrollersconfigurations
    verbs:
      # read its own config
      - get
      # create a default if none exists
      - create
      # update status
      - update
      # watch for changes
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-kube-controllers
subjects:
- kind: ServiceAccount
  name: calico-kube-controllers
  namespace: kube-system
---
---
# Source: calico/templates/calico-node-rbac.yaml
# Include a clusterrole for the calico-node DaemonSet,
# and bind it to the calico-node serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-node
rules:
  # The CNI plugin needs to get pods, nodes, and namespaces.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - endpoints
      - services
    verbs:
      # Used to discover service IPs for advertisement.
      - watch
      - list
      # Used to discover Typhas.
      - get
  # Pod CIDR auto-detection on kubeadm needs access to config maps.
  - apiGroups: [""]
    resources:
      - configmaps
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - nodes/status
    verbs:
      # Needed for clearing NodeNetworkUnavailable flag.
      - patch
      # Calico stores some configuration information in node annotations.
      - update
  # Watch for changes to Kubernetes NetworkPolicies.
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
    verbs:
      - watch
      - list
  # Used by Calico for policy information.
  - apiGroups: [""]
    resources:
      - pods
      - namespaces
      - serviceaccounts
    verbs:
      - list
      - watch
  # The CNI plugin patches pods/status.
  - apiGroups: [""]
    resources:
      - pods/status
    verbs:
      - patch
  # Calico monitors various CRDs for config.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - globalfelixconfigs
      - felixconfigurations
      - bgppeers
      - globalbgpconfigs
      - bgpconfigurations
      - ippools
      - ipamblocks
      - globalnetworkpolicies
      - globalnetworksets
      - networkpolicies
      - networksets
      - clusterinformations
      - hostendpoints
      - blockaffinities
    verbs:
      - get
      - list
      - watch
  # Calico must create and update some CRDs on startup.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
      - felixconfigurations
      - clusterinformations
    verbs:
      - create
      - update
  # Calico stores some configuration information on the node.
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
  # These permissions are only required for upgrade from v2.6, and can
  # be removed after upgrade or on fresh installations.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - bgpconfigurations
      - bgppeers
    verbs:
      - create
      - update
  # These permissions are required for Calico CNI to perform IPAM allocations.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
      - ipamblocks
      - ipamhandles
    verbs:
      - get
      - list
      - create
      - update
      - delete
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ipamconfigs
    verbs:
      - get
  # Block affinities must also be watchable by confd for route aggregation.
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
    verbs:
      - watch
  # The Calico IPAM migration needs to get daemonsets. These permissions can be
  # removed if not upgrading from an installation using host-local IPAM.
  - apiGroups: ["apps"]
    resources:
      - daemonsets
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: calico-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-node
subjects:
- kind: ServiceAccount
  name: calico-node
  namespace: kube-system
---
# Source: calico/templates/calico-node.yaml
# This manifest installs the calico-node container, as well
# as the CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        k8s-app: calico-node
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      hostNetwork: true
      tolerations:
        # Make sure calico-node gets scheduled on all nodes.
        - effect: NoSchedule
          operator: Exists
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoExecute
          operator: Exists
      serviceAccountName: calico-node
      # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
      # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
      terminationGracePeriodSeconds: 0
      priorityClassName: system-node-critical
      initContainers:
        # This container performs upgrade from host-local IPAM to calico-ipam.
        # It can be deleted if this is a fresh installation, or if you have already
        # upgraded to use calico-ipam.
        - name: upgrade-ipam
          image: docker.io/calico/cni:v3.18.0
          command: ["/opt/cni/bin/calico-ipam", "-upgrade"]
          envFrom:
          - configMapRef:
              # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
              name: kubernetes-services-endpoint
              optional: true
          env:
            - name: KUBERNETES_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
          volumeMounts:
            - mountPath: /var/lib/cni/networks
              name: host-local-net-dir
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
          securityContext:
            privileged: true
        # This container installs the CNI binaries
        # and CNI network config file on each node.
        - name: install-cni
          image: docker.io/calico/cni:v3.18.0
          command: ["/opt/cni/bin/install"]
          envFrom:
          - configMapRef:
              # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
              name: kubernetes-services-endpoint
              optional: true
          env:
            # Name of the CNI config file to create.
            - name: CNI_CONF_NAME
              value: "10-calico.conflist"
            # The CNI network config to install on each node.
            - name: CNI_NETWORK_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cni_network_config
            # Set the hostname based on the k8s node name.
            - name: KUBERNETES_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # CNI MTU Config variable
            - name: CNI_MTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Prevents the container from sleeping forever.
            - name: SLEEP
              value: "false"
          volumeMounts:
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
          securityContext:
            privileged: true
        # Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes
        # to communicate with Felix over the Policy Sync API.
        - name: flexvol-driver
          image: docker.io/calico/pod2daemon-flexvol:v3.18.0
          volumeMounts:
          - name: flexvol-driver-host
            mountPath: /host/driver
          securityContext:
            privileged: true
      containers:
        # Runs calico-node container on each Kubernetes node. This
        # container programs network policy and routes on each
        # host.
        - name: calico-node
          image: docker.io/calico/node:v3.18.0
          envFrom:
          - configMapRef:
              # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
              name: kubernetes-services-endpoint
              optional: true
          env:
            # Use Kubernetes API as the backing datastore.
            - name: DATASTORE_TYPE
              value: "kubernetes"
            # Wait for the datastore.
            - name: WAIT_FOR_DATASTORE
              value: "true"
            # Set based on the k8s node name.
            - name: NODENAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # Choose the backend to use.
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            # Cluster type to identify the deployment type
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            # Auto-detect the BGP IP address.
            - name: IP
              value: "autodetect"
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Always"
            # Enable or Disable VXLAN on the default IP pool.
            - name: CALICO_IPV4POOL_VXLAN
              value: "Never"
            # Set MTU for tunnel device used if ipip is enabled
            - name: FELIX_IPINIPMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Set MTU for the VXLAN tunnel device.
            - name: FELIX_VXLANMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Set MTU for the Wireguard tunnel device.
            - name: FELIX_WIREGUARDMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            # - name: CALICO_IPV4POOL_CIDR
            #   value: "192.168.0.0/16"
            # Disable file logging so `kubectl logs` works.
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            # Set Felix endpoint to host default action to ACCEPT.
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "ACCEPT"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "false"
            # Set Felix logging to "info"
            - name: FELIX_LOGSEVERITYSCREEN
              value: "info"
            - name: FELIX_HEALTHENABLED
              value: "true"
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 250m
          livenessProbe:
            exec:
              command:
              - /bin/calico-node
              - -felix-live
              - -bird-live
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
          readinessProbe:
            exec:
              command:
              - /bin/calico-node
              - -felix-ready
              - -bird-ready
            periodSeconds: 10
          volumeMounts:
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /run/xtables.lock
              name: xtables-lock
              readOnly: false
            - mountPath: /var/run/calico
              name: var-run-calico
              readOnly: false
            - mountPath: /var/lib/calico
              name: var-lib-calico
              readOnly: false
            - name: policysync
              mountPath: /var/run/nodeagent
            # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
            # parent directory.
            - name: sysfs
              mountPath: /sys/fs/
              # Bidirectional means that, if we mount the BPF filesystem at /sys/fs/bpf it will propagate to the host.
              # If the host is known to mount that filesystem already then Bidirectional can be omitted.
              mountPropagation: Bidirectional
            - name: cni-log-dir
              mountPath: /var/log/calico/cni
              readOnly: true
      volumes:
        # Used by calico-node.
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
        - name: var-lib-calico
          hostPath:
            path: /var/lib/calico
        - name: xtables-lock
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
        - name: sysfs
          hostPath:
            path: /sys/fs/
            type: DirectoryOrCreate
        # Used to install CNI.
        - name: cni-bin-dir
          hostPath:
            path: /opt/cni/bin
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d
        # Used to access CNI logs.
        - name: cni-log-dir
          hostPath:
            path: /var/log/calico/cni
        # Mount in the directory for host-local IPAM allocations. This is
        # used when upgrading from host-local to calico-ipam, and can be removed
        # if not using the upgrade-ipam init container.
        - name: host-local-net-dir
          hostPath:
            path: /var/lib/cni/networks
        # Used to create per-pod Unix Domain Sockets
        - name: policysync
          hostPath:
            type: DirectoryOrCreate
            path: /var/run/nodeagent
        # Used to install Flex Volume Driver
        - name: flexvol-driver-host
          hostPath:
            type: DirectoryOrCreate
            path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-node
  namespace: kube-system
---
# Source: calico/templates/calico-kube-controllers.yaml
# See https://github.com/projectcalico/kube-controllers
apiVersion: apps/v1
kind: Deployment
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
spec:
  # The controllers can only have a single active instance.
  replicas: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-kube-controllers
      namespace: kube-system
      labels:
        k8s-app: calico-kube-controllers
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      serviceAccountName: calico-kube-controllers
      priorityClassName: system-cluster-critical
      containers:
        - name: calico-kube-controllers
          image: docker.io/calico/kube-controllers:v3.18.0
          env:
            # Choose which controllers to run.
            - name: ENABLED_CONTROLLERS
              value: node
            - name: DATASTORE_TYPE
              value: kubernetes
          readinessProbe:
            exec:
              command:
              - /usr/bin/check-status
              - -r
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-kube-controllers
  namespace: kube-system
---
# This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers
---
# Source: calico/templates/calico-etcd-secrets.yaml
---
# Source: calico/templates/calico-typha.yaml
---
# Source: calico/templates/configure-canal.yaml