Containerd ctr、crictl、nerdctl 客户端命令介绍与实战操作

一、概述

https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/#docker

二、Containerd 常见命令操作

• crictl** 是遵循 CRI 接口规范的一个命令行工具，通常用它来检查和管理kubelet**节点上的容器运行时和镜像。
• ctr** 是 containerd **的一个客户端工具。
• ctr -v** 输出的是 containerd 的版本，crictl -v 输出的是当前 k8s 的版本，从结果显而易见可以认为 crictl 是用于 k8s 的。**
• 一般来说某个主机安装了 k8s 后，命令行才会有 crictl 命令。而 ctr 是跟 k8s 无关的，主机安装了 containerd 服务后就可以操作 ctr 命令。

runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false

crictl config runtime-endpoint unix:///run/containerd/containerd.sock
crictl config image-endpoint unix:///run/containerd/containerd.sock

docker --help
ctr --help
crictl --help

# 注意-n不能放在命令最后面，下面几行查看的镜像是一样的
ctr -n=k8s.io image ls
ctr -n k8s.io image ls
# crictl 没有-n参数，操作都在`k8s.io`命名空间下。
crictl image ls
crictl images
# crictl image list = ctr -n=k8s.io image list
# crictl image ls = ctr -n=k8s.io image ls
# crictl images = ctr -n=k8s.io image list
# crictl images = ctr -n=k8s.io image ls
# 使用ctr命令指定命名空间导入镜像
ctr -n=k8s.io image import dashboard.tar
#查看镜像，可以看到可以查询到了
crictl images

三、containerd 客户端工具 nerdctl

https://github.com/containerd/nerdctl/releases

• 精简 (nerdctl—linux-amd64.tar.gz): 只包含 nerdctl
• 完整 (nerdctl-full—linux-amd64.tar.gz): **包含 containerd, runc, and CNI 等依赖**

延迟拉取镜像功能可以参考这篇文章：Containerd 使用 Stargz Snapshotter 延迟拉取镜像

https://icloudnative.io/posts/startup-containers-in-lightning-speed-with-lazy-image-distribution-on-containerd/

1）安装 nerdctl（精简版）

wget https://github.com/containerd/nerdctl/releases/download/v0.22.2/nerdctl-0.22.2-linux-amd64.tar.gz
# 解压
tar -xf nerdctl-0.22.2-linux-amd64.tar.gz
ln -s /opt/k8s/nerdctl/nerdctl /usr/local/bin/nerdctl

2）安装 nerdctl（完整版，这里不装）

wget https://github.com/containerd/nerdctl/releases/download/v0.22.2/nerdctl-full-0.22.2-linux-amd64.tar.gz
tar -xf nerdctl-full-0.16.0-linux-amd64.tar.gz -C /usr/local/
cp /usr/local/lib/systemd/system/*.service /etc/systemd/system/

systemctl enable  buildkit containerd --now
systemctl status buildkit containerd

3）安装 buildkit 支持构建镜像

• 服务端 buildkitd，当前支持 runc 和 containerd 作为 worker，默认是 runc；
• 客户端 buildctl，负责解析 Dockerfile，并向服务端 buildkitd 发出构建请求。

# https://github.com/moby/buildkit/releases
wget https://github.com/moby/buildkit/releases/download/v0.10.4/buildkit-v0.10.4.linux-amd64.tar.gz
tar -xf buildkit-v0.10.4.linux-amd64.tar.gz  -C /usr/local/

• /usr/lib/systemd/system/buildkit.socket

cat > /usr/lib/systemd/system/buildkit.socket <<EOF
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit
[Socket]
ListenStream=%t/buildkit/buildkitd.sock
SocketMode=0660
[Install]
WantedBy=sockets.target
EOF

• /usr/lib/systemd/system/buildkit.service

cat > /usr/lib/systemd/system/buildkit.service << EOF
[Unit]
Description=BuildKit
Requires=buildkit.socket
After=buildkit.socket
Documentation=https://github.com/moby/buildkit
[Service]
# Replace runc builds with containerd builds
ExecStart=/usr/local/bin/buildkitd --addr fd://
[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable buildkit --now

四、实战操作

1）修改 containerd 配置文件

containerd config default > /etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""
      [plugins."io.containerd.grpc.v1.cri".registry.auths]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."myharbor-minio.com".tls]
          insecure_skip_verify = true  #跳过认证
          ca_file = "/etc/containerd/myharbor-minio.com/ca.crt"
        [plugins."io.containerd.grpc.v1.cri".registry.configs."myharbor-minio.com".auth]
          username = "admin"
          password = "Harbor12345"
      [plugins."io.containerd.grpc.v1.cri".registry.headers]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."myharbor-minio.com"]
          endpoint = ["https://myharbor-minio.com"]

重启 containerd

#重新加载配置
systemctl daemon-reload
#重启containerd
systemctl restart containerd

2）ctr 拉取推送镜像

# 推送镜像到harbor
ctr --namespace=k8s.io images push myharbor-minio.com/bigdata/minio:2022.8.22-debian-11-r0 --skip-verify --user admin:Harbor12345
# --namespace=k8s.io 指定命名空间，不是必须，根据环境而定
# --skip-verify 跳过认证
# --user 指定harbor用户名及密码
ctr  images pull --user admin:Harbor12345  --tlscacert=/etc/containerd/myharbor-minio.com/ca.crt myharbor-minio.com/bigdata/minio:2022.8.22-debian-11-r0

3）镜像构建

cat > Dockerfile <<EOF
FROM nginx:alpine
RUN echo 'Hello Nerdctl From Containerd' > /usr/share/nginx/html/index.html
EOF

# 不加-n指定命名空间，crictl看不到，kubelet也不能使用它，默认在default命名空间下
nerdctl -n k8s.io build -t nginx:nerctl -f ./Dockerfile .
### 参数解释
# -t：指定镜像名称
# . ：当前目录Dockerfile
# -f：指定Dockerfile路径
#  --no-cache：不缓存

4）打标签 tag

# crictl没有tag命令，只能使用nerdctl和ctr，必须指定命名空间，要不然kubelet无法使用。
ctr -n k8s.io i tag
nerdctl -n k8s.io tag nginx:nerctl myharbor-minio.com/bigdata/nginx:nerctl
# ctr -n k8s.io tag nginx:nerctl myharbor-minio.com/bigdata/nginx:nerctl
# 查看镜像
nerdctl  -n k8s.io  images myharbor-minio.com/bigdata/nginx:nerctl

5）将镜像推送到 Harbor

# 以下两个哪个都可以
# mkdir -p /etc/docker/certs.d/myharbor-minio.com:443
mkdir -p /etc/containerd/certs.d/myharbor-minio.com:443
cat > /etc/containerd/certs.d/myharbor-minio.com\:443/hosts.toml <<EOF
server = "https://docker.io"
[host."http://myharbor-minio.com:80"]
  capabilities = ["pull", "resolve","push"]
  #skip_verify = true
  #ca = "ca.crt"   #相对路径
  #ca = "/opt/auth/ca.crt"  #绝对路径
  #ca = ["/opt/auth/ca.crt"]
  #ca = ["ca.crt"]
  #client = [["/opt/auth/nginx.cclinux.cn.crt", "/opt/auth/nginx.cclinux.cn.key"]]
EOF

# 以下两个哪个都可以
# mkdir -p /etc/docker/certs.d/myharbor-minio.com:443
mkdir -p /etc/containerd/certs.d/myharbor-minio.com:443
cat > /etc/containerd/certs.d/myharbor-minio.com\:443/hosts.toml <<EOF
server = "https://docker.io"
[host."https://myharbor-minio.com:443"]
  capabilities = ["pull", "resolve","push"]
  skip_verify = true
  #ca = "ca.crt"   #相对路径
  #ca = "/opt/auth/ca.crt"  #绝对路径
  #ca = ["/opt/auth/ca.crt"]
  ca = ["/etc/containerd/myharbor-minio.com/ca.crt"]
  #client = [["/opt/auth/nginx.cclinux.cn.crt", "/opt/auth/nginx.cclinux.cn.key"]]
EOF

echo Harbor12345 | nerdctl login --username "admin" --password-stdin  myharbor-minio.com:443
# nerdctl login --username "admin" --password Harbor12345 myharbor-minio.com:443
# 登出
# nerdctl logout

### 推送到Harbor
# --insecure-registry        skips verifying HTTPS certs, and allows falling back to plain HTTP
nerdctl --insecure-registry --namespace=k8s.io push myharbor-minio.com/bigdata/nginx:nerctl
# ctr --namespace=k8s.io images push myharbor-minio.com/bigdata/nginx:nerctl --skip-verify --user admin:Harbor12345
# --namespace=k8s.io 指定命名空间，跟-n一样，不是必须，根据环境而定
# --skip-verify 跳过认证
# --user 指定harbor用户名及密码