背景

当前 ciro 版本

  1. $ ciro version
  3. crio version 1.22.1
  4. Version:          1.22.1
  5. GitCommit:        63ca93845d5fe05cdca826367afcb601ece8d7ad
  6. GitTreeState:     clean
  7. BuildDate:        2022-01-29T08:15:54Z
  8. GoVersion:        go1.16.13
  9. Compiler:         gc
  10. Platform:         linux/amd64
  11. Linkmode:         dynamic
  12. BuildTags:        exclude_graphdriver_devicemapper, seccomp
  13. SeccompEnabled:   true
  14. AppArmorEnabled:  false

遇到问题，通过crictl下载大于1GB镜像时候，经常hung住15~30分钟，有一定失败率，下载10GB本地仓库基本有时候需要1个小时，甚至经常失败

  1. sudo /usr/local/bin/crictl pull <image-name> 
  3. FATA[0333] pulling image: rpc error: code = Unknown desc = writing blob: storing blob to file "/var/tmp/storage038803475/2": happened during read: unexpected EOF
  5. Failed to pull image "xxxx.xxxx/gpu/app/py-gpu-service:cuda11-0.0.1.dev.2": rpc error: code = Unknown desc = writing blob: storing blob to file "/var/tmp/storage989958523/4": happened during read: context canceled

打开 crio debug日志 /etc/crio/crio.conf

  1. [crio.runtime]
  2. ...
  3. # Changes the verbosity of the logs based on the level it is set to. Options
  4. # are fatal, panic, error, warn, info, debug and trace. This option supports
  5. # live configuration reload.
  6. log_level = "info"
  1. $ sudo systemctl restart crio

/var/log/message

Failed to retrieve partial blob: blob type not supported for partial retrieval

错误描述参考

Failed to retrieve partial blob: blob type not supported for partial retrieval I’m pulling images that are constructed in an iterative fashion (each image is constructed from the previous one) and the reproducibility seems to vary depending on the layer. For instance, I have an image that only adds an environment variable (the filesystem diff is empty) and that one seems to reproduce the issue more than the others.

Besides, the images are constructed with buildah but then pushed to the local docker daemon which, in turn, pushes them to an AWS ECR registry.

  1.    vbauerster/mpb#100 包导致, 需要升级 Update to github.com/vbauerster/mpb v7.1.5 to fix [vbauerster/mpb#100](https://github.com/vbauerster/mpb/issues/100) .
  2. <a name="DFfJL"></a>
  3. ### 修复方法
  4. crio 升级 1.23 版本
  5. <a name="z2uzU"></a>
  6. ## 解决步骤
  7. <a name="KkiOV"></a>
  8. ### 域名重定向配置x509不支持ignoreCN 问题
  9. 私有仓库和`gcr.io`,`quay.io`,`k8s.gcr.io`不能通过https访问，可以通过perfix重定向通过ip访问
  11. ```bash
  12. [[registry]]
  13. prefix = "docker.io"
  14. insecure = false
  15. blocked = false
  16. location = "registry.docker-cn.com"
  18. [[registry.mirror]]
  19. location = "docker.mirrors.ustc.edu.cn"
  20. insecure = true
  22. [[registry.mirror]]
  23. location = "hub-mirror.c.163.com"
  24. insecure = true
  26. [[registry]]
  27. prefix = "docker.your-comany.local"
  28. insecure = true
  29. blocked = false
  30. location = "harbor:port"
  32. [[registry]]
  33. prefix = "gcr.io"
  34. insecure = true
  35. blocked = false
  36. location = "grc-mirrors-have-proxy_ip_1:port"
  38. [[registry.mirror]]
  39. location = "grc-mirrors-have-proxy_ip_2:port"
  40. insecure = true
  42. [[registry]]
  43. prefix = "quay.io"
  44. insecure = true
  45. blocked = false
  46. location = "quay-mirrors-have-proxy_ip_1:port"
  48. [[registry.mirror]]
  49. location = "quay-mirrors-have-proxy_ip_2:port"
  50. insecure = true
  52. [[registry]]
  53. prefix = "k8s.gcr.io"
  54. insecure = true
  55. blocked = false
  56. location = "k8s-grc-mirrors-have-proxy_ip_1:port"
  58. [[registry.mirror]]
  59. location = "k8s-grc-mirrors-have-proxy_ip_1:port"
  60. insecure = true

代理配置修改

之前更新以后，发现 pull image 时候也会卡住，由于 harbor 使用ceph 对象网关进行镜像存储，crio镜像时候harbor返回 对象网关地址，在对象网关机器发现，通过代理地址下载连接对象网关。如果使用代理需要修改代理配置

  1. Environment="HTTP_PROXY=http://squit_proxy_ip:port" "HTTPS_PROXY=http://squit_proxy_ip:port" "NO_PROXY=localhost,127.0.0.1,docker.yourcomany.loca,gcr.io,quay.io,k8s.gcr.io,内网网段/24"

更新版本

编辑仓库/etc/yum.repo.d/

  1. [devel_kubic_libcontainers_stable_cri-o_1.23]
  2. baseurl = http://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:/cri-o:/1.23/CentOS_7/
  3. gpgcheck = 1
  4. gpgkey = http://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:/cri-o:/1.23/CentOS_7/repodata/repomd.xml.key
  5. name = CRI-O 1.23 (CentOS_$releasever)

更新crio

  1. $ sudo yum install 
  2. $ sudo yum install cri-o-1.23.2-5.1.el7
  4. $ sudo systemctl restark crio
  5. $ sudo systemctl restart kubelet