CustomResourceDefinition(CRD)是 v1.7 新增的无需改变代码就可以扩展 Kubernetes API 的机制,用来管理自定义对象。它实际上是 ThirdPartyResources(TPR)的升级版本,而 TPR 已经在 v1.8 中弃用。

API 版本对照表

Kubernetes 版本 CRD API 版本
v1.8+ apiextensions.k8s.io/v1beta1

CRD 示例

下面的例子会创建一个 /apis/stable.example.com/v1/namespaces/<namespace>/crontabs/… 的自定义 API:

  1. apiVersion: apiextensions.k8s.io/v1beta1
  2. kind: CustomResourceDefinition
  3. metadata:
  4. # name must match the spec fields below, and be in the form: <plural>.<group>
  5. name: crontabs.stable.example.com
  6. spec:
  7. # group name to use for REST API: /apis/<group>/<version>
  8. group: stable.example.com
  9. # versions to use for REST API: /apis/<group>/<version>
  10. versions:
  11. - name: v1beta1
  12. # Each version can be enabled/disabled by Served flag.
  13. served: true
  14. # One and only one version must be marked as the storage version.
  15. storage: true
  16. - name: v1
  17. served: true
  18. storage: false
  19. # either Namespaced or Cluster
  20. scope: Namespaced
  21. names:
  22. # plural name to be used in the URL: /apis/<group>/<version>/<plural>
  23. plural: crontabs
  24. # singular name to be used as an alias on the CLI and for display
  25. singular: crontab
  26. # kind is normally the CamelCased singular type. Your resource manifests use this.
  27. kind: CronTab
  28. # shortNames allow shorter string to match your resource on the CLI
  29. shortNames:
  30. - ct

API 创建好后,就可以创建具体的 CronTab 对象了

  1. $ cat my-cronjob.yaml
  2. apiVersion: "stable.example.com/v1"
  3. kind: CronTab
  4. metadata:
  5. name: my-new-cron-object
  6. spec:
  7. cronSpec: "* * * * /5"
  8. image: my-awesome-cron-image
  9. $ kubectl create -f my-crontab.yaml
  10. crontab "my-new-cron-object" created
  11. $ kubectl get crontab
  12. NAME KIND
  13. my-new-cron-object CronTab.v1.stable.example.com
  14. $ kubectl get crontab my-new-cron-object -o yaml
  15. apiVersion: stable.example.com/v1
  16. kind: CronTab
  17. metadata:
  18. creationTimestamp: 2017-07-03T19:00:56Z
  19. name: my-new-cron-object
  20. namespace: default
  21. resourceVersion: "20630"
  22. selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object
  23. uid: 5c82083e-5fbd-11e7-a204-42010a8c0002
  24. spec:
  25. cronSpec: '* * * * /5'
  26. image: my-awesome-cron-image

Finalizer

Finalizer 用于实现控制器的异步预删除钩子,可以通过 metadata.finalizers 来指定 Finalizer。

  1. apiVersion: "stable.example.com/v1"
  2. kind: CronTab
  3. metadata:
  4. finalizers:
  5. - finalizer.stable.example.com

Finalizer 指定后,客户端删除对象的操作只会设置 metadata.deletionTimestamp 而不是直接删除。这会触发正在监听 CRD 的控制器,控制器执行一些删除前的清理操作,从列表中删除自己的 finalizer,然后再重新发起一个删除操作。此时,被删除的对象才会真正删除。

Validation

v1.8 开始新增了实验性的基于 OpenAPI v3 schema 的验证(Validation)机制,可以用来提前验证用户提交的资源是否符合规范。使用该功能需要配置 kube-apiserver 的 --feature-gates=CustomResourceValidation=true

比如下面的 CRD 要求

  • spec.cronSpec 必须是匹配正则表达式的字符串
  • spec.replicas 必须是从 1 到 10 的整数
  1. apiVersion: apiextensions.k8s.io/v1beta1
  2. kind: CustomResourceDefinition
  3. metadata:
  4. name: crontabs.stable.example.com
  5. spec:
  6. group: stable.example.com
  7. version: v1
  8. scope: Namespaced
  9. names:
  10. plural: crontabs
  11. singular: crontab
  12. kind: CronTab
  13. shortNames:
  14. - ct
  15. validation:
  16. # openAPIV3Schema is the schema for validating custom objects.
  17. openAPIV3Schema:
  18. properties:
  19. spec:
  20. properties:
  21. cronSpec:
  22. type: string
  23. pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
  24. replicas:
  25. type: integer
  26. minimum: 1
  27. maximum: 10

这样,在创建下面的 CronTab 时

  1. apiVersion: "stable.example.com/v1"
  2. kind: CronTab
  3. metadata:
  4. name: my-new-cron-object
  5. spec:
  6. cronSpec: "* * * *"
  7. image: my-awesome-cron-image
  8. replicas: 15

会报验证失败的错误:

  1. The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "selfLink":"","clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
  2. validation failure list:
  3. spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
  4. spec.replicas in body should be less than or equal to 10

Subresources

v1.10 开始 CRD 还支持 /status/scale 等两个子资源(Beta),并且从 v1.11 开始默认开启。

v1.10 版本使用前需要在 kube-apiserver 开启 --feature-gates=CustomResourceSubresources=true

  1. # resourcedefinition.yaml
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5. name: crontabs.stable.example.com
  6. spec:
  7. group: stable.example.com
  8. version: v1
  9. scope: Namespaced
  10. names:
  11. plural: crontabs
  12. singular: crontab
  13. kind: CronTab
  14. shortNames:
  15. - ct
  16. # subresources describes the subresources for custom resources.
  17. subresources:
  18. # status enables the status subresource.
  19. status: {}
  20. # scale enables the scale subresource.
  21. scale:
  22. # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
  23. specReplicasPath: .spec.replicas
  24. # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
  25. statusReplicasPath: .status.replicas
  26. # labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
  27. labelSelectorPath: .status.labelSelector
  1. $ kubectl create -f resourcedefinition.yaml
  2. $ kubectl create -f- <<EOF
  3. apiVersion: "stable.example.com/v1"
  4. kind: CronTab
  5. metadata:
  6. name: my-new-cron-object
  7. spec:
  8. cronSpec: "* * * * */5"
  9. image: my-awesome-cron-image
  10. replicas: 3
  11. EOF
  12. $ kubectl scale --replicas=5 crontabs/my-new-cron-object
  13. crontabs "my-new-cron-object" scaled
  14. $ kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}'
  15. 5

Categories

Categories 用来将 CRD 对象分组,这样就可以使用 kubectl get <category-name> 来查询属于该组的所有对象。

  1. # resourcedefinition.yaml
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5. name: crontabs.stable.example.com
  6. spec:
  7. group: stable.example.com
  8. version: v1
  9. scope: Namespaced
  10. names:
  11. plural: crontabs
  12. singular: crontab
  13. kind: CronTab
  14. shortNames:
  15. - ct
  16. # categories is a list of grouped resources the custom resource belongs to.
  17. categories:
  18. - all
  1. # my-crontab.yaml
  2. apiVersion: "stable.example.com/v1"
  3. kind: CronTab
  4. metadata:
  5. name: my-new-cron-object
  6. spec:
  7. cronSpec: "* * * * */5"
  8. image: my-awesome-cron-image
  1. $ kubectl create -f resourcedefinition.yaml
  2. $ kubectl create -f my-crontab.yaml
  3. $ kubectl get all
  4. NAME AGE
  5. crontabs/my-new-cron-object 3s

CRD 控制器

在使用 CRD 扩展 Kubernetes API 时,通常还需要实现一个新建资源的控制器,监听新资源的变化情况,并作进一步的处理。

https://github.com/kubernetes/sample-controller 提供了一个 CRD 控制器的示例,包括

  • 如何注册资源 Foo
  • 如何创建、删除和查询 Foo 对象
  • 如何监听 Foo 资源对象的变化情况

Kubebuilder

从上面的实例中可以看到从头构建一个 CRD 控制器并不容易,需要对 Kubernetes 的 API 有深入了解,并且RBAC 集成、镜像构建、持续集成和部署等都需要很大工作量。

kubebuilder 正是为解决这个问题而生,为 CRD 控制器提供了一个简单易用的框架,并可直接生成镜像构建、持续集成、持续部署等所需的资源文件。

安装

  1. # Install kubebuilder
  2. VERSION=1.0.1
  3. wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v${VERSION}/kubebuilder_${VERSION}_linux_amd64.tar.gz
  4. tar zxvf kubebuilder_${VERSION}_linux_amd64.tar.gz
  5. sudo mv kubebuilder_${VERSION}_linux_amd64 /usr/local/kubebuilder
  6. export PATH=$PATH:/usr/local/kubebuilder/bin
  7. # Install dep kustomize
  8. go get -u github.com/golang/dep/cmd/dep
  9. go get github.com/kubernetes-sigs/kustomize

使用方法

初始化项目

  1. mkdir -p $GOPATH/src/demo
  2. cd $GOPATH/src/demo
  3. kubebuilder init --domain k8s.io --license apache2 --owner "The Kubernetes Authors"

创建 API

  1. kubebuilder create api --group ships --version v1beta1 --kind Sloop

然后按照实际需要修改 pkg/apis/ship/v1beta1/sloop_types.gopkg/controller/sloop/sloop_controller.go 增加业务逻辑。

本地运行测试

  1. make install
  2. make run

如果碰到错误 ValidationError(CustomResourceDefinition.status): missing required field "storedVersions" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinitionStatus],可以手动修改 config/crds/ships_v1beta1_sloop.yaml:

```yaml status: acceptedNames: kind: “” plural: “” conditions: [] storedVersions: []

然后运行 kubectl apply -f config/crds 创建 CRD。

  1. 然后就可以用 `ships.k8s.io/v1beta1` 来创建 Kind `Sloop` 的资源了,比如
  2. ```sh
  3. kubectl apply -f config/samples/ships_v1beta1_sloop.yaml

构建镜像并部署控制器

  1. # 替换 IMG 为你自己的
  2. export IMG=feisky/demo-crd:v1
  3. make docker-build
  4. make docker-push
  5. make deploy

kustomize 已经不再支持通配符,因而上述 make deploy 可能会碰到 Load from path ../rbac/*.yaml failed 错误,解决方法是手动修改 config/default/kustomization.yaml:

resources:

  • ../rbac/rbac_role.yaml
  • ../rbac/rbac_role_binding.yaml
  • ../manager/manager.yaml

然后执行 kustomize build config/default | kubectl apply -f - 部署,默认部署到 demo-system namespace 中。

文档和测试

  1. # run unit tests
  2. make test
  3. # generate docs
  4. kubebuilder docs

参考文档