Extend the Kubernetes API with CustomResourceDefinitions

This page shows how to install a custom resource into the Kubernetes API by creating a CustomResourceDefinition.

本页展示如何通过创建CRD来向k8s中安装定制资源

Before you begin开始之前

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

• Katacoda
• Play with Kubernetes

Your Kubernetes server must be at or later than version 1.16. To check the version, enter kubectl version. If you are using an older version of Kubernetes that is still supported, switch to the documentation for that version to see advice that is relevant for your cluster.

Create a CustomResourceDefinition 创建CRD

When you create a new CustomResourceDefinition (CRD), the Kubernetes API Server creates a new RESTful resource path for each version you specify. The CRD can be either namespaced or cluster-scoped, as specified in the CRD’s scope field. As with existing built-in objects, deleting a namespace deletes all custom objects in that namespace. CustomResourceDefinitions themselves are non-namespaced and are available to all namespaces.

For example, if you save the following CustomResourceDefinition to resourcedefinition.yaml:

当你创建一个新的CRD时，k8s API服务器会为你定义的每一个版本创建一个新的restful路径。CRD作用域可以是命名空间或集群，通过CRD中的scope字段进行设定。和其他现有的内置对象相同，删除一个命名空间，会删除该命名空间下所有的定制对象。CRD本身是不受命名空间限制的，对所有命名空间可见。

例如，例如存储一下yaml文件 ：

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: crontabs.stable.example.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: stable.example.com
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: crontabs
    # singular name to be used as an alias on the CLI and for display
    singular: crontab
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: CronTab
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - ct

and create it:

kubectl apply -f resourcedefinition.yaml

Then a new namespaced RESTful API endpoint is created at:

/apis/stable.example.com/v1/namespaces/*/crontabs/...

This endpoint URL can then be used to create and manage custom objects. The kind of these objects will be CronTab from the spec of the CustomResourceDefinition object you created above.

之后此端点的URL可被用于创建和管理定制对象。这项对象的kind字段必须和上文创建的CRD对象中的一致。

It might take a few seconds for the endpoint to be created. You can watch the Established condition of your CustomResourceDefinition to be true or watch the discovery information of the API server for your resource to show up.

创建端点的过程可能会需要几秒钟，可以看到CRD的状态变为Established

Create custom objects创建定制对象

After the CustomResourceDefinition object has been created, you can create custom objects. Custom objects can contain custom fields. These fields can contain arbitrary JSON. In the following example, the cronSpec and image custom fields are set in a custom object of kind CronTab. The kind CronTab comes from the spec of the CustomResourceDefinition object you created above.

If you save the following YAML to my-crontab.yaml:

创建CRD对象创建之后，可以创建定制对象。定制对象可以包含多个自定义字段

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image

and create it:

kubectl apply -f my-crontab.yaml

You can then manage your CronTab objects using kubectl. For example:

Should print a list like this:

NAME                 AGE
my-new-cron-object   6s

Resource names are not case-sensitive when using kubectl, and you can use either the singular or plural forms defined in the CRD, as well as any short names.

使用kubectl时，资源名称不是大小写敏感的，并且使用以定义的CRD的单数、复数，或者简写都是可以的。

You can also view the raw YAML data:

You should see that it contains the custom cronSpec and image fields from the YAML you used to create it:

apiVersion: v1
kind: List
items:
- apiVersion: stable.example.com/v1
  kind: CronTab
  metadata:
    creationTimestamp: 2017-05-31T12:56:35Z
    generation: 1
    name: my-new-cron-object
    namespace: default
    resourceVersion: "285"
    uid: 9423255b-4600-11e7-af6a-28d2447dc82b
  spec:
    cronSpec: '* * * * */5'
    image: my-awesome-cron-image
metadata:
  resourceVersion: ""

Delete a CustomResourceDefinition删除CRD

When you delete a CustomResourceDefinition, the server will uninstall the RESTful API endpoint and delete all custom objects stored in it.

当删除一个CRD资源时，服务器会自动卸载resutfult api端点并删除里面存储的所有定制对象。

kubectl delete -f resourcedefinition.yaml
kubectl get crontabs

Error from server (NotFound): Unable to list {"stable.example.com" "v1" "crontabs"}: the server could not find the requested resource (get crontabs.stable.example.com)

If you later recreate the same CustomResourceDefinition, it will start out empty.

对于之后创建的相同的CRD,启动时其内部会是空的。

Specifying a structural schema定义结构化的模式

CustomResources store structured data in custom fields (alongside the built-in fields apiVersion, kind and metadata, which the API server validates implicitly). With OpenAPI v3.0 validation a schema can be specified, which is validated during creation and updates, compare below for details and limits of such a schema.

定制资源使用定制字段存储结构化数据，（这些数据和内置字段，例如apiVersion，kind，以及metadata一起存储`，API服务器会隐式的进行有效性校验)。

可以使用OpenAPI v3校验规则来定义一个模式，该模式的合法性会在创建和更新的时候进行检查。参阅下文来了解这类模式的细节以及限制。

With apiextensions.k8s.io/v1 the definition of a structural schema is mandatory for CustomResourceDefinitions. In the beta version of CustomResourceDefinition, the structural schema was optional.

在apiextensions.k8s.io/v1 API下，CRD必须要有结构模式的定义。在beate版本中，结构化模式是可选的。

A structural schema is an OpenAPI v3.0 validation schema which:

1. specifies a non-empty type (via type in OpenAPI) for the root, for each specified field of an object node (via properties or additionalProperties in OpenAPI) and for each item in an array node (via items in OpenAPI), with the exception of:

   • a node with x-kubernetes-int-or-string: true
   • a node with x-kubernetes-preserve-unknown-fields: true
2. for each field in an object and each item in an array which is specified within any of allOf, anyOf, oneOf or not, the schema also specifies the field/item outside of those logical junctors (compare example 1 and 2).
3. does not set description, type, default, additionalProperties, nullable within an allOf, anyOf, oneOf or not, with the exception of the two pattern for x-kubernetes-int-or-string: true (see below).
4. if metadata is specified, then only restrictions on metadata.name and metadata.generateName are allowed.

结构化模式是一个OpenAPI v3.0 有效性检验模式 ，包含

1. 为根字段定义一个非空的type值(通过OpenAPI中的type字段定义)。对于每一个定义的对象结点(通过 OpenAPI中的proterties 或 additionalProperties)以及数组结点中的每个条目(通过OpenAPI中的items字段)，也需要设置一个非空的type值。以下情况例外：

   • 结点属性包括x-kubernetes-int-or-string: true
   • 结点属性包括x-kuberbetes-preserve-unknown-files: true

2. 对于对象中的每个字段，以及数组中的每个条目，如果定义中包含了allOf,anyOf,oneOf,not，则模式中也要指定这些逻辑组合之外的字段或条目，（参照样例1和样例2）。

3. 在allOf,anyOf,oneOf,not中，不设置description,type,default,additionalProperties,nullable字段，但当属性中包含了x-kubernetes-int-or-string: true时除外
4. 如果定义了metadata字段，则只允许对metadata.name和metadata.generateName字段设置约束

Non-structural example 1:

allOf:
- properties:
    foo:
      ...

conflicts with rule 2. The following would be correct:

properties:
  foo:
    ...
allOf:
- properties:
    foo:
      ...

Non-structural example 2:

allOf:
- items:
    properties:
      foo:
        ...

conflicts with rule 2. The following would be correct:

items:
  properties:
    foo:
      ...
allOf:
- items:
    properties:
      foo:
        ...

Non-structural example 3:

properties:
  foo:
    pattern: "abc"
  metadata:
    type: object
    properties:
      name:
        type: string
        pattern: "^a"
      finalizers:
        type: array
        items:
          type: string
          pattern: "my-finalizer"
anyOf:
- properties:
    bar:
      type: integer
      minimum: 42
  required: ["bar"]
  description: "foo bar object"

is not a structural schema because of the following violations:

• the type at the root is missing (rule 1).
• the type of foo is missing (rule 1).
• bar inside of anyOf is not specified outside (rule 2).
• bar‘s type is within anyOf (rule 3).
• the description is set within anyOf (rule 3).
• metadata.finalizers might not be restricted (rule 4).

In contrast, the following, corresponding schema is structural:

type: object
description: "foo bar object"
properties:
  foo:
    type: string
    pattern: "abc"
  bar:
    type: integer
  metadata:
    type: object
    properties:
      name:
        type: string
        pattern: "^a"
anyOf:
- properties:
    bar:
      minimum: 42
  required: ["bar"]

Violations of the structural schema rules are reported in the NonStructural condition in the CustomResourceDefinition.

Field pruning

CustomResourceDefinitions store validated resource data in the cluster’s persistence store, etcd. As with native Kubernetes resources such as ConfigMap, if you specify a field that the API server does not recognize, the unknown field is pruned (removed) before being persisted.

CRD在集群持久化存储etcd钟存储验证有效的资源数据。和k8s原始的资源（例如ConfigMap)一样，如果用户定义了一个服务器无法识别的字段，则该字段会在持久化之前被精简掉(移除)。

Note:

CRDs converted from apiextensions.k8s.io/v1beta1 to apiextensions.k8s.io/v1 might lack structural schemas, and spec.preserveUnknownFields might be true.

CRD从v1beta1转换到v1室可能缺少结构化模式,spec.preserveUnknownFields字段可能会被置为true

For legacy CustomResourceDefinition objects created as apiextensions.k8s.io/v1beta1 with spec.preserveUnknownFields set to true, the following is also true:

• Pruning is not enabled.
• You can store arbitrary data.

对于 使用v1beta1，将preserveUnknownFields字段设置为true的旧的CRD对象来说，有以下表现：

• 裁剪未启用
• 可以存储任意数据

For compatibility with apiextensions.k8s.io/v1, update your custom resource definitions to:

1. Use a structural OpenAPI schema.
2. Set spec.preserveUnknownFields to false.

为了与v1版本兼容，将你的定制资源定义为：

1. 使用结构化OpenAPI模式
2. spec.preserveUnknowFields设置为false

If you save the following YAML to my-crontab.yaml:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
  someRandomField: 42

and create it:

kubectl create --validate=false -f my-crontab.yaml -o yaml

your output is similar to:

apiVersion: stable.example.com/v1
kind: CronTab
metadata:
  creationTimestamp: 2017-05-31T12:56:35Z
  generation: 1
  name: my-new-cron-object
  namespace: default
  resourceVersion: "285"
  uid: 9423255b-4600-11e7-af6a-28d2447dc82b
spec:
  cronSpec: '* * * * */5'
  image: my-awesome-cron-image

Notice that the field someRandomField was pruned.

This example turned off client-side validation to demonstrate the API server’s behavior, by adding the --validate=false command line option. Because the OpenAPI validation schemas are also published to clients, kubectl also checks for unknown fields and rejects those objects well before they would be sent to the API server.

Controlling pruning控制裁剪

By default, all unspecified fields for a custom resource, across all versions, are pruned. It is possible though to opt-out of that for specifc sub-trees of fields by adding x-kubernetes-preserve-unknown-fields: true in the structural OpenAPI v3 validation schema. For example:

type: object
properties:
  json:
    x-kubernetes-preserve-unknown-fields: true

The field json can store any JSON value, without anything being pruned.

You can also partially specify the permitted JSON; for example:

type: object
properties:
  json:
    x-kubernetes-preserve-unknown-fields: true
    type: object
    description: this is arbitrary JSON

With this, only object type values are allowed.

Pruning is enabled again for each specified property (or additionalProperties):

type: object
properties:
  json:
    x-kubernetes-preserve-unknown-fields: true
    type: object
    properties:
      spec:
        type: object
        properties:
          foo:
            type: string
          bar:
            type: string

With this, the value:

json:
  spec:
    foo: abc
    bar: def
    something: x
  status:
    something: x

is pruned to:

json:
  spec:
    foo: abc
    bar: def
  status:
    something: x

This means that the something field in the specified spec object is pruned, but everything outside is not.

IntOrString

Nodes in a schema with x-kubernetes-int-or-string: true are excluded from rule 1, such that the following is structural:

type: object
properties:
  foo:
    x-kubernetes-int-or-string: true

Also those nodes are partially excluded from rule 3 in the sense that the following two patterns are allowed (exactly those, without variations in order to additional fields):

x-kubernetes-int-or-string: true
anyOf:
- type: integer
- type: string
...

and

x-kubernetes-int-or-string: true
allOf:
- anyOf:
  - type: integer
  - type: string
- ... # zero or more
...

With one of those specification, both an integer and a string validate.

In Validation Schema Publishing, x-kubernetes-int-or-string: true is unfolded to one of the two patterns shown above.

RawExtension

RawExtensions (as in runtime.RawExtension defined in k8s.io/apimachinery) holds complete Kubernetes objects, i.e. with apiVersion and kind fields.

It is possible to specify those embedded objects (both completely without constraints or partially specified) by setting x-kubernetes-embedded-resource: true. For example:

type: object
properties:
  foo:
    x-kubernetes-embedded-resource: true
    x-kubernetes-preserve-unknown-fields: true

Here, the field foo holds a complete object, e.g.:

foo:
  apiVersion: v1
  kind: Pod
  spec:
    ...

Because x-kubernetes-preserve-unknown-fields: true is specified alongside, nothing is pruned. The use of x-kubernetes-preserve-unknown-fields: true is optional though.

With x-kubernetes-embedded-resource: true, the apiVersion, kind and metadata are implicitly specified and validated.

Serving multiple versions of a CRD提供CRD的多个版本

See Custom resource definition versioning for more information about serving multiple versions of your CustomResourceDefinition and migrating your objects from one version to another.

关于如何为你的 CustomResourceDefinition 提供多个版本的支持，以及如何将你的对象 从一个版本迁移到另一个版本， 详细信息可参阅 定制资源定义的版本。

Advanced topics高级主题

Finalizers

Finalizers allow controllers to implement asynchronous pre-delete hooks. Custom objects support finalizers similar to built-in objects.

You can add a finalizer to a custom object like this:

Finalizers 允许控制器在删除前实现异步的回调。和内建对象一样，定制对象也支持finalizer.

以下是添加finalizer的样例

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  finalizers:
  - stable.example.com/finalizer

Identifiers of custom finalizers consist of a domain name, a forward slash and the name of the finalizer. Any controller can add a finalizer to any object’s list of finalizers.

定义Finalizer包含一个域名，一/和finalizer的名字。任何控制器均可以任意finalizers的对象列表上添加一个finalizer.

The first delete request on an object with finalizers sets a value for the metadata.deletionTimestamp field but does not delete it. Once this value is set, entries in the finalizers list can only be removed. While any finalizers remain it is also impossible to force the deletion of an object.

第一次请求删除带有finalizer的对象时，会在metadata,deletionTimestap字段设置一个值，不会真的删除它。在值设定之后，finalizers列表的表项只能被移除。若任意finalizer残留，则无法强制删除该对象。

When the metadata.deletionTimestamp field is set, controllers watching the object execute any finalizers they handle and remove the finalizer from the list after they are done. It is the responsibility of each controller to remove its finalizer from the list.

当metadata.deletionTimestamp字段被设定时，监控对象的控制器会执行他们所能处理的所有finalizer，并在处理完毕后将其从列表中删除。由控制器负责将finalizer从列表中删除。

The value of metadata.deletionGracePeriodSeconds controls the interval between polling updates.

Once the list of finalizers is empty, meaning all finalizers have been executed, the resource is deleted by Kubernetes.

metadata.deletionGracePeriodseconds控制滚动更新的间隔

一旦finalizer列表为空 ，意味着所有的finalizer都被执行完毕，资源被k8s删除了。

Validation合法性检查

Custom resources are validated via OpenAPI v3 schemas and you can add additional validation using admission webhooks.

Additionally, the following restrictions are applied to the schema:

定制资源通过OpenAPI v3模式进行有效性检查，也可以使用准入控制 webhook来添加额外的合法性检查逻辑

此外，对模式的定义存在以下限制

• These fields cannot be set:

  • definitions,
  • dependencies,
  • deprecated,
  • discriminator,
  • id,
  • patternProperties,
  • readOnly,
  • writeOnly,
  • xml,
  • $ref.
• The field uniqueItems cannot be set to true.
• The field additionalProperties cannot be set to false.
• The field additionalProperties is mutually exclusive with properties.

The default field can be set when the Defaulting feature is enabled, which is the case with apiextensions.k8s.io/v1 CustomResourceDefinitions. Defaulting is in GA since 1.17 (beta since 1.16 with the CustomResourceDefaulting feature gate enabled, which is the case automatically for many clusters for beta features).

Refer to the structural schemas section for other restrictions and CustomResourceDefinition features.

The schema is defined in the CustomResourceDefinition. In the following example, the CustomResourceDefinition applies the following validations on the custom object:

• spec.cronSpec must be a string and must be of the form described by the regular expression.
• spec.replicas must be an integer and must have a minimum value of 1 and a maximum value of 10.

Save the CustomResourceDefinition to resourcedefinition.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        # openAPIV3Schema is the schema for validating custom objects.
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                  pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
                image:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 10
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct

and create it:

kubectl apply -f resourcedefinition.yaml

A request to create a custom object of kind CronTab is rejected if there are invalid values in its fields. In the following example, the custom object contains fields with invalid values:

• spec.cronSpec does not match the regular expression.
• spec.replicas is greater than 10.

If you save the following YAML to my-crontab.yaml:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * *"
  image: my-awesome-cron-image
  replicas: 15

and attempt to create it:

kubectl apply -f my-crontab.yaml

then you get an error:

The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
validation failure list:
spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
spec.replicas in body should be less than or equal to 10

If the fields contain valid values, the object creation request is accepted.

Save the following YAML to my-crontab.yaml:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
  replicas: 5

And create it:

kubectl apply -f my-crontab.yaml
crontab "my-new-cron-object" created

Defaulting

Note: To use defaulting, your CustomResourceDefinition must use API version apiextensions.k8s.io/v1.

Defaulting allows to specify default values in the OpenAPI v3 validation schema:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        # openAPIV3Schema is the schema for validating custom objects.
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                  pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
                  default: "5 0 * * *"
                image:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 10
                  default: 1
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct

With this both cronSpec and replicas are defaulted:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  image: my-awesome-cron-image

leads to

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "5 0 * * *"
  image: my-awesome-cron-image
  replicas: 1

Defaulting happens on the object

• in the request to the API server using the request version defaults,
• when reading from etcd using the storage version defaults,
• after mutating admission plugins with non-empty patches using the admission webhook object version defaults.

Defaults applied when reading data from etcd are not automatically written back to etcd. An update request via the API is required to persist those defaults back into etcd.

Default values must be pruned (with the exception of defaults for metadata fields) and must validate against a provided schema.

Default values for metadata fields of x-kubernetes-embedded-resources: true nodes (or parts of a default value covering metadata) are not pruned during CustomResourceDefinition creation, but through the pruning step during handling of requests.

Defaulting and Nullable

New in 1.20: null values for fields that either don’t specify the nullable flag, or give it a false value, will be pruned before defaulting happens. If a default is present, it will be applied. When nullable is true, null values will be conserved and won’t be defaulted.

For example, given the OpenAPI schema below:

type: object
properties:
  spec:
    type: object
    properties:
      foo:
        type: string
        nullable: false
        default: "default"
      bar:
        type: string
        nullable: true
      baz:
        type: string

creating an object with null values for foo and bar and baz

spec:
  foo: null
  bar: null
  baz: null

leads to

spec:
  foo: "default"
  bar: null

with foo pruned and defaulted because the field is non-nullable, bar maintaining the null value due to nullable: true, and baz pruned because the field is non-nullable and has no default.

Publish Validation Schema in OpenAPI v2

CustomResourceDefinition OpenAPI v3 validation schemas which are structural and enable pruning are published as part of the OpenAPI v2 spec from Kubernetes API server.

The kubectl command-line tool consumes the published schema to perform client-side validation (kubectl create and kubectl apply), schema explanation (kubectl explain) on custom resources. The published schema can be consumed for other purposes as well, like client generation or documentation.

The OpenAPI v3 validation schema is converted to OpenAPI v2 schema, and show up in definitions and paths fields in the OpenAPI v2 spec.

The following modifications are applied during the conversion to keep backwards compatibility with kubectl in previous 1.13 version. These modifications prevent kubectl from being over-strict and rejecting valid OpenAPI schemas that it doesn’t understand. The conversion won’t modify the validation schema defined in CRD, and therefore won’t affect validation in the API server.

1. The following fields are removed as they aren’t supported by OpenAPI v2 (in future versions OpenAPI v3 will be used without these restrictions)

   • The fields allOf, anyOf, oneOf and not are removed
2. If nullable: true is set, we drop type, nullable, items and properties because OpenAPI v2 is not able to express nullable. To avoid kubectl to reject good objects, this is necessary.

Additional printer columns额外打印列

The kubectl tool relies on server-side output formatting. Your cluster’s API server decides which columns are shown by the kubectl get command. You can customize these columns for a CustomResourceDefinition. The following example adds the Spec, Replicas, and Age columns.

Save the CustomResourceDefinition to resourcedefinition.yaml:

kubectl工具依赖于服务端输出格式。API服务器决定了kubectl get命令会得到哪些返回结果，可以通过CRD来定制输出格式。下例添加了Spec，Replicas以及Age列。

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              cronSpec:
                type: string
              image:
                type: string
              replicas:
                type: integer
    additionalPrinterColumns:
    - name: Spec
      type: string
      description: The cron spec defining the interval a CronJob is run
      jsonPath: .spec.cronSpec
    - name: Replicas
      type: integer
      description: The number of jobs launched by the CronJob
      jsonPath: .spec.replicas
    - name: Age
      type: date
      jsonPath: .metadata.creationTimestamp

Create the CustomResourceDefinition:

kubectl apply -f resourcedefinition.yaml

Create an instance using the my-crontab.yaml from the previous section.

Invoke the server-side printing:

kubectl get crontab my-new-cron-object

Notice the NAME, SPEC, REPLICAS, and AGE columns in the output:

NAME                 SPEC        REPLICAS   AGE
my-new-cron-object   * * * * *   1          7s

Note: The NAME column is implicit and does not need to be defined in the CustomResourceDefinition.

Priority

Each column includes a priority field. Currently, the priority differentiates between columns shown in standard view or wide view (using the -o wide flag).

• Columns with priority 0 are shown in standard view.
• Columns with priority greater than 0 are shown only in wide view.

Type

A column’s type field can be any of the following (compare OpenAPI v3 data types):

• integer – non-floating-point numbers
• number – floating point numbers
• string – strings
• boolean – true or false
• date – rendered differentially as time since this timestamp.

If the value inside a CustomResource does not match the type specified for the column, the value is omitted. Use CustomResource validation to ensure that the value types are correct.

Format

A column’s format field can be any of the following:

• int32
• int64
• float
• double
• byte
• date
• date-time
• password

The column’s format controls the style used when kubectl prints the value.

Subresources

Custom resources support /status and /scale subresources.

The status and scale subresources can be optionally enabled by defining them in the CustomResourceDefinition.

定制自然支持/status以及/scale子资源

可以通过CRD来定义来使用这两种子资源。

Status subresource

When the status subresource is enabled, the /status subresource for the custom resource is exposed.

当启用了status子资源时，对应定制资源的/status子资源会被暴露出来。

• The status and the spec stanzas are represented by the .status and .spec JSONPaths respectively inside of a custom resource.

对status而spec内容分别用定制资源的.status以及.specJSON路径来进行表示

• PUT requests to the /status subresource take a custom resource object and ignore changes to anything except the status stanza.

对status子自然的PUT请求要求使用定制资源对象作为其输入，但会忽略status之外的所有内容。

• PUT requests to the /status subresource only validate the status stanza of the custom resource.

对/status子资源的PUT请求，仅对定制资源的status内容进行合法性检查

• PUT/POST/PATCH requests to the custom resource ignore changes to the status stanza.

对定制资源的PUT，post，PATCH请求会忽略status内容的变化

• The .metadata.generation value is incremented for all changes, except for changes to .metadata or .status.

• Only the following constructs are allowed at the root of the CRD OpenAPI validation schema:

  • description
  • example
  • exclusiveMaximum
  • exclusiveMinimum
  • externalDocs
  • format
  • items
  • maximum
  • maxItems
  • maxLength
  • minimum
  • minItems
  • minLength
  • multipleOf
  • pattern
  • properties
  • required
  • title
  • type
  • uniqueItems

Scale subresource

When the scale subresource is enabled, the /scale subresource for the custom resource is exposed. The autoscaling/v1.Scale object is sent as the payload for /scale.

To enable the scale subresource, the following fields are defined in the CustomResourceDefinition.

• specReplicasPath defines the JSONPath inside of a custom resource that corresponds to scale.spec.replicas.

  • It is a required value.
  • Only JSONPaths under .spec and with the dot notation are allowed.
  • If there is no value under the specReplicasPath in the custom resource, the /scale subresource will return an error on GET.

• statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to scale.status.replicas.

  • It is a required value.
  • Only JSONPaths under .status and with the dot notation are allowed.
  • If there is no value under the statusReplicasPath in the custom resource, the status replica value in the /scale subresource will default to 0.

• labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.

  • It is an optional value.
  • It must be set to work with HPA.
  • Only JSONPaths under .status or .spec and with the dot notation are allowed.
  • If there is no value under the labelSelectorPath in the custom resource, the status selector value in the /scale subresource will default to the empty string.
  • The field pointed by this JSON path must be a string field (not a complex selector struct) which contains a serialized label selector in string form.

In the following example, both status and scale subresources are enabled.

Save the CustomResourceDefinition to resourcedefinition.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
            status:
              type: object
              properties:
                replicas:
                  type: integer
                labelSelector:
                  type: string
      # subresources describes the subresources for custom resources.
      subresources:
        # status enables the status subresource.
        status: {}
        # scale enables the scale subresource.
        scale:
          # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
          specReplicasPath: .spec.replicas
          # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
          statusReplicasPath: .status.replicas
          # labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
          labelSelectorPath: .status.labelSelector
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct

And create it:

kubectl apply -f resourcedefinition.yaml

After the CustomResourceDefinition object has been created, you can create custom objects.

If you save the following YAML to my-crontab.yaml:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
  replicas: 3

and create it:

kubectl apply -f my-crontab.yaml

Then new namespaced RESTful API endpoints are created at:

/apis/stable.example.com/v1/namespaces/*/crontabs/status

and

/apis/stable.example.com/v1/namespaces/*/crontabs/scale

A custom resource can be scaled using the kubectl scale command. For example, the following command sets .spec.replicas of the custom resource created above to 5:

kubectl scale --replicas=5 crontabs/my-new-cron-object
crontabs "my-new-cron-object" scaled

kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}'
5

You can use a PodDisruptionBudget to protect custom resources that have the scale subresource enabled.

Categories

Categories is a list of grouped resources the custom resource belongs to (eg. all). You can use kubectl get <category-name> to list the resources belonging to the category.

The following example adds all in the list of categories in the CustomResourceDefinition and illustrates how to output the custom resource using kubectl get all.

Save the following CustomResourceDefinition to resourcedefinition.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct
    # categories is a list of grouped resources the custom resource belongs to.
    categories:
    - all

and create it:

kubectl apply -f resourcedefinition.yaml

After the CustomResourceDefinition object has been created, you can create custom objects.

Save the following YAML to my-crontab.yaml:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image

and create it:

kubectl apply -f my-crontab.yaml

You can specify the category when using kubectl get:

kubectl get all

and it will include the custom resources of kind CronTab:

NAME                          AGE
crontabs/my-new-cron-object   3s

What’s next

• Read about custom resources.

• See CustomResourceDefinition.

• Serve multiple versions of a CustomResourceDefinition.

Last modified February 11, 2021 at 3:51 PM PST: clean up use of word: just (3ff5ec1ef)