实现API接口服务
代码生成
虽然官方给了一个sample-apiserver，也可以在kubernetes源码中找到。我们可以照着实现自己的Aggregated APIServer。但完全手工编写还是太费劲了，这里使用官方推荐的工具apiserver-builder-alpha帮助快速创建项目骨架。
apiserver-builder构建AA方案的API接口服务的原理还是比较清晰的，总之就是kubernetes里最常见的控制器模式，这里就不具体介绍了，官方文档既有文字又有图片讲得还是挺细致的，强烈推荐大家多看看，学习一下。
apiserver-builder的安装可以直接从github上下载release版本
下载后解压出来的二进制文件，需要放到/usr/bin/或者/usr/local/bin/或者$GOPATH/bin目录下，以便于执行。
以下用到的：
apiserver-builder版本：v1.16.alpha.0 kubernetes版本：v1.16.3
以下参考apiserver-builder的官方文档，以及网上的资料，形成的较为详细的文档。
1、创建项目目录
mkdir $GOPATH/src/harmonycloud.cn/middleware-apiservercd $GOPATH/src/harmonycloud.cn/middleware-apiserver
2、初始化项目：
直接执行apiserver-boot init repo --domain hc.middleware会报错，提示需要boilerplate.go.txt文件，该文件就是个版权和文件头。
# apiserver-boot init repo --domain hc.middleware
F1221 22:50:48.900240 2923 util.go:81] Must create boilerplate.go.txt file with copyright and file header
在项目目录下新建一个名为boilerplate.go.txt，里面是代码的头部版权声明。boilerplate.go.txt中的内容：
/*
Copyright YEAR The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
将以上内容保存在待生成代码的项目根目录下，重新执行，会生成以下文件。
[root@liabio middleware-apiserver]# apiserver-boot init repo --domain hc.middleware
I1221 22:52:59.513834 4325 repo.go:81] installing vendor/ directory. To disable this, run with --install-deps=false.
[root@liabio middleware-apiserver]# ls
bin boilerplate.go.txt BUILD.bazel cmd Gopkg.lock Gopkg.toml pkg PROJECT vendor WORKSPACE
2、创建一个集群（非命名空间）范围的api-resource：
--group要符合正则表达式：^[a-z]+$--version要符合正则表达式：^v\d+(alpha\d+|beta\d+)*
[root@liabio middleware-apiserver]# apiserver-boot create group version resource --group middleware --version v1alpha1 --non-namespaced=true --kind RemoveNode
Create Resource [y/n]
y
Create Controller [y/n]
y
Create Admission Controller [y/n]
y
[root@liabio middleware-apiserver]#
稍等片刻，会生成以下文件：
[root@liabio middleware-apiserver]# ls
bin boilerplate.go.txt BUILD.bazel cmd docs Gopkg.lock Gopkg.toml pkg plugin PROJECT sample vendor WORKSPACE
3、创建RemoveNode这个api-resource的子资源：--subresource要符合正则表达式：^[a-z]+$
[root@liabio middleware-apiserver]# apiserver-boot create subresource --subresource fixnode --group middleware --version v1alpha1 --kind RemoveNode
[root@liabio middleware-apiserver]#
4、生成上述创建的api-resource类型的相关代码，包括deepcopy接口实现代码、versioned/unversioned类型转换代码、api-resource类型注册代码、api-resource类型的Controller代码、api-resource类型的AdmissionController代码等，详细看下面生成过程。
[root@liabio middleware-apiserver]# apiserver-boot build generated
I1221 23:00:40.602555 9213 generate.go:139] /data/go/bin/apiregister-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/... --input-dirs harmonycloud.cn/middleware-apiserver/pkg/controller/...
I1221 23:01:58.899356 9213 generate.go:154] /data/go/bin/conversion-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.conversion --extra-peer-dirs k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/conversion,k8s.io/apimachinery/pkg/runtime
I1221 23:03:25.487616 9213 generate.go:168] /data/go/bin/deepcopy-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.deepcopy
I1221 23:04:17.633786 9213 generate.go:218] /data/go/bin/openapi-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt -i k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/api/resource,k8s.io/apimachinery/pkg/version,k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/util/intstr,k8s.io/api/admission/v1,k8s.io/api/admission/v1beta1,k8s.io/api/admissionregistration/v1,k8s.io/api/admissionregistration/v1beta1,k8s.io/api/apps/v1,k8s.io/api/apps/v1beta1,k8s.io/api/apps/v1beta2,k8s.io/api/auditregistration/v1alpha1,k8s.io/api/authentication/v1,k8s.io/api/authentication/v1beta1,k8s.io/api/authorization/v1,k8s.io/api/authorization/v1beta1,k8s.io/api/autoscaling/v1,k8s.io/api/autoscaling/v2beta1,k8s.io/api/autoscaling/v2beta2,k8s.io/api/batch/v1,k8s.io/api/batch/v1beta1,k8s.io/api/batch/v2alpha1,k8s.io/api/certificates/v1beta1,k8s.io/api/coordination/v1,k8s.io/api/coordination/v1beta1,k8s.io/api/core/v1,k8s.io/api/discovery/v1alpha1,k8s.io/api/events/v1beta1,k8s.io/api/extensions/v1beta1,k8s.io/api/imagepolicy/v1alpha1,k8s.io/api/networking/v1,k8s.io/api/networking/v1beta1,k8s.io/api/node/v1alpha1,k8s.io/api/node/v1beta1,k8s.io/api/policy/v1beta1,k8s.io/api/rbac/v1,k8s.io/api/rbac/v1alpha1,k8s.io/api/rbac/v1beta1,k8s.io/api/scheduling/v1,k8s.io/api/scheduling/v1alpha1,k8s.io/api/scheduling/v1beta1,k8s.io/api/settings/v1alpha1,k8s.io/api/storage/v1,k8s.io/api/storage/v1alpha1,k8s.io/api/storage/v1beta1,k8s.io/client-go/pkg/apis/clientauthentication/v1alpha1,k8s.io/client-go/pkg/apis/clientauthentication/v1beta1,k8s.io/api/core/v1 --report-filename violations.report --output-package harmonycloud.cn/middleware-apiserver/pkg/openapi
I1221 23:05:15.922598 9213 generate.go:233] /data/go/bin/defaulter-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.defaults --extra-peer-dirs= k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/conversion,k8s.io/apimachinery/pkg/runtime
I1221 23:06:00.841978 9213 generate.go:252] /data/go/bin/client-gen -o /data/go/src --go-header-file boilerplate.go.txt --input-base harmonycloud.cn/middleware-apiserver/pkg/apis --input middleware/v1alpha1 --clientset-path harmonycloud.cn/middleware-apiserver/pkg/client/clientset_generated --clientset-name clientset
I1221 23:06:58.656544 9213 generate.go:282] /data/go/bin/lister-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt --output-package harmonycloud.cn/middleware-apiserver/pkg/client/listers_generated
I1221 23:07:40.084794 9213 generate.go:297] /data/go/bin/informer-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt --output-package harmonycloud.cn/middleware-apiserver/pkg/client/informers_generated --listers-package harmonycloud.cn/middleware-apiserver/pkg/client/listers_generated --versioned-clientset-package harmonycloud.cn/middleware-apiserver/pkg/client/clientset_generated/clientset
[root@liabio middleware-apiserver]#
编译二进制文件，启动后会先生成代码，可以用参数：--generate=false禁止生成代码。直接编译二进制
[root@liabio middleware-apiserver]# apiserver-boot run local
I1222 20:44:28.984617 9609 build_executables.go:140] regenerating generated code. To disable regeneration, run with --generate=false.
I1222 20:44:28.990139 9609 generate.go:139] /data/go/bin/apiregister-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/... --input-dirs harmonycloud.cn/middleware-apiserver/pkg/controller/...
I1222 20:45:24.536580 9609 generate.go:154] /data/go/bin/conversion-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.conversion --extra-peer-dirs k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/conversion,k8s.io/apimachinery/pkg/runtime
I1222 20:46:11.254261 9609 generate.go:168] /data/go/bin/deepcopy-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.deepcopy
I1222 20:46:28.187091 9609 generate.go:218] /data/go/bin/openapi-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt -i k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/api/resource,k8s.io/apimachinery/pkg/version,k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/util/intstr,k8s.io/api/admission/v1,k8s.io/api/admission/v1beta1,k8s.io/api/admissionregistration/v1,k8s.io/api/admissionregistration/v1beta1,k8s.io/api/apps/v1,k8s.io/api/apps/v1beta1,k8s.io/api/apps/v1beta2,k8s.io/api/auditregistration/v1alpha1,k8s.io/api/authentication/v1,k8s.io/api/authentication/v1beta1,k8s.io/api/authorization/v1,k8s.io/api/authorization/v1beta1,k8s.io/api/autoscaling/v1,k8s.io/api/autoscaling/v2beta1,k8s.io/api/autoscaling/v2beta2,k8s.io/api/batch/v1,k8s.io/api/batch/v1beta1,k8s.io/api/batch/v2alpha1,k8s.io/api/certificates/v1beta1,k8s.io/api/coordination/v1,k8s.io/api/coordination/v1beta1,k8s.io/api/core/v1,k8s.io/api/discovery/v1alpha1,k8s.io/api/events/v1beta1,k8s.io/api/extensions/v1beta1,k8s.io/api/imagepolicy/v1alpha1,k8s.io/api/networking/v1,k8s.io/api/networking/v1beta1,k8s.io/api/node/v1alpha1,k8s.io/api/node/v1beta1,k8s.io/api/policy/v1beta1,k8s.io/api/rbac/v1,k8s.io/api/rbac/v1alpha1,k8s.io/api/rbac/v1beta1,k8s.io/api/scheduling/v1,k8s.io/api/scheduling/v1alpha1,k8s.io/api/scheduling/v1beta1,k8s.io/api/settings/v1alpha1,k8s.io/api/storage/v1,k8s.io/api/storage/v1alpha1,k8s.io/api/storage/v1beta1,k8s.io/client-go/pkg/apis/clientauthentication/v1alpha1,k8s.io/client-go/pkg/apis/clientauthentication/v1beta1,k8s.io/api/core/v1 --report-filename violations.report --output-package harmonycloud.cn/middleware-apiserver/pkg/openapi
I1222 20:46:50.775143 9609 generate.go:233] /data/go/bin/defaulter-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.defaults --extra-peer-dirs=k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/conversion,k8s.io/apimachinery/pkg/runtime
I1222 20:47:09.165296 9609 generate.go:252] /data/go/bin/client-gen -o /data/go/src --go-header-file boilerplate.go.txt --input-base harmonycloud.cn/middleware-apiserver/pkg/apis --input middleware/v1alpha1 --clientset-path harmonycloud.cn/middleware-apiserver/pkg/client/clientset_generated --clientset-name clientset
I1222 20:48:05.917988 9609 generate.go:282] /data/go/bin/lister-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt --output-package harmonycloud.cn/middleware-apiserver/pkg/client/listers_generated
I1222 20:48:29.712595 9609 generate.go:297] /data/go/bin/informer-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt --output-package harmonycloud.cn/middleware-apiserver/pkg/client/informers_generated --listers-package harmonycloud.cn/middleware-apiserver/pkg/client/listers_generated --versioned-clientset-package harmonycloud.cn/middleware-apiserver/pkg/client/clientset_generated/clientset
I1222 20:48:41.265401 9609 build_executables.go:151] CGO_ENABLED=0
I1222 20:48:41.265495 9609 build_executables.go:161] go build -o bin/apiserver cmd/apiserver/main.go
启动报错：
[root@liabio apiserver]# ./apiserver --etcd-servers https://127.0.0.1:2379 --secure-port 1443 --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --kubeconfig=/root/.kube/config
I1222 19:38:07.788805 28678 start.go:359] loading out-of-cluster loopback client according to `--kubeconfig` settings...
W1222 19:38:08.330874 28678 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
W1222 19:38:08.330907 28678 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
W1222 19:38:08.330940 28678 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
I1222 19:38:08.332077 28678 plugins.go:158] Loaded 3 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionWebhook,RemoveNode.
I1222 19:38:08.332091 28678 plugins.go:161] Loaded 2 validating admission controller(s) successfully in the following order: ValidatingAdmissionWebhook,RemoveNode.
I1222 19:38:08.338029 28678 client.go:361] parsed scheme: "endpoint"
I1222 19:38:08.338077 28678 endpoint.go:66] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}]
I1222 19:38:08.357130 28678 client.go:361] parsed scheme: "endpoint"
I1222 19:38:08.357160 28678 endpoint.go:66] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}]
I1222 19:38:08.366985 28678 client.go:361] parsed scheme: "endpoint"
I1222 19:38:08.367007 28678 endpoint.go:66] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}]
E1222 19:38:08.465937 28678 pathrecorder.go:107] registered "/healthz/etcd" from goroutine 1 [running]:
runtime/debug.Stack(0x1a8f760, 0xc0002b8c00, 0xc000b34640)
/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).trackCallers(0xc0004b5ce0, 0xc000b34640, 0xd)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:109 +0x86
harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).Handle(0xc0004b5ce0, 0xc000b34640, 0xd, 0x3d616a0, 0xc000b195b0)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:173 +0x86
harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server/healthz.InstallPathHandler(0x3d5ec00, 0xc0004b5ce0, 0x1d14c49, 0x8, 0xc000403280, 0x6, 0x8)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:127 +0x39e
harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server/healthz.InstallHandler(...)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:93
harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server.(*GenericAPIServer).installHealthz(0xc0002b50e0)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server/healthz.go:96 +0xc6
harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server.(*GenericAPIServer).PrepareRun(0xc0002b50e0, 0xc0000fa5e8)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/k8s.io/apiserver/pkg/server/genericapiserver.go:292 +0x5c
harmonycloud.cn/middleware-apiserver/vendor/sigs.k8s.io/apiserver-builder-alpha/pkg/cmd/server.(*ServerOptions).RunServer(0xc00009ea80, 0xc0000a8120, 0x1d0ff71, 0x3, 0x1d0fcb8, 0x2, 0x0, 0x0, 0x0, 0x1d10a03, ...)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/sigs.k8s.io/apiserver-builder-alpha/pkg/cmd/server/start.go:410 +0x483
harmonycloud.cn/middleware-apiserver/vendor/sigs.k8s.io/apiserver-builder-alpha/pkg/cmd/server.NewCommandStartServer.func1(0xc000161680, 0xc000503400, 0x0, 0x8, 0x0, 0x0)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/sigs.k8s.io/apiserver-builder-alpha/pkg/cmd/server/start.go:184 +0xf3
harmonycloud.cn/middleware-apiserver/vendor/github.com/spf13/cobra.(*Command).execute(0xc000161680, 0xc00004e0a0, 0x8, 0x8, 0xc000161680, 0xc00004e0a0)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/github.com/spf13/cobra/command.go:599 +0x3e6
harmonycloud.cn/middleware-apiserver/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000161680, 0xc000476510, 0x3d5fc40, 0xc000010018)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/github.com/spf13/cobra/command.go:689 +0x2be
harmonycloud.cn/middleware-apiserver/vendor/github.com/spf13/cobra.(*Command).Execute(...)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/github.com/spf13/cobra/command.go:648
harmonycloud.cn/middleware-apiserver/vendor/sigs.k8s.io/apiserver-builder-alpha/pkg/cmd/server.StartApiServerWithOptions(0xc0003d7f08, 0xc0000fa2a0, 0xc000589440)
/data/go/src/harmonycloud.cn/middleware-apiserver/vendor/sigs.k8s.io/apiserver-builder-alpha/pkg/cmd/server/start.go:118 +0x2cd
main.main()
/data/go/src/harmonycloud.cn/middleware-apiserver/cmd/apiserver/main.go:39 +0x10d
I1222 19:38:08.467916 28678 secure_serving.go:123] Serving securely on [::]:1443
I1222 19:38:08.793856 28678 client.go:361] parsed scheme: "endpoint"
I1222 19:38:08.793913 28678 endpoint.go:66] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}]
I1222 19:38:09.333099 28678 client.go:361] parsed scheme: "endpoint"
I1222 19:38:09.333140 28678 endpoint.go:66] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}]
^CI1222 19:38:13.717121 28678 secure_serving.go:167] Stopped listening on [::]:1443
[root@liabio apiserver]#
[root@master middleware-apiserver]# echo 'apiVersion: middleware.hc.middleware/v1alpha1
kind: RemoveNode
metadata:
name: remove-node-example
namespace: api-ext
spec: {}' > removenodes.yaml
[root@master middleware-apiserver]# kubectl --kubeconfig kubeconfig apply -f removenodes.yaml
removenode.middleware.hc.middleware/remove-node-example created
[root@master middleware-apiserver]# kubectl --kubeconfig kubeconfig get removenode
NAME CREATED AT
remove-node-example 2019-12-22T14:28:01Z
[root@master middleware-apiserver]#
[root@master middleware-apiserver]# kubectl --kubeconfig kubeconfig api-resources
NAME SHORTNAMES APIGROUP NAMESPACED KIND
removenodes middleware.hc.middleware false RemoveNode
[root@master middleware-apiserver]# curl -k https://127.0.0.1:9443/apis/middleware.hc.middleware/v1alpha1/removenodes
{
"kind": "RemoveNodeList",
"apiVersion": "middleware.hc.middleware/v1alpha1",
"metadata": {
"selfLink": "/apis/middleware.hc.middleware/v1alpha1/removenodes",
"resourceVersion": "2288899"
},
"items": [
{
"metadata": {
"name": "remove-node-example",
"selfLink": "/apis/middleware.hc.middleware/v1alpha1/removenodes/remove-node-example",
"uid": "80fc8b27-8bcb-4152-96ab-ed1c1d494193",
"resourceVersion": "2288351",
"generation": 1,
"creationTimestamp": "2019-12-22T14:28:01Z",
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"middleware.hc.middleware/v1alpha1\",\"kind\":\"RemoveNode\",\"metadata\":{\"annotations\":{},\"name\":\"remove-node-example\"},\"spec\":{}}\n"
}
},
"spec": {},
"status": {}
}
]
}
[root@master middleware-apiserver]# curl -k https://127.0.0.1:9443/apis/middleware.hc.middleware/v1alpha1/removenodes/remove-node-example
{
"kind": "RemoveNode",
"apiVersion": "middleware.hc.middleware/v1alpha1",
"metadata": {
"name": "remove-node-example",
"selfLink": "/apis/middleware.hc.middleware/v1alpha1/removenodes/remove-node-example",
"uid": "80fc8b27-8bcb-4152-96ab-ed1c1d494193",
"resourceVersion": "2288351",
"generation": 1,
"creationTimestamp": "2019-12-22T14:28:01Z",
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"middleware.hc.middleware/v1alpha1\",\"kind\":\"RemoveNode\",\"metadata\":{\"annotations\":{},\"name\":\"remove-node-example\"},\"spec\":{}}\n"
}
},
"spec": {},
"status": {}
}
[root@master middleware-apiserver]# ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key get /registry/hc.middleware/middleware.hc.middleware/removenodes/remove-node-example
/registry/hc.middleware/middleware.hc.middleware/removenodes/remove-node-example
{"kind":"RemoveNode","apiVersion":"middleware.hc.middleware/v1alpha1","metadata":{"name":"remove-node-example","uid":"80fc8b27-8bcb-4152-96ab-ed1c1d494193","generation":1,"creationTimestamp":"2019-12-22T14:28:01Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"middleware.hc.middleware/v1alpha1\",\"kind\":\"RemoveNode\",\"metadata\":{\"annotations\":{},\"name\":\"remove-node-example\"},\"spec\":{}}\n"}},"spec":{},"status":{}}
编译生成镜像：
[root@master middleware-apiserver]# apiserver-boot build container --image hc-middleware/middleware-apiserver:v1
I1223 09:59:37.306181 19285 build_container.go:64] Will build docker Image from directory /tmp/apiserver-boot-build-container529891239
I1223 09:59:37.306242 19285 build_container.go:66] Writing the Dockerfile.
I1223 09:59:37.306915 19285 build_container.go:71] Building binaries for linux amd64.
I1223 09:59:37.306935 19285 build_executables.go:140] regenerating generated code. To disable regeneration, run with --generate=false.
I1223 09:59:37.424403 19285 generate.go:139] /root/gopath/src/bin/apiregister-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/... --input-dirs harmonycloud.cn/middleware-apiserver/pkg/controller/...
I1223 09:59:41.300081 19285 generate.go:154] /root/gopath/src/bin/conversion-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /root/gopath/src --go-header-file boilerplate.go.txt -O zz_generated.conversion --extra-peer-dirs k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/conversion,k8s.io/apimachinery/pkg/runtime
I1223 09:59:44.703834 19285 generate.go:168] /root/gopath/src/bin/deepcopy-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /root/gopath/src --go-header-file boilerplate.go.txt -O zz_generated.deepcopy
I1223 09:59:48.427380 19285 generate.go:218] /root/gopath/src/bin/openapi-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /root/gopath/src --go-header-file boilerplate.go.txt -i k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/api/resource,k8s.io/apimachinery/pkg/version,k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/util/intstr,k8s.io/api/admission/v1,k8s.io/api/admission/v1beta1,k8s.io/api/admissionregistration/v1,k8s.io/api/admissionregistration/v1beta1,k8s.io/api/apps/v1,k8s.io/api/apps/v1beta1,k8s.io/api/apps/v1beta2,k8s.io/api/auditregistration/v1alpha1,k8s.io/api/authentication/v1,k8s.io/api/authentication/v1beta1,k8s.io/api/authorization/v1,k8s.io/api/authorization/v1beta1,k8s.io/api/autoscaling/v1,k8s.io/api/autoscaling/v2beta1,k8s.io/api/autoscaling/v2beta2,k8s.io/api/batch/v1,k8s.io/api/batch/v1beta1,k8s.io/api/batch/v2alpha1,k8s.io/api/certificates/v1beta1,k8s.io/api/coordination/v1,k8s.io/api/coordination/v1beta1,k8s.io/api/core/v1,k8s.io/api/discovery/v1alpha1,k8s.io/api/events/v1beta1,k8s.io/api/extensions/v1beta1,k8s.io/api/imagepolicy/v1alpha1,k8s.io/api/networking/v1,k8s.io/api/networking/v1beta1,k8s.io/api/node/v1alpha1,k8s.io/api/node/v1beta1,k8s.io/api/policy/v1beta1,k8s.io/api/rbac/v1,k8s.io/api/rbac/v1alpha1,k8s.io/api/rbac/v1beta1,k8s.io/api/scheduling/v1,k8s.io/api/scheduling/v1alpha1,k8s.io/api/scheduling/v1beta1,k8s.io/api/settings/v1alpha1,k8s.io/api/storage/v1,k8s.io/api/storage/v1alpha1,k8s.io/api/storage/v1beta1,k8s.io/client-go/pkg/apis/clientauthentication/v1alpha1,k8s.io/client-go/pkg/apis/clientauthentication/v1beta1,k8s.io/api/core/v1 --report-filename violations.report --output-package harmonycloud.cn/middleware-apiserver/pkg/openapi
I1223 09:59:54.266663 19285 generate.go:233] /root/gopath/src/bin/defaulter-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /root/gopath/src --go-header-file boilerplate.go.txt -O zz_generated.defaults --extra-peer-dirs= k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/conversion,k8s.io/apimachinery/pkg/runtime
I1223 09:59:57.528109 19285 generate.go:252] /root/gopath/src/bin/client-gen -o /root/gopath/src --go-header-file boilerplate.go.txt --input-base harmonycloud.cn/middleware-apiserver/pkg/apis --input middleware/v1alpha1 --clientset-path harmonycloud.cn/middleware-apiserver/pkg/client/clientset_generated --clientset-name clientset
I1223 10:00:00.698888 19285 generate.go:282] /root/gopath/src/bin/lister-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /root/gopath/src --go-header-file boilerplate.go.txt --output-package harmonycloud.cn/middleware-apiserver/pkg/client/listers_generated
I1223 10:00:04.096144 19285 generate.go:297] /root/gopath/src/bin/informer-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /root/gopath/src --go-header-file boilerplate.go.txt --output-package harmonycloud.cn/middleware-apiserver/pkg/client/informers_generated --listers-package harmonycloud.cn/middleware-apiserver/pkg/client/listers_generated --versioned-clientset-package harmonycloud.cn/middleware-apiserver/pkg/client/clientset_generated/clientset
I1223 10:00:07.317993 19285 build_executables.go:151] CGO_ENABLED=0
I1223 10:00:07.318025 19285 build_executables.go:154] GOOS=linux
I1223 10:00:07.318033 19285 build_executables.go:158] GOARCH=amd64
I1223 10:00:07.318050 19285 build_executables.go:161] go build -o /tmp/apiserver-boot-build-container529891239/apiserver cmd/apiserver/main.go
I1223 10:00:13.294148 19285 build_executables.go:180] go build -o /tmp/apiserver-boot-build-container529891239/controller-manager cmd/manager/main.go
I1223 10:00:18.488008 19285 build_container.go:79] Building the docker Image using /tmp/apiserver-boot-build-container529891239/Dockerfile.
I1223 10:00:18.488080 19285 util.go:113] docker build -t hc-middleware/middleware-apiserver:v1 /tmp/apiserver-boot-build-container529891239
Sending build context to Docker daemon 117.8MB
Step 1/5 : FROM ubuntu:14.04
14.04: Pulling from library/ubuntu
2e6e20c8e2e6: Pull complete
30bb187ac3fc: Pull complete
b7a5bcc4a58a: Pull complete
Digest: sha256:ffc76f71dd8be8c9e222d420dc96901a07b61616689a44c7b3ef6a10b7213de4
Status: Downloaded newer image for ubuntu:14.04
---> 6e4f1fe62ff1
Step 2/5 : RUN apt-get update
---> Running in f791260931d5
Get:1 http://security.ubuntu.com trusty-security InRelease [65.9 kB]
Get:2 http://security.ubuntu.com trusty-security/main amd64 Packages [1032 kB]
Ign http://archive.ubuntu.com trusty InRelease
Get:3 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:4 http://archive.ubuntu.com trusty-backports InRelease [65.9 kB]
Get:5 http://security.ubuntu.com trusty-security/restricted amd64 Packages [18.1 kB]
Get:6 https://esm.ubuntu.com trusty-infra-security InRelease
Get:7 http://security.ubuntu.com trusty-security/universe amd64 Packages [377 kB]
Hit http://archive.ubuntu.com trusty Release.gpg
Get:8 http://archive.ubuntu.com trusty-updates/main amd64 Packages [1460 kB]
Get:9 http://security.ubuntu.com trusty-security/multiverse amd64 Packages [4730 B]
Get:10 https://esm.ubuntu.com trusty-infra-updates InRelease
Get:11 https://esm.ubuntu.com trusty-infra-security/main amd64 Packages
Get:12 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [21.4 kB]
Get:13 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [671 kB]
Get:14 http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages [16.1 kB]
Get:15 http://archive.ubuntu.com trusty-backports/main amd64 Packages [14.7 kB]
Get:16 http://archive.ubuntu.com trusty-backports/restricted amd64 Packages [40 B]
Get:17 http://archive.ubuntu.com trusty-backports/universe amd64 Packages [52.5 kB]
Get:18 https://esm.ubuntu.com trusty-infra-updates/main amd64 Packages
Get:19 http://archive.ubuntu.com trusty-backports/multiverse amd64 Packages [1392 B]
Hit http://archive.ubuntu.com trusty Release
Get:20 http://archive.ubuntu.com trusty/main amd64 Packages [1743 kB]
Get:21 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
Get:22 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]
Get:23 http://archive.ubuntu.com trusty/multiverse amd64 Packages [169 kB]
Fetched 13.6 MB in 6s (2091 kB/s)
Reading package lists...
Removing intermediate container f791260931d5
---> fff13be6f010
Step 3/5 : RUN apt-get install -y ca-certificates
---> Running in d72fbf95029a
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version.
ca-certificates set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Removing intermediate container d72fbf95029a
---> 7338d676955d
Step 4/5 : ADD apiserver .
---> 036e6a6be28d
Step 5/5 : ADD controller-manager .
---> 6286c241891f
Successfully built 6286c241891f
Successfully tagged hc-middleware/middleware-apiserver:v1
只生成apiserver：
apiserver-boot build container --image hc-middleware/middleware-apiserver:v1 --generate=false --targets=apiserver
生成deployment部署文件：
[root@master middleware-apiserver]# apiserver-boot build config --name custom-api-server-middleware --namespace kube-system --image hc-middleware/middleware-apiserver:v1
I1223 10:07:00.455348 2236 util.go:113] openssl req -x509 -newkey rsa:2048 -keyout config/certificates/apiserver_ca.key -out config/certificates/apiserver_ca.crt -days 365 -nodes -subj /C=un/ST=st/L=l/O=o/OU=ou/CN=custom-api-server-middleware-certificate-authority
Generating a 2048 bit RSA private key
..+++
........+++
writing new private key to 'config/certificates/apiserver_ca.key'
-----
I1223 10:07:00.475918 2236 util.go:113] openssl req -out config/certificates/apiserver.csr -new -newkey rsa:2048 -nodes -keyout config/certificates/apiserver.key -subj /C=un/ST=st/L=l/O=o/OU=ou/CN=custom-api-server-middleware.kube-system.svc
Generating a 2048 bit RSA private key
.....+++
....+++
writing new private key to 'config/certificates/apiserver.key'
-----
I1223 10:07:00.513096 2236 util.go:113] openssl x509 -req -days 365 -in config/certificates/apiserver.csr -CA config/certificates/apiserver_ca.crt -CAkey config/certificates/apiserver_ca.key -CAcreateserial -out config/certificates/apiserver.crt
Signature ok
subject=/C=un/ST=st/L=l/O=o/OU=ou/CN=custom-api-server-middleware.kube-system.svc
Getting CA Private Key
I1223 10:07:00.526055 2236 build_resource_config.go:224] Adding APIs:
I1223 10:07:00.526233 2236 build_resource_config.go:234] middleware.v1alpha1
会在config目录下生成以下文件：
[root@master config]# tree
.
|-- apiserver.yaml
`-- certificates
|-- apiserver_ca.crt
|-- apiserver_ca.key
|-- apiserver_ca.srl
|-- apiserver.crt
|-- apiserver.csr
`-- apiserver.key
1 directory, 7 files
生成的部署yaml文件如下：
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1alpha1.middleware.hc.middleware
labels:
api: middleware-server
apiserver: "true"
spec:
version: v1alpha1
group: middleware.hc.middleware
groupPriorityMinimum: 2000
service:
name: middleware-server
namespace: kube-system
versionPriority: 10
caBundle: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR0VENDQXAyZ0F3SUJBZ0lKQU9PajNwS2tGYUlTTUEwR0NTcUdTSWIzRFFFQkN3VUFNSEV4Q3pBSkJnTlYKQkFZVEFuVnVNUXN3Q1FZRFZRUUlEQUp6ZERFS01BZ0dBMVVFQnd3QmJERUtNQWdHQTFVRUNnd0JiekVMTUFrRwpBMVVFQ3d3Q2IzVXhNREF1QmdOVkJBTU1KMjFwWkdSc1pYZGhjbVV0YzJWeWRtVnlMV05sY25ScFptbGpZWFJsCkxXRjFkR2h2Y21sMGVUQWVGdzB5TURBMU1qTXhOakF4TWpoYUZ3MHlNVEExTWpNeE5qQXhNamhhTUhFeEN6QUoKQmdOVkJBWVRBblZ1TVFzd0NRWURWUVFJREFKemRERUtNQWdHQTFVRUJ3d0JiREVLTUFnR0ExVUVDZ3dCYnpFTApNQWtHQTFVRUN3d0NiM1V4TURBdUJnTlZCQU1NSjIxcFpHUnNaWGRoY21VdGMyVnlkbVZ5TFdObGNuUnBabWxqCllYUmxMV0YxZEdodmNtbDBlVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFPeXUKeXpuUC9wM2NMejFOZllnTFdsTFdDNGUvSW5iY1k5NGtrY3NpV1R6ZzhqTVU4TzBGY0tHQVRRU2hSRkFWYm1BbQpjT0l0ZlZNLzVVS3JJaUlJSVM3MmVubXo5SjNUSXFxN2o2UkVmdmRNSWNUSDlaWW55RzNIT0FkaDA2NklLOERrCkllRjVjemppbmdYU2tqeVNQQ3hLeHg5QitvUEZYeGFXdWwyQkhOTk5oSEh0aElzV2VIYmZ2UkEzL3YwWEl0cHEKc25jdWRCQ3hBQms4NUl3Z21JNExhYk9Md3JEWmUycklkblNVNXd4d0lPMjlyZ2dXYm0vVmNIbk1KMWQrNldNagpFN2R3cDFweThHVVRyQXZnR05obTRUYThhTDluOTErZDlTV0FVcDRRT2VYdUJVeE5SUEI0S3JuWGlrSFZlODlvCmw1dXNZS3RGVDVPUHArWDJ3Y2tDQXdFQUFhTlFNRTR3SFFZRFZSME9CQllFRkZQR3daS3RqTjVYMndTLzB5c0IKM0pTbHZLclVNQjhHQTFVZEl3UVlNQmFBRkZQR3daS3RqTjVYMndTLzB5c0IzSlNsdktyVU1Bd0dBMVVkRXdRRgpNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFEL1JoaTYxTzZiY2NSdExuS1lnejlHNExqLzRLa2NqCmZEYTdEd3A3K05DVWpyRHFqeGEvbFl1R1pSSGFqdi9SWndFSkwzWEJaRmhOT3dIUmlVVU8xK3hucHhNWFU1SWcKcTNTZzFNd2xIdng3SUZNQlVkVWlMYzhySFpTYkZTa2RNOFBxY0lWaGFwWEFtQ2d6U3VyeSs4R2t3SnpzUTdsTApjSUVrbkU3OFVTNkdIbjhxTHhVTEo4NzU4UmNtNHJXUEw3elc0c0RVNThKSktyV3h0WExzNDJRZzRRS09XaFRMClJ2MHNZOWlYRndGMmhaZk02Q2l6NEc4aWhWMG9sOWhvTTVIb09oMzlpQklic09DN09zN2N6emp3bUZkOTJ6azUKNjJIaEFFL0pHcThzVDZMc2FxdE1Cb0VBbHRuWlhENmZmVkZSZlRLczA4U3F4RXJMaVhuc0NJWT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
---
apiVersion: v1
kind: Service
metadata:
name: middleware-server
namespace: kube-system
labels:
api: middleware-server
apiserver: "true"
spec:
ports:
- port: 443
protocol: TCP
targetPort: 443
selector:
api: middleware-server
apiserver: "true"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: middleware-server
namespace: kube-system
labels:
api: middleware-server
apiserver: "true"
spec:
selector:
matchLabels:
api: middleware-server
apiserver: "true"
replicas: 1
template:
metadata:
labels:
api: middleware-server
apiserver: "true"
spec:
containers:
- name: apiserver
image: hc-middleware/middleware-apiserver:v1
readinessProbe:
httpGet:
port: 443
path: /readyz
failureThreshold: 1
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
livenessProbe:
httpGet:
port: 443
path: /healthz
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
volumeMounts:
- name: apiserver-certs
mountPath: /apiserver.local.config/certificates
readOnly: true
command:
- "./apiserver"
args:
- --etcd-servers=http://etcd-svc:2379
- --tls-cert-file=/apiserver.local.config/certificates/tls.crt
- --tls-private-key-file=/apiserver.local.config/certificates/tls.key
- --v=5
- --etcd-servers https://10.10.101.203:2379
- --etcd-cafile=D:\SoftwareAndProgram\program\Go\Development\src\harmonycloud.cn\middleware-apiserver\artifact\203\pki\etcd\ca.crt
- --etcd-certfile=D:\SoftwareAndProgram\program\Go\Development\src\harmonycloud.cn\middleware-apiserver\artifact\203\pki\apiserver-etcd-client.crt
- --etcd-keyfile=D:\SoftwareAndProgram\program\Go\Development\src\harmonycloud.cn\middleware-apiserver\artifact\203\pki\apiserver-etcd-client.key
- --kubeconfig=D:\SoftwareAndProgram\program\Go\Development\src\harmonycloud.cn\middleware-apiserver\artifact\203\config
- --authorization-kubeconfig=D:\SoftwareAndProgram\program\Go\Development\src\harmonycloud.cn\middleware-apiserver\artifact\203\config
- --authentication-kubeconfig=D:\SoftwareAndProgram\program\Go\Development\src\harmonycloud.cn\middleware-apiserver\artifact\203\config
- --delegated-auth=false
resources:
requests:
cpu: 100m
memory: 20Mi
limits:
cpu: 100m
memory: 30Mi
volumes:
- name: apiserver-certs
secret:
secretName: middleware-server
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: etcd
namespace: kube-system
spec:
selector:
matchLabels:
app: etcd
serviceName: "etcd"
replicas: 1
template:
metadata:
labels:
app: etcd
spec:
terminationGracePeriodSeconds: 10
containers:
- name: etcd
image: quay.io/coreos/etcd:latest
imagePullPolicy: Always
resources:
requests:
cpu: 100m
memory: 20Mi
limits:
cpu: 100m
memory: 30Mi
env:
- name: ETCD_DATA_DIR
value: /etcd-data-dir
command:
- /usr/local/bin/etcd
- --listen-client-urls
- http://0.0.0.0:2379
- --advertise-client-urls
- http://localhost:2379
ports:
- containerPort: 2379
volumeMounts:
- name: etcd-data-dir
mountPath: /etcd-data-dir
readinessProbe:
httpGet:
port: 2379
path: /health
failureThreshold: 1
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
livenessProbe:
httpGet:
port: 2379
path: /health
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
volumeClaimTemplates:
- metadata:
name: etcd-data-dir
annotations:
volume.beta.kubernetes.io/storage-class: standard
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: etcd-svc
namespace: kube-system
labels:
app: etcd
spec:
ports:
- port: 2379
name: etcd
targetPort: 2379
selector:
app: etcd
---
apiVersion: v1
kind: Secret
type: kubernetes.io/tls
metadata:
name: middleware-server
namespace: kube-system
labels:
api: middleware-server
apiserver: "true"
data:
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURXRENDQWtBQ0NRQ3VLVEJaR2lraUpUQU5CZ2txaGtpRzl3MEJBUXNGQURCeE1Rc3dDUVlEVlFRR0V3SjEKYmpFTE1Ba0dBMVVFQ0F3Q2MzUXhDakFJQmdOVkJBY01BV3d4Q2pBSUJnTlZCQW9NQVc4eEN6QUpCZ05WQkFzTQpBbTkxTVRBd0xnWURWUVFERENkdGFXUmtiR1YzWVhKbExYTmxjblpsY2kxalpYSjBhV1pwWTJGMFpTMWhkWFJvCmIzSnBkSGt3SGhjTk1qQXdOVEl6TVRZd01USTRXaGNOTWpFd05USXpNVFl3TVRJNFdqQnJNUXN3Q1FZRFZRUUcKRXdKMWJqRUxNQWtHQTFVRUNBd0NjM1F4Q2pBSUJnTlZCQWNNQVd3eENqQUlCZ05WQkFvTUFXOHhDekFKQmdOVgpCQXNNQW05MU1Tb3dLQVlEVlFRRERDRnRhV1JrYkdWM1lYSmxMWE5sY25abGNpNXJkV0psTFhONWMzUmxiUzV6CmRtTXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFESU1Qd1VObHZOL0FnN3lJY2EKNnNGN0hOL1AzbWhNSkcwam1nUkNuWm9xVkhlSkFiOFJ6VXdKOVY4UWZLM2ZaYmRJOHN6Z1M2cExCRkJTQXNMcQozMWNSYVNrT0JxRkV0UUJEQVFnVHI4YW5TMnYxT0pVT2xRdy9PdUxiTW1MdmYweXpGaTRYQStLTTBhRGkxVVJJClZrWVNIcnBWdEdNdHI3d0dyU2pvZXhUY2wyZEZLN3lZR2t1OUNwSFRvczBHVE5TVHFjYTJSUHorVnVjYXdFN2YKMjN1Y0hrUUF3Y05HT1Z0am01ZlNGOFBiNWVxWDlYUko1bGR5ZTZMUmVlazRNNGxDWjF3YlVUcTY0YU5RV2xyOQpTc2JheU43b25LSWs3OXNEM1hoOHFGMnhqS3MxNDIwUCtmSmdjVWZ1cjN3RnI4V3lQQ3ZqcnZXS2hEcHJ0alIvCk13bkRBZ01CQUFFd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFERUpUU1dvV1UvcDQrMEFFTzNEcFpaV0xnOW0KS0ZKTGttdUF3Vy9mZVVZenFzdEdVVnRzTVc3eU5tN3NhQU5PbGJteXZPMTZLSmVpY2hQMVk4MG9XeGlSNXc4NQpPamVjUnVTNm13ZzJjcHU5eGU5dnRqSjRNbkxmSWdJWTFnRVpiMFJZd1ppYi9nWUc5U2VWaThKQkdYdC82Rm5lCjIrY0YxV3ZXS2lmSS9pTEFIbnZteURqa1ZYelVQUEprbTQyaTkwVkg3VS9Kb1hvSjJmbFh2SHJqdGFjZmVYbE4KN1NueWNiU1hRTGl5aFg1OUtmcHZBMTlCZWtvOG43aG5QcURYL2dNdE1sSGhqbUFkayt2aDdiRGtxclFPMUFkdQpzYnZOaVp6bC9VSEZTbHEwdEJqQjFGTm1iN1R3aVFEc1Jhdy8zNXk0dHp5YXoxNDZJMzRqa1YrNUh0az0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRRElNUHdVTmx2Ti9BZzcKeUljYTZzRjdITi9QM21oTUpHMGptZ1JDblpvcVZIZUpBYjhSelV3SjlWOFFmSzNmWmJkSThzemdTNnBMQkZCUwpBc0xxMzFjUmFTa09CcUZFdFFCREFRZ1RyOGFuUzJ2MU9KVU9sUXcvT3VMYk1tTHZmMHl6Rmk0WEErS00wYURpCjFVUklWa1lTSHJwVnRHTXRyN3dHclNqb2V4VGNsMmRGSzd5WUdrdTlDcEhUb3MwR1ROU1RxY2EyUlB6K1Z1Y2EKd0U3ZjIzdWNIa1FBd2NOR09WdGptNWZTRjhQYjVlcVg5WFJKNWxkeWU2TFJlZWs0TTRsQ1oxd2JVVHE2NGFOUQpXbHI5U3NiYXlON29uS0lrNzlzRDNYaDhxRjJ4aktzMTQyMFArZkpnY1VmdXIzd0ZyOFd5UEN2anJ2V0toRHByCnRqUi9Nd25EQWdNQkFBRUNnZ0VBUWEvUkJ0RFBUc2RqN0RuZ0d2VHNoM2p4OTBNQzlmajkxckN5dFNrQmJPSksKVnFvOEI1dFZWKzhyUWoybXo2YVFmVjh0Mm1yNWRhUDB0dlF4QU9CSHNXN1FDY29qSUdoRHJiaW83Y3kvWWlWZApSeGxKejM4WWhhYk15UisvVnRadVZGVHl5cHA3dTdrTkVXRHBCdE5JS3FucWVqUzJWa2RUYXQxbWp1SFQ5bzBpCmNhRW0yZGFjWGpGWll5OFhOYkMrTjJpR1ova2VxQS9rc0trQ0orM3ZlSEZzYmdJWExYdGxhMmt3bkFMUTR2b2QKMndLRlFCUld5TkFRYytNMmtLVTBoNjBWMVRoYy9wTmd6NUk5cFNLaU9CV2dTKzVndVF6R05RTlUvYUFtYkgxegpVUjNHRHdYdmlhWE1ZZFp6aEl4dDRUMU5ycy90RU1pU3BPKzFmTUpnQVFLQmdRRHg2RDI5V05QaVhmdW1QelFSCkkzYXRnWE5ZcnNRbFpseEw4ZUdWWU9qdEx2K3E5SjVOZEdBUUt0SUNCdEQ3ZkxCaXpOUzhIYk5FWngxbXRodVUKRUduL2ZQNnFWcFROME4zYXJNVGk0MzAxKzVwMHhPc2lHL0Vzc1JHZnJtTGdsT0xpQUdPWmVOYzh2OUtOSjlwRwo2N09GN1hTNXRnNWNiY1RpQ2pDNkxyM09sd0tCZ1FEVDJwbmxadXlmbVkvSVNhbzZyN0c1cEFtQTRXbm45ZWFsCkRzMWJsSGhmZmtCeDBPc0VmZEZ6SkFyR2ZubEx1MFRDTldpM1lUcHIxeGlaeGk2VzB1S2VrZllPc1lUNGMxZjcKUlVNR3JmMTZkVitsN2k5eDJmb2VUVmx4TE9CODlpUVM4TGwwbExVREovd3Vmc3lTZm82eVJjbnZYZ2xvdVErdgpkVTJMVHdmdnRRS0JnRytXV0x4Nk9Dc29mVWUveWUzSjlSbXVLQjRWYUFkMGcwakpCKzBYS2ZzVy9mckhJaXJLCmJ2R21QditPVTJKZXJ5SzBKOEdxMWFITkN3UHhKeUdTdk1LTXp4Q1dKY2dXKysrK1dJZnN0anBybUdFYTRQZjMKK2JWSzdMMUMvUHRwMG9xNzhUVy9SMUlQampsczA4UjVQM09kQ0lZT0UrdlErc3ZtTlhuOHk3TjdBb0dBTDZoeQpqR2V0L1cxWEtoQmdTaGx5R3JmckoySU9LQTlxVm1GUkV6UUFMRU5EamhjWWQ0Y3lVZnVlck82UXJOZXc3UnVFCnRoUWFSMTQ0KzRiWXZqY3NvYzFtRkFJZFRxTHIvT3VrMjJEU3IvU3RrcTlWWHdCUlF2WFZsQjhyVlJFdUpsQ2UKOTdub1NwQjVLNkI4dHN2Zm4vMFJJMWI5T09EQXhaRFc0K3JnZHkwQ2dZRUFxOUNYY0cwZ2l3UTNCRWFTVFVNUwoxM2FJS3FSYlF6dFByL3hsczhoZklsMXQwdlNWRGZYZndvZ1lObDlwTk8vT0N3RVh1NmdncFRza2tYOStrQjZECkhQYUNsTVF4TFFkamNpN1hWTjhuUHZlSzVmNkZsRFZ5eDFLS2xFbWhLYmpjNmkwS2xxY0Y1YlREbER6OVlMS0cKMHBNZlNyWk5ld0xXc1p1eW0zVWVUdTA9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
启动报错：apiserver容器报错，导致controller报错：
[root@master config]# kubectl logs -f custom-api-server-middleware-6fcfb6c9c4-mqqkn -n kube-system apiserver
[root@liabio config]# kubectl logs -f -n kube-system middleware-server-5dc79f9888-htqpd
I0524 01:38:32.559944 1 start.go:359] loading in-cluster loopback client...
I0524 01:38:33.561737 1 client.go:361] parsed scheme: "endpoint"
I0524 01:38:33.561851 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{http://etcd-svc:2379 0 <nil>}]
W0524 01:38:37.958713 1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0524 01:38:37.958772 1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0524 01:38:37.964742 1 authentication.go:348] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
Error: unable to load configmap based request-header-client-ca-file: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
Usage:
[flags]
Flags:
--add_dir_header If true, adds the file directory to the header
--admission-control-config-file string File with admission control configuration.
--alsologtostderr log to standard error as well as files
--audit-dynamic-configuration Enables dynamic audit configuration. This feature also requires the DynamicAuditing feature flag
--audit-log-batch-buffer-size int The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
--audit-log-batch-max-size int The maximum size of a batch. Only used in batch mode. (default 1)
--audit-log-batch-max-wait duration The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode.
--audit-log-batch-throttle-burst int Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode.
--audit-log-batch-throttle-enable Whether batching throttling is enabled. Only used in batch mode.
--audit-log-batch-throttle-qps float32 Maximum average number of batches per second. Only used in batch mode.
--audit-log-format string Format of saved audits. "legacy" indicates 1-line text format for each event. "json" indicates structured json format. Known formats are legacy,json. (default "json")
--audit-log-maxage int The maximum number of days to retain old audit log files based on the timestamp encoded in their filename.
--audit-log-maxbackup int The maximum number of old audit log files to retain.
--audit-log-maxsize int The maximum size in megabytes of the audit log file before it gets rotated.
--audit-log-mode string Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict. (default "blocking")
--audit-log-path string If set, all requests coming to the apiserver will be logged to this file. '-' means standard out.
--audit-log-truncate-enabled Whether event and batch truncating is enabled.
--audit-log-truncate-max-batch-size int Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size. (default 10485760)
--audit-log-truncate-max-event-size int Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded. (default 102400)
--audit-log-version string API group and version used for serializing audit events written to log. (default "audit.k8s.io/v1")
--audit-policy-file string Path to the file that defines the audit policy configuration.
--audit-webhook-batch-buffer-size int The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
--audit-webhook-batch-max-size int The maximum size of a batch. Only used in batch mode. (default 400)
--audit-webhook-batch-max-wait duration The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode. (default 30s)
--audit-webhook-batch-throttle-burst int Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode. (default 15)
--audit-webhook-batch-throttle-enable Whether batching throttling is enabled. Only used in batch mode. (default true)
--audit-webhook-batch-throttle-qps float32 Maximum average number of batches per second. Only used in batch mode. (default 10)
--audit-webhook-config-file string Path to a kubeconfig formatted file that defines the audit webhook configuration.
--audit-webhook-initial-backoff duration The amount of time to wait before retrying the first failed request. (default 10s)
--audit-webhook-mode string Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict. (default "batch")
--audit-webhook-truncate-enabled Whether event and batch truncating is enabled.
--audit-webhook-truncate-max-batch-size int Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size. (default 10485760)
--audit-webhook-truncate-max-event-size int Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded. (default 102400)
--audit-webhook-version string API group and version used for serializing audit events written to webhook. (default "audit.k8s.io/v1")
--authentication-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io. This is optional. If empty, all token requests are considered to be anonymous and no client CA is looked up in the cluster.
--authentication-skip-lookup If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
--authentication-token-webhook-cache-ttl duration The duration to cache responses from the webhook token authenticator. (default 10s)
--authentication-tolerate-lookup-failure If true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.
--authorization-always-allow-paths strings A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server.
--authorization-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io. This is optional. If empty, all requests not skipped by authorization are forbidden.
--authorization-webhook-cache-authorized-ttl duration The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
--authorization-webhook-cache-unauthorized-ttl duration The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
--bind-address ip The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces). (default 0.0.0.0)
--cert-dir string The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
--client-ca-file string If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
--contention-profiling Enable lock contention profiling, if profiling is enabled
--default-watch-cache-size int Default watch cache size. If zero, watch cache will be disabled for resources that do not have a default watch size set. (default 100)
--delegated-auth Setup delegated auth (default true)
--delete-collection-workers int Number of workers spawned for DeleteCollection call. These are used to speed up namespace cleanup. (default 1)
--disable-admission-plugins strings admission plugins that should be disabled although they are in the default enabled plugins list (NamespaceLifecycle, MutatingAdmissionWebhook, ValidatingAdmissionWebhook). Comma-delimited list of admission plugins: MutatingAdmissionWebhook, NamespaceLifecycle, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
--egress-selector-config-file string File with apiserver egress selector configuration.
--enable-admission-plugins strings admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, MutatingAdmissionWebhook, ValidatingAdmissionWebhook). Comma-delimited list of admission plugins: MutatingAdmissionWebhook, NamespaceLifecycle, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
--enable-garbage-collector Enables the generic garbage collector. MUST be synced with the corresponding flag of the kube-controller-manager. (default true)
--encryption-provider-config string The file containing configuration for encryption providers to be used for storing secrets in etcd
--etcd-cafile string SSL Certificate Authority file used to secure etcd communication.
--etcd-certfile string SSL certification file used to secure etcd communication.
--etcd-compaction-interval duration The interval of compaction requests. If 0, the compaction request from apiserver is disabled. (default 5m0s)
--etcd-count-metric-poll-period duration Frequency of polling etcd for number of resources per type. 0 disables the metric collection. (default 1m0s)
--etcd-keyfile string SSL key file used to secure etcd communication.
--etcd-prefix string The prefix to prepend to all resource paths in etcd. (default "/registry/hc.middleware")
--etcd-servers strings List of etcd servers to connect with (scheme://ip:port), comma separated.
--etcd-servers-overrides strings Per-resource etcd servers overrides, comma separated. The individual override format: group/resource#servers, where servers are URLs, semicolon separated.
--feature-gates mapStringBool A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (ALPHA - default=false)
APIResponseCompression=true|false (BETA - default=true)
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RemoveSelfLink=true|false (ALPHA - default=false)
ServerSideApply=true|false (BETA - default=true)
StorageVersionHash=true|false (BETA - default=true)
StreamingProxyRedirects=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (BETA - default=true)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (ALPHA - default=false)
-h, --help help for this command
--http2-max-streams-per-connection int The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default. (default 1000)
--insecure-bind-address ip The IP address on which to serve the --insecure-port (set to 0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces). (DEPRECATED: This flag will be removed in a future version.)
--insecure-port int The port on which to serve unsecured, unauthenticated access. (DEPRECATED: This flag will be removed in a future version.)
--kubeconfig string kubeconfig file pointing at the 'core' kubernetes server.
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--print-bearer-token Print a curl command with the bearer token to test the server
--print-openapi Print the openapi json and exit
--profiling Enable profiling via web interface host:port/debug/pprof/ (default true)
--requestheader-allowed-names strings List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
--requestheader-client-ca-file string Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.
--requestheader-extra-headers-prefix strings List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
--requestheader-group-headers strings List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
--requestheader-username-headers strings List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
--secure-port int The port on which to serve HTTPS with authentication and authorization.If 0, don't serve HTTPS at all. (default 443)
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
--storage-backend string The storage backend for persistence. Options: 'etcd3' (default).
--storage-media-type string The media type to use to store objects in storage. Some resources or storage backends may only support a specific media type and will ignore this setting. (default "application/json")
--tls-cert-file string File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
--tls-cipher-suites strings Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be use. Possible values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_RC4_128_SHA
--tls-min-version string Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13
--tls-private-key-file string File containing the default x509 private key matching --tls-cert-file.
--tls-sni-cert-key namedCertKey A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
-v, --v Level number for the log level verbosity
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
--watch-cache Enable watch caching in the apiserver (default true)
--watch-cache-sizes strings Watch cache size settings for some resources (pods, nodes, etc.), comma separated. The individual setting format: resource[.group]#size, where resource is lowercase plural (no version), group is omitted for resources of apiVersion v1 (the legacy core API) and included for others, and size is a number. It takes effect when watch-cache is enabled. Some resources (replicationcontrollers, endpoints, nodes, pods, services, apiservices.apiregistration.k8s.io) have system defaults set by heuristics, others default to default-watch-cache-size
panic: unable to load configmap based request-header-client-ca-file: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
会创建pvc：
[root@liabio config]# kubectl get pvc -n kube-system
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
etcd-data-dir-etcd-0 Pending standard 40m
[root@liabio config]# kubectl get pv -n kube-system
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
oss-csi-pv 5Gi RWX Retain Bound default/oss-pvc 160d
pvc-26df65be-8312-442b-a018-a3044d643a16 2Gi RWO Retain Released kubesphere-system/openldap-pvc-openldap-0 nfs 160d
pvc-4318828d-3d75-4412-af61-2b0413795973 2Gi RWO Retain Released kubesphere-system/redis-pvc nfs 160d
pvc-664405bf-92f6-4032-a3bb-3fee1e7679be 20Gi RWO Retain Released kubesphere-monitoring-system/prometheus-k8s-system-db-prometheus-k8s-system-0 nfs 160d
pvc-ec88cbf7-2e95-4d59-85b8-796eb4846c01 20Gi RWO Retain Released kubesphere-monitoring-system/prometheus-k8s-db-prometheus-k8s-0 nfs 160d
[root@liabio config]# kubectl describe pvc etcd-data-dir-etcd-0 -n kube-system
Name: etcd-data-dir-etcd-0
Namespace: kube-system
StorageClass: standard
Status: Pending
Volume:
Labels: app=etcd
Annotations: volume.beta.kubernetes.io/storage-class: standard
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Mounted By: etcd-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 78s (x162 over 41m) persistentvolume-controller storageclass.storage.k8s.io "standard" not found
[root@10 config]# curl -k https://10.10.101.203:6443/apis/middleware.hc.middleware/v1alpha1 -H 'Authorization: Bearer aabb00.middleware050323' -v
* About to connect() to 10.10.101.203 port 6443 (#0)
* Trying 10.10.101.203...
* Connected to 10.10.101.203 (10.10.101.203) port 6443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=kube-apiserver
* start date: 4月 27 08:52:37 2020 GMT
* expire date: 4月 03 08:52:39 2120 GMT
* common name: kube-apiserver
* issuer: CN=kubernetes
> GET /apis/middleware.hc.middleware/v1alpha1 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.10.101.203:6443
> Accept: */*
> Authorization: Bearer aabb00.middleware050323
>
< HTTP/1.1 503 Service Unavailable
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Sun, 24 May 2020 08:45:49 GMT
< Content-Length: 123
<
* Connection #0 to host 10.10.101.203 left intact
Error trying to reach service: 'x509: certificate is valid for localhost, localhost, not middleware-server.kube-system.svc'
参考了metrics-server的部署文件，APIService中要加insecureSkipTLSVerify: true
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1alpha1.middleware.hc.middleware
labels:
api: middleware-server
apiserver: "true"
spec:
version: v1alpha1
group: middleware.hc.middleware
groupPriorityMinimum: 2000
insecureSkipTLSVerify: true
service:
name: middleware-server
namespace: kube-system
versionPriority: 10
报错：
unexpected ListAndWatch error: storage/cacher.go:/middleware.hc.middleware/removenodes: Unable to sync list result: couldn't compute key: Namespace parameter required
--watch-cache参数关闭EnableWatchCache
E0524 11:49:59.778204 1 pathrecorder.go:107] registered "/healthz/etcd" from goroutine 1 [running]:
runtime/debug.Stack(0x1b13460, 0xc0005cc8d0, 0xc000d8cb50)
1.17.0
使用文档：https://github.com/kubernetes-sigs/apiserver-builder-alpha/blob/master/docs/tools_user_guide.md--non-namespaced=true表示生成不区分namespace的资源：
[root@liabio harmonycloud.cn]# cd middleware-apiserver/
[root@liabio middleware-apiserver]# ll
total 4
-rw-r--r-- 1 root root 570 May 15 17:19 boilerplate.go.txt
[root@liabio middleware-apiserver]# apiserver-boot -h
apiserver-boot development kit for building Kubernetes extensions in go.
Usage:
apiserver-boot [flags]
apiserver-boot [command]
Examples:
# Initialize your repository with scaffolding directories and go files.
apiserver-boot init repo --domain example.com
# Create new resource "Bee" in the "insect" group with version "v1beta1"
apiserver-boot create group version resource --group insect --version v1beta1 --kind Bee
# Build the generated code, apiserver and controller-manager so they be run locally.
apiserver-boot build executables
# Run the tests that were created for your resources
# Requires generated code was already built by "build executables" or "build generated"
go test ./pkg/...
# Run locally by starting a local etcd, apiserver and controller-manager
# Produces a kubeconfig to talk to the local server
apiserver-boot run local
# Check the api versions of the locally running server
kubectl --kubeconfig kubeconfig api-versions
# Build an image and run in a cluster in the default namespace
# Note: after running this you should clear the discovery service
# cache before running kubectl with "rm -rf ~/.kube/cache/discovery/"
apiserver-boot run in-cluster --name creatures --namespace default --image repo/name:tag
Available Commands:
build Command group for building source into artifacts.
create Command group for bootstrapping new resources.
help Help about any command
init Command group for bootstrapping new go projects.
run Command group for launching instances.
version Print the apisever-builder version.
Flags:
--copyright string Location of copyright boilerplate file. (default "boilerplate.go.txt")
-h, --help help for apiserver-boot
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
Use "apiserver-boot [command] --help" for more information about a command.
[root@liabio middleware-apiserver]# apiserver-boot init repo --domain hc.middleware
I0515 19:06:56.415427 5991 repo.go:94] rendering go mod file
2020-05-15 19:06:56.416216 I | extracted tarball into .: 2 files, 1 dirs (1.217545ms)
[root@liabio middleware-apiserver]# ll
total 116
drwx------ 2 root root 4096 May 15 19:06 bin
-rw-r--r-- 1 root root 570 May 15 17:19 boilerplate.go.txt
-rw-r--r-- 1 root root 185 May 15 19:06 BUILD.bazel
drwx------ 4 root root 4096 May 15 19:06 cmd
-rw-r--r-- 1 root root 1889 Jan 1 2000 go.mod
-rw-r--r-- 1 root root 85603 Jan 1 2000 go.sum
drwx------ 6 root root 4096 May 15 19:06 pkg
-rw-r--r-- 1 root root 79 May 15 19:06 PROJECT
-rw-r--r-- 1 root root 650 May 15 19:06 WORKSPACE
[root@liabio middleware-apiserver]# apiserver-boot create group version resource --group middleware --version v1alpha1 --non-namespaced=true --kind RemoveNode
Create Resource [y/n]
y
Create Controller [y/n]
y
Create Admission Controller [y/n]
y
[root@liabio middleware-apiserver]# apiserver-boot build generated
I0515 19:07:26.008686 6400 generate.go:140] /data/go/bin/apiregister-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/... --input-dirs harmonycloud.cn/middleware-apiserver/pkg/controller/... --go-header-file boilerplate.go.txt
I0515 19:08:30.407001 6400 generate.go:155] /data/go/bin/conversion-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.conversion --extra-peer-dirs k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/conversion,k8s.io/apimachinery/pkg/runtime
I0515 19:10:34.649246 6400 generate.go:169] /data/go/bin/deepcopy-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.deepcopy
I0515 19:12:22.740100 6400 generate.go:221] /data/go/bin/openapi-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt -i k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/api/resource,k8s.io/apimachinery/pkg/version,k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/util/intstr,k8s.io/api/core/v1,k8s.io/api/apps/v1 --report-filename violations.report --output-package harmonycloud.cn/middleware-apiserver/pkg/openapi
I0515 19:14:10.920033 6400 generate.go:236] /data/go/bin/defaulter-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.defaults --extra-peer-dirs= k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/conversion,k8s.io/apimachinery/pkg/runtime
I0515 19:15:57.929931 6400 generate.go:255] /data/go/bin/client-gen -o /data/go/src --go-header-file boilerplate.go.txt --input-base harmonycloud.cn/middleware-apiserver/pkg/apis --input middleware/v1alpha1 --clientset-path harmonycloud.cn/middleware-apiserver/pkg/client/clientset_generated --clientset-name clientset
I0515 19:17:45.316671 6400 generate.go:285] /data/go/bin/lister-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt --output-package harmonycloud.cn/middleware-apiserver/pkg/client/listers_generated
I0515 19:19:31.807611 6400 generate.go:300] /data/go/bin/informer-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt --output-package harmonycloud.cn/middleware-apiserver/pkg/client/informers_generated --listers-package harmonycloud.cn/middleware-apiserver/pkg/client/listers_generated --versioned-clientset-package harmonycloud.cn/middleware-apiserver/pkg/client/clientset_generated/clientset
[root@liabio middleware-apiserver]#
[root@liabio middleware-apiserver]#
[root@liabio middleware-apiserver]#
[root@liabio middleware-apiserver]#
[root@liabio middleware-apiserver]#
[root@liabio middleware-apiserver]#
[root@liabio middleware-apiserver]#
[root@liabio middleware-apiserver]# apiserver-boot run local
I0515 19:23:47.560536 16479 build_executables.go:156] regenerating generated code. To disable regeneration, run with --generate=false.
I0515 19:23:47.560793 16479 generate.go:140] /data/go/bin/apiregister-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/... --input-dirs harmonycloud.cn/middleware-apiserver/pkg/controller/... --go-header-file boilerplate.go.txt
I0515 19:25:45.631890 16479 generate.go:155] /data/go/bin/conversion-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.conversion --extra-peer-dirs k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/conversion,k8s.io/apimachinery/pkg/runtime
I0515 19:27:33.442132 16479 generate.go:169] /data/go/bin/deepcopy-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.deepcopy
I0515 19:29:20.709494 16479 generate.go:221] /data/go/bin/openapi-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt -i k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/api/resource,k8s.io/apimachinery/pkg/version,k8s.io/apimachinery/pkg/runtime,k8s.io/apimachinery/pkg/util/intstr,k8s.io/api/core/v1,k8s.io/api/apps/v1 --report-filename violations.report --output-package harmonycloud.cn/middleware-apiserver/pkg/openapi
I0515 19:31:09.340656 16479 generate.go:236] /data/go/bin/defaulter-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware -o /data/go/src --go-header-file boilerplate.go.txt -O zz_generated.defaults --extra-peer-dirs= k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/conversion,k8s.io/apimachinery/pkg/runtime
I0515 19:32:56.450561 16479 generate.go:255] /data/go/bin/client-gen -o /data/go/src --go-header-file boilerplate.go.txt --input-base harmonycloud.cn/middleware-apiserver/pkg/apis --input middleware/v1alpha1 --clientset-path harmonycloud.cn/middleware-apiserver/pkg/client/clientset_generated --clientset-name clientset
I0515 19:34:44.597559 16479 generate.go:285] /data/go/bin/lister-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt --output-package harmonycloud.cn/middleware-apiserver/pkg/client/listers_generated
I0515 19:36:32.225560 16479 generate.go:300] /data/go/bin/informer-gen --input-dirs harmonycloud.cn/middleware-apiserver/pkg/apis/middleware/v1alpha1 -o /data/go/src --go-header-file boilerplate.go.txt --output-package harmonycloud.cn/middleware-apiserver/pkg/client/informers_generated --listers-package harmonycloud.cn/middleware-apiserver/pkg/client/listers_generated --versioned-clientset-package harmonycloud.cn/middleware-apiserver/pkg/client/clientset_generated/clientset
I0515 19:38:19.144098 16479 build_executables.go:168] CGO_ENABLED=0
I0515 19:38:19.144141 16479 build_executables.go:178] go build -o bin/apiserver cmd/apiserver/main.go
go: downloading github.com/go-openapi/loads v0.19.4
go: downloading github.com/spf13/cobra v0.0.5
go: downloading gopkg.in/natefinch/lumberjack.v2 v2.0.0
go: extracting gopkg.in/natefinch/lumberjack.v2 v2.0.0
go: extracting github.com/spf13/cobra v0.0.5
go: extracting github.com/go-openapi/loads v0.19.4
go: downloading github.com/go-openapi/analysis v0.19.5
go: extracting github.com/go-openapi/analysis v0.19.5
go: downloading github.com/go-openapi/strfmt v0.19.3
go: extracting github.com/go-openapi/strfmt v0.19.3
go: downloading github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go: downloading go.mongodb.org/mongo-driver v1.1.2
go: downloading github.com/mitchellh/mapstructure v1.1.2
go: downloading github.com/go-openapi/errors v0.19.2
go: extracting github.com/mitchellh/mapstructure v1.1.2
go: extracting github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go: extracting github.com/go-openapi/errors v0.19.2
go: extracting go.mongodb.org/mongo-driver v1.1.2
go: downloading github.com/go-stack/stack v1.8.0
go: extracting github.com/go-stack/stack v1.8.0
I0515 19:39:50.805300 16479 build_executables.go:201] go build -o bin/controller-manager cmd/manager/main.go
go: downloading github.com/go-logr/zapr v0.1.1
go: extracting github.com/go-logr/zapr v0.1.1
I0515 19:40:03.284529 16479 local.go:199] etcd
I0515 19:40:03.595652 16479 local.go:203] Failed to run etcd, error: exit status 1
I0515 19:40:05.284658 16479 local.go:199] bin/apiserver --etcd-servers=http://localhost:2379 --secure-port=9443 --insecure-port=8080 --insecure-bind-address=127.0.0.1 --delegated-auth=false
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
I0515 19:40:05.773989 30675 serving.go:306] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0515 19:40:05.774059 30675 start.go:359] loading in-cluster loopback client...
W0515 19:40:05.774070 30675 start.go:250] attempting to instantiate loopback client but failed: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
W0515 19:40:06.168032 30675 authorization.go:47] Authorization is disabled
W0515 19:40:06.168074 30675 authentication.go:92] Authentication is disabled
I0515 19:40:06.169877 30675 client.go:361] parsed scheme: "endpoint"
I0515 19:40:06.169911 30675 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{http://localhost:2379 0 <nil>}]
I0515 19:40:06.774719 30675 client.go:361] parsed scheme: "endpoint"
I0515 19:40:06.774811 30675 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{http://localhost:2379 0 <nil>}]
I0515 19:40:07.284767 16479 local.go:131] to test the server run `kubectl --kubeconfig kubeconfig api-versions`
[root@liabio middleware-apiserver]#
[root@liabio middleware-apiserver]# panic: context deadline exceeded
goroutine 1 [running]:
sigs.k8s.io/apiserver-builder-alpha/pkg/builders.(*versionedResourceBuilder).Build(0xc000293e50, 0x1d123bb, 0x18, 0x2037aa0, 0xc00009ed80, 0xc0001ab040, 0xeb00000001c2ef40)
/data/go/pkg/mod/sigs.k8s.io/apiserver-builder-alpha@v1.17.0/pkg/builders/api_versioned_resource_builder.go:153 +0x477
sigs.k8s.io/apiserver-builder-alpha/pkg/builders.(*versionedResourceBuilder).registerEndpoints(0xc000293e50, 0x1d123bb, 0x18, 0x2037aa0, 0xc00009ed80, 0xc000149aa0)
/data/go/pkg/mod/sigs.k8s.io/apiserver-builder-alpha@v1.17.0/pkg/builders/api_versioned_resource_builder.go:196 +0x19f
sigs.k8s.io/apiserver-builder-alpha/pkg/builders.(*VersionedApiBuilder).registerEndpoints(0xc0004db960, 0x2037aa0, 0xc00009ed80, 0xc000149950)
/data/go/pkg/mod/sigs.k8s.io/apiserver-builder-alpha@v1.17.0/pkg/builders/api_version_builder.go:67 +0xcd
sigs.k8s.io/apiserver-builder-alpha/pkg/builders.(*APIGroupBuilder).registerEndpoints(0xc000209da0, 0x2037aa0, 0xc00009ed80, 0xc000149950)
/data/go/pkg/mod/sigs.k8s.io/apiserver-builder-alpha@v1.17.0/pkg/builders/api_group_builder.go:84 +0x68
sigs.k8s.io/apiserver-builder-alpha/pkg/builders.(*APIGroupBuilder).Build(0xc000209da0, 0x2037aa0, 0xc00009ed80, 0x1d0b919)
/data/go/pkg/mod/sigs.k8s.io/apiserver-builder-alpha@v1.17.0/pkg/builders/api_group_builder.go:106 +0x2b0
sigs.k8s.io/apiserver-builder-alpha/pkg/apiserver.completedConfig.New(0xc0002c48c0, 0xc0000e8870, 0xc0000e8480, 0xc00001a028)
/data/go/pkg/mod/sigs.k8s.io/apiserver-builder-alpha@v1.17.0/pkg/apiserver/apiserver.go:111 +0x208
sigs.k8s.io/apiserver-builder-alpha/pkg/cmd/server.(*ServerOptions).RunServer(0xc0000ec6c0, 0xc00009a1e0, 0x1cf4338, 0x3, 0x1cf3d6d, 0x2, 0x0, 0x0, 0x0, 0x1cf52cb, ...)
/data/go/pkg/mod/sigs.k8s.io/apiserver-builder-alpha@v1.17.0/pkg/cmd/server/start.go:405 +0x3b1
sigs.k8s.io/apiserver-builder-alpha/pkg/cmd/server.NewCommandStartServer.func1(0xc000111900, 0xc0000b4690, 0x0, 0x5, 0x0, 0x0)
/data/go/pkg/mod/sigs.k8s.io/apiserver-builder-alpha@v1.17.0/pkg/cmd/server/start.go:185 +0xf1
github.com/spf13/cobra.(*Command).execute(0xc000111900, 0xc00004c1f0, 0x5, 0x5, 0xc000111900, 0xc00004c1f0)
/data/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0xc000111900, 0xc0000ca400, 0x2038820, 0xc00000e018)
/data/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
/data/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
sigs.k8s.io/apiserver-builder-alpha/pkg/cmd/server.StartApiServerWithOptions(0xc000533ed0, 0xc00000e938, 0xc00000e928)
/data/go/pkg/mod/sigs.k8s.io/apiserver-builder-alpha@v1.17.0/pkg/cmd/server/start.go:119 +0x2cc
main.main()
/data/go/src/harmonycloud.cn/middleware-apiserver/cmd/apiserver/main.go:37 +0x10b
参考
若有收获，就点个赞吧
0 人点赞
