rx-m-kubernetes-bootcamp-lab-08 - 图1

Kubernetes

Lab 8 – Metrics

Starting from Kubernetes 1.8, resource usage metrics, such as container CPU and memory usage, are available in
Kubernetes through the Metrics API. These metrics can be either accessed directly by a user with kubectl top, or used
by a controller in the cluster, such as the Horizontal Pod Autoscaler.

The Metrics API reports the current resource utilization of a node or pod or set thereof. The metrics API is designed to
expose the current resource utilization only. Those wishing to view a history of resource utilization should use
a monitoring tool and database such as Prometheus or InfluxDB/Grafana.

Kubernetes 1.7 introduced the aggregator API feature. This allows add on services to extend the base K8s API. The
Metrics API uses the aggregator feature and exposes an API available at /apis/metrics.k8s.io/, offering the same
security, scalability and reliability guarantees as the rest of the api-server hosted end points.

The Metrics server was first released in September 2017 and is not installed automatically by all K8s installers. The
Metrics Server does not run as a module within the Controller Manager, rather it runs as a stand alone deployment,
typically in the kube-system namespace. The Metrics Server requires an SA with appropriate RBAC roles so that it can
communicate with the api-server and extend the K8s API.

The Metrics Server must also be equipped with certificates/keys that will allow it to connect to each of the Kubelets in
the system. Kubelets expose a /metrics end point, the Metrics Server scrapes this endpoint for metrics data regularly,
storing the results in memory. When in bound requests for metrics arrive at the api-server the metrics server is called
to answer them.

In this lab we’ll install the Metrics Server and explore the metrics enabled features of Kubernetes. Delete all user
defined deployments, services, and other resources before starting.

1. Explore the Metrics Server repository

Basic Metrics Server deployment manifests are available in the /deploy directory in the Metrics Server source repo:

https://github.com/kubernetes-incubator/metrics-server

The Metrics Server repository includes a set of manifests that create all of the cluster resources needed to run the
Metrics Server. Clone the Metrics Server repository from GitHub:

  1. user@ubuntu:~$ git clone --depth 1 https://github.com/kubernetes-incubator/metrics-server
  3. Cloning into 'metrics-server'...
  4. remote: Enumerating objects: 88, done.
  5. remote: Counting objects: 100% (88/88), done.
  6. remote: Compressing objects: 100% (74/74), done.
  7. remote: Total 88 (delta 11), reused 39 (delta 5), pack-reused 0
  8. Unpacking objects: 100% (88/88), done.
  9. Checking connectivity... done.
  11. user@ubuntu:~$

Now change into the repository directory and display the manifests in the deploy/1.8+ directory:

  1.   user@ubuntu:~$ cd metrics-server/
  3. user@ubuntu:~/metrics-server$ ls -l
  5. total 152
  6. drwxrwxr-x  3 user user  4096 Jan 24 15:44 cmd
  7. -rw-rw-r--  1 user user   148 Jan 24 15:44 code-of-conduct.md
  8. -rw-rw-r--  1 user user   418 Jan 24 15:44 CONTRIBUTING.md
  9. drwxrwxr-x  6 user user  4096 Jan 24 15:44 deploy
  10. -rw-rw-r--  1 user user  3364 Jan 24 15:44 go.mod
  11. -rw-rw-r--  1 user user 84975 Jan 24 15:44 go.sum
  12. drwxrwxr-x  2 user user  4096 Jan 24 15:44 hack
  13. -rw-rw-r--  1 user user 11357 Jan 24 15:44 LICENSE
  14. -rw-rw-r--  1 user user  4020 Jan 24 15:44 Makefile
  15. -rw-rw-r--  1 user user   201 Jan 24 15:44 OWNERS
  16. -rw-rw-r--  1 user user   177 Jan 24 15:44 OWNERS_ALIASES
  17. drwxrwxr-x 11 user user  4096 Jan 24 15:44 pkg
  18. -rw-rw-r--  1 user user  5305 Jan 24 15:44 README.md
  19. -rw-rw-r--  1 user user   541 Jan 24 15:44 SECURITY_CONTACTS
  20. drwxrwxr-x  2 user user  4096 Jan 24 15:44 test
  22. user@ubuntu:~/metrics-server$ ls -l deploy/
  24. total 16
  25. drwxrwxr-x 2 user user 4096 Jan 24 15:44 1.7
  26. drwxrwxr-x 2 user user 4096 Jan 24 15:44 1.8+
  27. drwxrwxr-x 2 user user 4096 Jan 24 15:44 docker
  28. drwxrwxr-x 2 user user 4096 Jan 24 15:44 minikube
  30. user@ubuntu:~/metrics-server$ ls -l deploy/1.8+/
  32. total 28
  33. -rw-rw-r-- 1 user user  397 Jan 24 15:44 aggregated-metrics-reader.yaml
  34. -rw-rw-r-- 1 user user  303 Jan 24 15:44 auth-delegator.yaml
  35. -rw-rw-r-- 1 user user  324 Jan 24 15:44 auth-reader.yaml
  36. -rw-rw-r-- 1 user user  298 Jan 24 15:44 metrics-apiservice.yaml
  37. -rw-rw-r-- 1 user user 1183 Jan 24 15:44 metrics-server-deployment.yaml
  38. -rw-rw-r-- 1 user user  297 Jan 24 15:44 metrics-server-service.yaml
  39. -rw-rw-r-- 1 user user  532 Jan 24 15:44 resource-reader.yaml
  41. user@ubuntu:~/metrics-server$

The Metrics Server deployment uses sever manifests. Generate a listing of the types of resources that will be created:

  1. user@ubuntu:~/metrics-server$ kubectl apply -f deploy/1.8+/ --dry-run
  3. clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created (dry run)
  4. clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created (dry run)
  5. rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created (dry run)
  6. apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created (dry run)
  7. serviceaccount/metrics-server created (dry run)
  8. deployment.apps/metrics-server created (dry run)
  9. service/metrics-server created (dry run)
  10. clusterrole.rbac.authorization.k8s.io/system:metrics-server created (dry run)
  11. clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created (dry run)
  13. user@ubuntu:~/metrics-server$

Though a range of resources are defined, you can see that most are RBAC elements:

  • deploy/1.8+/aggregated-metrics-reader.yaml:kind: ClusterRole
  • deploy/1.8+/auth-delegator.yaml:kind: ClusterRoleBinding
  • deploy/1.8+/auth-reader.yaml:kind: RoleBinding
  • deploy/1.8+/metrics-server-deployment.yaml:kind: ServiceAccount
  • deploy/1.8+/resource-reader.yaml:kind: ClusterRole
  • deploy/1.8+/resource-reader.yaml:kind: ClusterRoleBinding

The manifests create two ClusterRoles. Display the rules defined:

  1. user@ubuntu:~/metrics-server$ grep rules -A21 deploy/1.8+/*
  3. deploy/1.8+/aggregated-metrics-reader.yaml:rules:
  4. deploy/1.8+/aggregated-metrics-reader.yaml-- apiGroups: ["metrics.k8s.io"]
  5. deploy/1.8+/aggregated-metrics-reader.yaml-  resources: ["pods", "nodes"]
  6. deploy/1.8+/aggregated-metrics-reader.yaml-  verbs: ["get", "list", "watch"]
  7. --
  8. deploy/1.8+/resource-reader.yaml:rules:
  9. deploy/1.8+/resource-reader.yaml-- apiGroups:
  10. deploy/1.8+/resource-reader.yaml-  - ""
  11. deploy/1.8+/resource-reader.yaml-  resources:
  12. deploy/1.8+/resource-reader.yaml-  - pods
  13. deploy/1.8+/resource-reader.yaml-  - nodes
  14. deploy/1.8+/resource-reader.yaml-  - nodes/stats
  15. deploy/1.8+/resource-reader.yaml-  - namespaces
  16. deploy/1.8+/resource-reader.yaml-  - configmaps
  17. deploy/1.8+/resource-reader.yaml-  verbs:
  18. deploy/1.8+/resource-reader.yaml-  - get
  19. deploy/1.8+/resource-reader.yaml-  - list
  20. deploy/1.8+/resource-reader.yaml-  - watch
  21. deploy/1.8+/resource-reader.yaml----
  22. deploy/1.8+/resource-reader.yaml-apiVersion: rbac.authorization.k8s.io/v1
  23. deploy/1.8+/resource-reader.yaml-kind: ClusterRoleBinding
  24. deploy/1.8+/resource-reader.yaml-metadata:
  25. deploy/1.8+/resource-reader.yaml-  name: system:metrics-server
  26. deploy/1.8+/resource-reader.yaml-roleRef:
  27. deploy/1.8+/resource-reader.yaml-  apiGroup: rbac.authorization.k8s.io
  28. deploy/1.8+/resource-reader.yaml-  kind: ClusterRole
  29. deploy/1.8+/resource-reader.yaml-  name: system:metrics-server
  31. user@ubuntu:~/metrics-server$

The aggregated-metrics-reader provides access to the pod information within the metrics.k8s.io API group. The
resource-reader provides access to information methods for pods, nodes, nodes/stats and namespaces across all API groups
and information access to deployments within the extensions API group.

The ClusterRoleBinding objects bind the ClusterRoles to the metrics-server ServiceAccount created in the kube-system
namespace. The one RoleBinding binds the preexisting extension-apiserver-authentication-reader Role to the Metric
Server’s service account:

  1. user@ubuntu:~/metrics-server$ cat deploy/1.8+/auth-reader.yaml
  1. ---
  2. apiVersion: rbac.authorization.k8s.io/v1
  3. kind: RoleBinding
  4. metadata:
  5.   name: metrics-server-auth-reader
  6.   namespace: kube-system
  7. roleRef:
  8.   apiGroup: rbac.authorization.k8s.io
  9.   kind: Role
  10.   name: extension-apiserver-authentication-reader
  11. subjects:
  12. - kind: ServiceAccount
  13.   name: metrics-server
  14.   namespace: kube-system
  1. user@ubuntu:~/metrics-server$ kubectl get role -n kube-system extension-apiserver-authentication-reader
  3. NAME                                        AGE
  4. extension-apiserver-authentication-reader   4h25m
  6. user@ubuntu:~/metrics-server$

The extension-apiserver-authentication-reader allows the addon API server to read the client CA file from the
extension-apiserver-authentication ConfigMap in the kube-system namespace. This is required in order for the Metrics
addon API to delegate authentication and authorization requests to the main Kubernetes API server.

The remaining three manifests support the Metrics Server operation:

  • metrics-apiservice.yaml:kind: APIService
  • metrics-server-deployment.yaml:kind: Deployment
  • metrics-server-service.yaml:kind: Service

The APIService manifest defines the metric-server service API extension to the core Kubernetes API:

  1. user@ubuntu:~/metrics-server$ cat deploy/1.8+/metrics-apiservice.yaml
  1. ---
  2. apiVersion: apiregistration.k8s.io/v1beta1
  3. kind: APIService
  4. metadata:
  5.   name: v1beta1.metrics.k8s.io
  6. spec:
  7.   service:
  8.     name: metrics-server
  9.     namespace: kube-system
  10.   group: metrics.k8s.io
  11.   version: v1beta1
  12.   insecureSkipTLSVerify: true
  13.   groupPriorityMinimum: 100
  14.   versionPriority: 100
  1. user@ubuntu:~/metrics-server$

Ad hoc API extensions make use of the Kubernetes API aggregation function. Kubernetes API aggregation enables the
dynamic installation of additional Kubernetes-style APIs in a cluster. These can either be pre-built, existing 3rd party
solutions, such as service-catalog, or user-created APIs like apiserver-builder.

The aggregation layer runs in-process with the kube-apiserver. Until an extension resource is registered, the
aggregation layer will do nothing. To register an API an APIService object is used to “claim” a URL path in the
Kubernetes API. At this point, the aggregation layer will proxy anything sent to that API path to the registered
APIService.

In this case the metrics-server claims the /apis/metrics.k8s.io/ path.

The metrics server deployment is fairly simple and just runs the k8s.gcr.io/metrics-server-amd64:v0.3.6 image. The
Metrics Server service is also simple, creating a “metrics-server” service that accepts traffic on port 443.

Examine any of the yaml manifests you find interesting.

2. Deploy the Metrics Server

N.B. If you have a multi-node cluster, make sure you deploy the metrics server to your master node. Use kubectl cordon <worker node> and remove the master taint if you have it with kubectl taint nodes $(hostname) node-role.kubernetes.io/master- to ensure this lab’s pod run on your master node.

We can launch the Metrics Server by creating all of the resources in the deploy/1.8+ directory by providing kubectl apply -f with the directory path. Try it:

  1. user@ubuntu:~/metrics-server$ kubectl apply -f deploy/1.8+/
  3. clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
  4. clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
  5. rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
  6. apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
  7. serviceaccount/metrics-server created
  8. deployment.apps/metrics-server created
  9. service/metrics-server created
  10. clusterrole.rbac.authorization.k8s.io/system:metrics-server created
  11. clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
  13. user@ubuntu:~/metrics-server$
  1. user@ubuntu:~/metrics-server$ kubectl get all -n kube-system -l k8s-app=metrics-server
  3. NAME                                  READY   STATUS    RESTARTS   AGE
  4. pod/metrics-server-694db48df9-lzm99   1/1     Running   0          43s
  6. NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
  7. deployment.apps/metrics-server   1/1     1            1           43s
  9. NAME                                        DESIRED   CURRENT   READY   AGE
  10. replicaset.apps/metrics-server-694db48df9   1         1         1       43s
  12. user@ubuntu:~/metrics-server$

Great, the Metrics Server is deployed and running! Now try a metrics api request:

  1. user@ubuntu:~/metrics-server$ curl -k https://localhost:6443/apis/metrics.k8s.io/
  1. {
  2.   "kind": "Status",
  3.   "apiVersion": "v1",
  4.   "metadata": {
  6.   },
  7.   "status": "Failure",
  8.   "message": "forbidden: User \"system:anonymous\" cannot get path \"/apis/metrics.k8s.io/\"",
  9.   "reason": "Forbidden",
  10.   "details": {
  12.   },
  13.   "code": 403
  14. }
  1. user@ubuntu:~/metrics-server$

We cannot connect to the apiserver without a user identity, let’s use an existing one:

  1. user@ubuntu:~/metrics-server$ kubectl get sa
  3. NAME      SECRETS   AGE
  4. default   1         24h
  6. user@ubuntu:~/metrics-server$

Get the token and certificate from the ServiceAccount’s token secret for use in your API requests.

This will require jq for JSON parsing on the command line, install it:

  1. user@ubuntu:~/metrics-server$ sudo apt-get install jq -y
  3. ...
  5. user@ubuntu:~/metrics-server$

Start by setting the SERVICE_ACCOUNT variable:

  1. user@ubuntu:~/metrics-server$ SERVICE_ACCOUNT=default
  3. user@ubuntu:~/metrics-server$

Get the ServiceAccount’s token Secret’s name:

  1. user@ubuntu:~/metrics-server$ SECRET=$(kubectl get serviceaccount ${SERVICE_ACCOUNT} -o json \
  2. | jq -Mr '.secrets[].name | select(contains("token"))') && echo $SECRET
  4. default-token-ksft2
  6. user@ubuntu:~/metrics-server$

Extract the Bearer token from the Secret and decode:

  1. user@ubuntu:~/metrics-server$ TOKEN=$(kubectl get secret ${SECRET} -o json \
  2. | jq -Mr '.data.token' | base64 -d) && echo $TOKEN
  4. eyJhbGciOiJSUzI1NiIsImtpZCI6IjRxQmtKN1g3ZlhWT1hFb0ItTVRYRVo5TGRZVzU0TGlwLXBtRWRVdWdhU0kifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4ta3NmdDIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImMwOTk2Y2ZmLTcwZDAtNGNlMy1iMTlmLWQ1ZTliMmQ2YWRmMCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.Fpnx58zm4By_IAjrU47um1AgxxjbK4cnbjZTgXlDc1Pbv-n9SUZs_lj0hZTLwXNyTx1IVfntinfSBY6g7i3MEsBcUs6Q6zCYAC_vRZKi6mkamqT_93cM8vSjrPqgZ9k8X9UOkYxD_kUbMLEWpzXjJiAenGuBMBnDplAK70qQUV2SO2JAf7GyOMF79KOqkiTaiNLNQmRrZoyOwmYCFWUTwgMrUkruT9gow2eyk7xhIYiUzfiG-C7v0QLXyoI_2pyNbYbULibhExbyg2S-dXfCfJmOuHNPt0G8hyXosHRQmVDI875rX-0SQJVQ01AOmPUxEaerc9S-ayL9ROPGlfA38g
  6. user@ubuntu:~/metrics-server$

Extract, decode and write the ca.crt to a temporary location:

  1. user@ubuntu:~/metrics-server$ kubectl get secret ${SECRET} -o json \
  2. | jq -Mr '.data["ca.crt"]' | base64 -d > /tmp/ca.crt
  4. user@ubuntu:~/metrics-server$

Now retry the metrics api request once more, using your new identity:

  1. user@ubuntu:~/metrics-server$ curl -k https://localhost:6443/apis/metrics.k8s.io/ \
  2. --header "Authorization: Bearer $TOKEN" --cacert /tmp/ca.crt
  1. {
  2.   "kind": "APIGroup",
  3.   "apiVersion": "v1",
  4.   "name": "metrics.k8s.io",
  5.   "versions": [
  6.     {
  7.       "groupVersion": "metrics.k8s.io/v1beta1",
  8.       "version": "v1beta1"
  9.     }
  10.   ],
  11.   "preferredVersion": {
  12.     "groupVersion": "metrics.k8s.io/v1beta1",
  13.     "version": "v1beta1"
  14.   }
  15. }
  1. user@ubuntu:~/metrics-server$

Great, the Metrics Server is running and we can query it!

3. Test the Metrics Server

The kubectl top command can be used to display metrics for pods or nodes. Try it:

  1. user@ubuntu:~/metrics-server$ kubectl top node
  3. error: metrics not available yet
  5. user@ubuntu:~/metrics-server$

Hmm, no metrics. Try checking the metrics server log to see if there are any issues indicated by the metrics server pod:

  1. user@ubuntu:~/metrics-server$ kubectl get pods -n kube-system -l k8s-app=metrics-server
  3. NAME                              READY   STATUS    RESTARTS   AGE
  4. metrics-server-694db48df9-lzm99   1/1     Running   0          78s
  6. user@ubuntu:~/metrics-server$ kubectl -n kube-system logs metrics-server-694db48df9-lzm99
  8. I0124 23:49:33.028127       1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
  9. I0124 23:49:33.303878       1 secure_serving.go:116] Serving securely on [::]:4443
  10. E0124 23:50:45.557373       1 manager.go:111] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:ubuntu: unable to fetch metrics from Kubelet ubuntu (ubuntu): Get https://ubuntu:10250/stats/summary?only_cpu_and_memory=true: dial tcp 66.96.162.149:10250: connect: connection refused
  11. E0124 23:51:19.105292       1 reststorage.go:135] unable to fetch node metrics for node "ubuntu": no metrics known for node
  12. E0124 23:51:45.520002       1 manager.go:111] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:ubuntu: unable to fetch metrics from Kubelet ubuntu (ubuntu): Get https://ubuntu:10250/stats/summary?only_cpu_and_memory=true: dial tcp 66.96.162.149:10250: connect: connection refused
  14. user@ubuntu:~/metrics-server$

One issue here is that the metrics server cannot communicate with any of the kubelets for this host. In the example
above, the hostname “ubuntu” is routing to an IP address that is rejecting the connection, meaning the DNS configuration
in this environment is not configurd correctly.

Even if it could reach the host, another problem is that the Metrics server requests its metrics from the Kubelets but
rejects the kubelet responses. This is because the Metrics Server attempts to use the api-server CA cert to verify the
Kubelet’s response. In the Kubeadm case today the Kubelet uses an api-server CA signed cert to reach out to the
api-server but when listening on its own API port (where the /metrics endpoint we want is located) is uses a self signed
certificate. This causes the Metrics Server to reject the TLS session upon connection. Existing workarounds will suggest
using --kubelet-insecure-tls to tell the Metrics Server to not check the Kubelet’s TLS cert. This approach is not
ideal for production deployments.

3a. Force the kubelet to request a new certificate from the cluster

To fix this you will need to regenerate the kubelet’s certificate using the cluster CA certificate. This is possible by
forcing the node kubelet to go through the TLS bootstrap process again to request a new serving certificate. The serving
certificate bears the cluster CA’s signature and also includes the hostname and IP addresses of the kubelet as subject
alternative names. Any services that need to verify the the kubelet’s identities will be able to do so using a new
serving certificate.

To begin, retrieve the bootstrap token from the cluster. This token is what your node’s kubelet used initially to
communicate with the API server. This bootstrap token is assigned to the system:bootstrappers group, which gives it all
the permissions it needs to connect to the API server and request a certificate. These tokens are backed by secrets in
the cluster.

Retrieve the bootstrap token from the cluster’s secrets:

  1. user@ubuntu:~/metrics-server$ kubectl get secrets -n kube-system | grep bootstrap
  3. bootstrap-signer-token-mnmtt                     kubernetes.io/service-account-token   3      24h
  4. bootstrap-token-pllvtu                           bootstrap.kubernetes.io/token         7      24h
  6. user@ubuntu:~/metrics-server$ export BOOTSTRAPSECRET=bootstrap-token-pllvtu
  8. user@ubuntu:~/metrics-server$ kubectl get secret -n kube-system $BOOTSTRAPSECRET -o yaml
  10. apiVersion: v1
  11. data:
  12.   auth-extra-groups: c3lzdGVtOmJvb3RzdHJhcHBlcnM6a3ViZWFkbTpkZWZhdWx0LW5vZGUtdG9rZW4=
  13.   expiration: MjAyMC0wMS0yNVQxNjoxMDoyMi0wODowMA==
  14.   token-id: YzJpaGo1
  15.   token-secret: bDZhOG5qOHJkeHBsZXpzOQ==
  16.   usage-bootstrap-authentication: dHJ1ZQ==
  17.   usage-bootstrap-signing: dHJ1ZQ==
  18. kind: Secret
  19. metadata:
  20.   creationTimestamp: "2020-01-25T00:10:22Z"
  21.   name: bootstrap-token-c2ihj5
  22.   namespace: kube-system
  23.   resourceVersion: "26242"
  24.   selfLink: /api/v1/namespaces/kube-system/secrets/bootstrap-token-c2ihj5
  25.   uid: 082db190-155c-4a3f-8d9d-e62a2080427d
  26. type: bootstrap.kubernetes.io/token
  28. user@ubuntu:~/metrics-server$

The token-id and token-secret are needed to present a complete bookstrap token.

Using -o jsonpath to extract specific values from the secret, retrieve the token-id and the token-secret. To
output the values in plain text, pipe the output into base64 --decode:

  1. user@ubuntu:~/metrics-server$ kubectl get secret -n kube-system $BOOTSTRAPSECRET -o jsonpath='{.data.token-id}' \
  2. | base64 --decode && echo
  4. c2ihj5
  6. user@ubuntu:~/metrics-server$ kubectl get secret -n kube-system $BOOTSTRAPSECRET -o jsonpath='{.data.token-secret}' \
  7. | base64 --decode && echo
  9. l6a8nj8rdxplezs9
  11. user@ubuntu:~/metrics-server$

The proper format for a token is token-id.token-secret. When properly formatted, the complete bootstrap token for this
cluster is: c2ihj5.l6a8nj8rdxplezs9

N.B. In Kubeadm clusters like ours, bootstrap tokens are only valid for 24 hours. If your bootstrap token is over 24 hours old, you can use kubeadm to create a new token with kubeadm token create. This command outputs a complete bootstrap token to the terminal and creates a corresponding secret in the cluster.

  1. user@ubuntu:~$ kubeadm token create
  3. ezgumy.l3p5h3hn4xsd9u2k
  5. user@ubuntu:~$

If this is how you retrieved the bootstrap token, make sure to use whatever output is put in the terminal for the following steps.

Now you need to create a bootstrap kubelet configuration. This configuration tells the Kubelet to re-register itself
with the cluster. First you need to see the cluster’s connection information by reading the current kubelet configurtion
at /etc/kubernetes/kubelet.conf:

  1. user@ubuntu:~/metrics-server$ sudo head /etc/kubernetes/kubelet.conf
  3. apiVersion: v1
  4. clusters:
  5. - cluster:
  6.     certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01ERXlNekl6TkRVeU5Wb1hEVE13TURFeU1ESXpORFV5TlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSjdJCmZNUGpQeUxTZWZldFQvZE9KYllaV1RKMWE0eEovSngzbFByV3ovcDFUcEVsUmZVemFwVFR5bUw2Tkk1MFd2VFAKM1I3ZXdsakc0WWxZWXRKN0tKRkJTR2N4UFBrdzdHV3AxSUR1YnRrRjhtYTVXU2NNNHplelI0K3FEbmYySzZUNwpVNldscWt0WEUzcDR2YXhBTERXNHNua0NPSlpZQXE1RlFWb1owQTR1VzU5bkY1ZFJ5RGNXMkw3dWRQYVJuQ3A2CmZIbmtPSUo4TUtsWjQ3UHJkWFhCWGU1U0lYZURVZGtJUkwrRXdRbmpaWjhUL05MSkYwbXpEeTVrYzd3SHIrMTEKaStCWDdPaDNUYzFtUDlVaVRha2V1cTA3M2JIN2c0TGFuR0ExRXkvK1VWZVIyTGg5V3grOUJJQjBSaFJQZ2FNdAozSldoL0I1ME9DU3FFVVMzZ204Q0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFIQVJPNkxwMEJ5NmpzTS9MWDRzaThITnJKTU8KdVlOejk5aTc0OXlHdEJOQjJHd1FJMHd5NzJLNGFvMWU2MG5XelhlYXcwQTJBWldNbTIwVzZjM1ltc1JpV1dCNAp2WDB2UTU0cGE5YzE5dmliUTk4NXYwNmNmNnFnYVRZQzNjWWtuTDVxSnZsN1RwdUpVQkQyUkhIVDFsS0lxSG5XCnpDVXhXZjdpVk55dmwxMFZmTlhFbUhYK0phN05jSTAraGR0ZHFqVFQxZ21MdmM4b0dUblFtN0tuM0d2My9aRlMKem1yK0JSQjNIYVg1ZUJoVHVkRFp6WDNQYTZtc28yYmtsVUR4Y2FWcHB1bmVmdWZXNllTWFVuNHc0aE5yMFhWYwo3Qll0bGVVMXVFeTF0UlBrdjFaNHpYclpXTDByOXpVdGJ4b0JONUZ1V1B1K1ZoRlNPd0V3Q25BVDVSYz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  7.     server: https://192.168.229.132:6443
  8.   name: default-cluster
  9. contexts:
  10. - context:
  11.     cluster: default-cluster
  12.     namespace: default
  14. user@ubuntu:~/metrics-server$

We will use the value of the server key under -cluster: to inform the new bootstrap config how to connect to the
cluster.

Now we can construct the bootstrap configuration using kubectl. First, add the cluster connection and certificate
authority information to the file using kubectl config set-cluster. Be sure to substitute the --server flag with the
value of server: from the kubelet.conf file:

  1. user@ubuntu:~/metrics-server$ kubectl config set-cluster bootstrap \
  2. --kubeconfig=bootstrap-kubelet.conf \
  3. --certificate-authority='/etc/kubernetes/pki/ca.crt' \
  4. --server='https://192.168.229.132:6443'
  7. Cluster "bootstrap" set.
  9. user@ubuntu:~/metrics-server$

Next, associate the bootstrap token you retrieved earlier to a user called kubelet-bootstrap. Be sure to substitute
the value of --token with the token you retrieved or created earlier:

  1. user@ubuntu:~/metrics-server$ kubectl config set-credentials kubelet-bootstrap \
  2. --kubeconfig=bootstrap-kubelet.conf \
  3. --token=c2ihj5.l6a8nj8rdxplezs9
  5. User "kubelet-bootstrap" set.
  7. user@ubuntu:~/metrics-server$

Now, set a context that associates the kubelet-bootstrap user to the cluster connection information:

  1. ubuntu@ubuntu:~$ kubectl config set-context bootstrap \
  2. --kubeconfig=bootstrap-kubelet.conf \
  3. --user=kubelet-bootstrap --cluster=bootstrap
  5. Context "bootstrap" created.
  7. user@ubuntu:~/metrics-server$

Finally, create the file by switching to the bootstrap context with kubectl config use-context:

  1. user@ubuntu:~/metrics-server$ kubectl config use-context bootstrap \
  2. --kubeconfig=bootstrap-kubelet.conf
  4. Switched to context "bootstrap".
  6. user@ubuntu:~/metrics-server$

The file is now available. Use cat to view the contents.

  1. user@ubuntu:~/metrics-server$ cat bootstrap-kubelet.conf
  3. apiVersion: v1
  4. clusters:
  5. - cluster:
  6.     certificate-authority: /etc/kubernetes/pki/ca.crt
  7.     server: https://192.168.229.132:6443
  8.   name: bootstrap
  9. contexts:
  10. - context:
  11.     cluster: bootstrap
  12.     user: kubelet-bootstrap
  13.   name: bootstrap
  14. current-context: bootstrap
  15. kind: Config
  16. preferences: {}
  17. users:
  18. - name: kubelet-bootstrap
  19.   user:
  20.     token: c2ihj5.l6a8nj8rdxplezs9
  22. user@ubuntu:~/metrics-server$

Copy bootstrap-kubelet.conf to /etc/kubernetes/:

  1. user@ubuntu:~/metrics-server$ sudo cp bootstrap-kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf
  3. user@ubuntu:~/metrics-server$ sudo cat /etc/kubernetes/bootstrap-kubelet.conf
  5. apiVersion: v1
  6. clusters:
  7. - cluster:
  8.     certificate-authority: /etc/kubernetes/pki/ca.crt
  9.     server: https://192.168.229.132:6443
  10.   name: bootstrap
  11. contexts:
  12. - context:
  13.     cluster: bootstrap
  14.     user: kubelet-bootstrap
  15.   name: bootstrap
  16. current-context: bootstrap
  17. kind: Config
  18. preferences: {}
  19. users:
  20. - name: kubelet-bootstrap
  21.   user:
  22.     token: c2ihj5.l6a8nj8rdxplezs9
  24. user@ubuntu:~/metrics-server$

To ensure the kubelet requests a new signed serving certificate, we need to adjust the kubelet arguments to use server
certificate rotation and force the cluster to distribute a new certificate to the kubelet. Add
--rotate-server-certificates to your kubelet. For kubeadm-bootstrapped clusters, you can add it to the
kubeadm-flags.env file:

  1. user@ubuntu:~/metrics-server$ sudo nano /var/lib/kubelet/kubeadm-flags.env
  3. user@ubuntu:~/metrics-server$ sudo cat /var/lib/kubelet/kubeadm-flags.env
  5. KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --rotate-server-certificates"
  7. user@ubuntu:~/metrics-server$

N.B. If your cluster does not have a kubeadm-flags.env file, then edit the kubelet systemd service file directly and place --rotate-server-certificates inline with the ExecStart line pointing to the kubelet binary.

Next, we need to force the kubelet to use the bootstrap configuration by removing the existing kubelet.conf file.
Renaming the old file will suffice:

  1. user@ubuntu:~/metrics-server$ sudo mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.old

Now restart the kubelet:

  1. user@ubuntu:~/metrics-server$ sudo systemctl restart kubelet
  3. user@ubuntu:~/metrics-server$ sudo systemctl status kubelet
  5. ● kubelet.service - kubelet: The Kubernetes Node Agent
  6.    Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  7.   Drop-In: /etc/systemd/system/kubelet.service.d
  8.            └─10-kubeadm.conf
  9.    Active: active (running) since Fri 2020-01-24 16:19:01 PST; 2s ago
  10.      Docs: https://kubernetes.io/docs/home/
  11.  Main PID: 125944 (kubelet)
  12.     Tasks: 15
  13.    Memory: 34.5M
  14.       CPU: 420ms
  15.    CGroup: /system.slice/kubelet.service
  16.            └─125944 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs
  18. Jan 24 16:19:02 ubuntu kubelet[125944]: I0124 16:19:02.434257  125944 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-
  19. Jan 24 16:19:02 ubuntu kubelet[125944]: I0124 16:19:02.434268  125944 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "weavedb" (UniqueName: "kubernetes.io/host-pat
  20. Jan 24 16:19:02 ubuntu kubelet[125944]: I0124 16:19:02.434282  125944 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "dbus" (UniqueName: "kubernetes.io/host-path/d
  21. Jan 24 16:19:02 ubuntu kubelet[125944]: I0124 16:19:02.434295  125944 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/co
  22. Jan 24 16:19:02 ubuntu kubelet[125944]: I0124 16:19:02.434308  125944 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/co
  23. Jan 24 16:19:02 ubuntu kubelet[125944]: I0124 16:19:02.434313  125944 reconciler.go:154] Reconciler: start to sync state
  25. ...
  27. q
  29. user@ubuntu:~/metrics-server$

The kubelet will restart and use the bootstrap configuration since you removed the kubelet.conf. The bootstrap
configuration file location is declared in the kubelet service’s --bootstrap-kubeconfig argument, and the permanent
kubeconfig file is declared using the --kubeconfig argument.

Check if a new certificate signing request for the kubelet is present. This is the kubelet’s request for a serving
certificate:

  1. user@ubuntu:~/metrics-server$ kubectl get csr
  3. NAME        AGE    REQUESTOR                 CONDITION
  4. csr-2wnsr   2m6s   system:bootstrap:c2ihj5   Approved,Issued
  5. csr-485fp   46s    system:node:ubuntu        Pending
  7. user@ubuntu:~/metrics-server$

Use kubectl certificate approve on the Pending node certificate:

  1. user@ubuntu:~/metrics-server$ kubectl certificate approve csr-485fp
  3. certificatesigningrequest.certificates.k8s.io/csr-485fp approved
  5. user@ubuntu:~/metrics-server$ kubectl get csr
  7. NAME        AGE     REQUESTOR                 CONDITION
  8. csr-2wnsr   2m42s   system:bootstrap:c2ihj5   Approved,Issued
  9. csr-485fp   82s     system:node:ubuntu        Approved,Issued
  11. user@ubuntu:~/metrics-server$

The kubelet certificate is now running with a certificate signed by the cluster CA, meaning the API server and other
services within the cluster will successfully validate any requests. This process needs to be performed for all
kubelets.

3b. Reconfigure the metrics server to use the internal IP addresses of the kubelets

Another issue we need to address is how the Metrics Server communicates with the kubelets. The default behavior of the metrics server is to try and connect to the kubelets via their hostnames. Display the local node
status and see how the node presents itself to the cluster:

  1. user@ubuntu:~/metrics-server$ kubectl describe $(kubectl get node -o name) | grep -A2 Addresses
  3. Addresses:
  4.   InternalIP:  192.168.229.132
  5.   Hostname:    ubuntu
  7. user@ubuntu:~/metrics-server$

Notice that the Node has a few IP address options and a hostname. Given that the metrics server runs in a Pod and may not have the
ability to look up the hostname of the Kubelet’s machine, so using the hostname (the metrics server default) is not
ideal.

We can fix this problem by editing the Metrics Server deployment to pass the following argument to the
Metrics Server:

  • --kubelet-preferred-address-types=InternalIP - This will tell the Metrics Server to try to communicate using
    the Kubelet’s InternalIP.

Display the deployments in the kube-system namespace:

  1. user@ubuntu:~/metrics-server$ kubectl get deploy -n kube-system
  3. NAME             READY   UP-TO-DATE   AVAILABLE   AGE
  4. coredns          2/2     2            2           24h
  5. metrics-server   1/1     1            1           32m
  7. user@ubuntu:~/metrics-server$

Make the changes to the Metrics Server deployment:

  1. user@ubuntu:~/metrics-server$ kubectl edit -n kube-system deploy metrics-server
  3. ...

In the editor add the args key and switches shown below:

  2. ...
  4. template:
  5.   metadata:
  6.     creationTimestamp: null
  7.     labels:
  8.       k8s-app: metrics-server
  9.     name: metrics-server
  10.   spec:
  11.     containers:
  12.     - args:
  13.       - --kubelet-preferred-address-types=InternalIP      
  14.       - --cert-dir=/tmp
  15.       - --secure-port=4443
  16.       image: k8s.gcr.io/metrics-server-amd64:v0.3.6
  17.       imagePullPolicy: Always
  18.       name: metrics-server
  20. ...

Save your changes and exit:

  1. ...
  3. deployment.apps/metrics-server edited
  5. user@ubuntu:~/metrics-server$

Checking the pods again, you will see that the metrics server pod has been recreated and shows an age in seconds:

  1. user@ubuntu:~/metrics-server$ kubectl get po -n kube-system
  3. NAME                             READY   STATUS    RESTARTS   AGE
  4. coredns-5644d7b6d9-gd8tv         1/1     Running   0          24h
  5. coredns-5644d7b6d9-p6rd6         1/1     Running   0          24h
  6. etcd-ubuntu                      1/1     Running   0          24h
  7. kube-apiserver-ubuntu            1/1     Running   0          24h
  8. kube-controller-manager-ubuntu   1/1     Running   0          24h
  9. kube-proxy-4c8tb                 1/1     Running   0          24h
  10. kube-scheduler-ubuntu            1/1     Running   0          24h
  11. metrics-server-c84799697-mhmwq   1/1     Running   0          36s
  12. weave-net-76twd                  2/2     Running   0          24h
  14. user@ubuntu:~/metrics-server$

It will take two or three minutes for the new Metrics Server to collect data but if you try a few times you should now
see that top returns actual metrics. Use the Linux watch (not to be confused with kubectl -watch) tool to continually
send a kubectl command. Use CTRL C once you see that the metrics are being reported:

  1. user@ubuntu:~/metrics-server$ watch kubectl top node
  3. ...
  5. Every 2.0s: kubectl top node                            Fri Jan 24 16:23:47 2020
  7. NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
  8. ubuntu   140m         7%     1995Mi          52%
  11. ^C
  13. user@ubuntu:~/metrics-server$

Try kubectl top on pods:

  1. user@ubuntu:~/metrics-server$ kubectl top pod -n kube-system
  3. NAME                             CPU(cores)   MEMORY(bytes)   
  4. coredns-5644d7b6d9-gd8tv         2m           15Mi            
  5. coredns-5644d7b6d9-p6rd6         2m           24Mi            
  6. etcd-ubuntu                      7m           59Mi            
  7. kube-apiserver-ubuntu            27m          301Mi           
  8. kube-controller-manager-ubuntu   6m           49Mi            
  9. kube-proxy-4c8tb                 1m           22Mi            
  10. kube-scheduler-ubuntu            1m           21Mi            
  11. metrics-server-c84799697-mhmwq   1m           12Mi            
  12. weave-net-76twd                  1m           94Mi
  14. user@ubuntu:~/metrics-server$

Awesome, metrics up and running.

4. Autoscaling

Kubernetes 1.7 and later use the metrics-server to provide metrics used by the autoscaler controller. The base metrics
used in most autoscaling scenarios are CPU and memory.

To test autoscaling we’ll need:

  • A target pod/container-image that we can load
  • A pod we can use to drive the target pod
  • A Horizontal Pod Autoscalor (HPA) to scale the target pod when it gets busy

To begin, create a simple php web server we can use as our target:

  1. user@ubuntu:~/metrics-server$ mkdir ~/hpa && cd ~/hpa
  3. user@ubuntu:~/hpa$ nano index.php && cat index.php
  1. <?php
  2.   $x = 0.0001;
  3.   for ($i = 0; $i <= 1000000; $i++) {
  4.     $x += sqrt($x);
  5.   }
  6.   echo "OK!";
  7. ?>
  1. user@ubuntu:~/hpa$

This app will simply compute a million square roots each time it is hit.

Now create a Dockerfile to create a container image for the PHP server:

  1. user@ubuntu:~/hpa$ nano Dockerfile && cat Dockerfile
  3. FROM php:5-apache
  4. ADD index.php /var/www/html/index.php
  5. RUN chmod a+rx /var/www/html/index.php
  7. user@ubuntu:~/hpa$

Build a container image for the PHP server:

  1. user@ubuntu:~/hpa$ docker image build -t target .
  3. Sending build context to Docker daemon  3.072kB
  4. Step 1/3 : FROM php:5-apache
  5. 5-apache: Pulling from library/php
  6. 5e6ec7f28fb7: Pull complete
  7. cf165947b5b7: Pull complete
  8. 7bd37682846d: Pull complete
  9. 99daf8e838e1: Pull complete
  10. ae320713efba: Pull complete
  11. ebcb99c48d8c: Pull complete
  12. 9867e71b4ab6: Pull complete
  13. 936eb418164a: Pull complete
  14. bc298e7adaf7: Pull complete
  15. ccd61b587bcd: Pull complete
  16. b2d4b347f67c: Pull complete
  17. 56e9dde34152: Pull complete
  18. 9ad99b17eb78: Pull complete
  19. Digest: sha256:0a40fd273961b99d8afe69a61a68c73c04bc0caa9de384d3b2dd9e7986eec86d
  20. Status: Downloaded newer image for php:5-apache
  21.  ---> 24c791995c1e
  22. Step 2/3 : ADD index.php /var/www/html/index.php
  23.  ---> 170b6d61f08f
  24. Step 3/3 : RUN chmod a+rx /var/www/html/index.php
  25.  ---> Running in 105f9f2ee939
  26. Removing intermediate container 105f9f2ee939
  27.  ---> ddafae5353b4
  28. Successfully built ddafae5353b4
  29. Successfully tagged target:latest
  31. user@ubuntu:~/hpa$

N.B. Remember that Docker build places the PHP image on the node that you built it on! If you are running a multi-node Kubernetes cluster, make sure that any pods using this image run on a node where the image is present.

Now we can launch a standard single replica deployment with our new server:

  1. user@ubuntu:~/hpa$ kubectl create deploy web1 --image=target --dry-run -o yaml > target-deploy.yaml
  3. user@ubuntu:~/hpa$ nano target-deploy.yaml
  5. user@ubuntu:~/hpa$ cat target-deploy.yaml
  7. apiVersion: apps/v1
  8. kind: Deployment
  9. metadata:
  10.   labels:
  11.     app: web1
  12.   name: web1
  13. spec:
  14.   replicas: 1
  15.   selector:
  16.     matchLabels:
  17.       app: web1
  18.   template:
  19.     metadata:
  20.       labels:
  21.         app: web1
  22.     spec:
  23.       nodeName: ubuntu
  24.       containers:
  25.       - image: target
  26.         imagePullPolicy: Never
  27.         name: target
  28.         resources:
  29.           requests:
  30.             cpu: "200m"
  32. user@ubuntu:~/hpa$ kubectl create -f target-deploy.yaml
  34. deployment.apps/web1 created
  36. user@ubuntu:~/hpa$ kubectl expose deploy web1 --port 80
  38. service/web1 exposed
  40. user@ubuntu:~/hpa$

Our pod will request 200 mils of CPU (1/5 of a CPU) and we also set imagePullPolicy to “Never” to
prevent the kubelet from asking Docker to download the image (which it would not find since we have just built it
locally).

N.B. A nodeName key keeps this deployment’s pods from running on a node that doesn’t have the PHP image you built. Make sure to set it to the hostname of the node that has the PHP image.

List your resources:

  1. user@ubuntu:~/hpa$ kubectl get deploy,rs,po,svc
  3. NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
  4. deployment.apps/web1   1/1     1            1           47s
  6. NAME                              DESIRED   CURRENT   READY   AGE
  7. replicaset.apps/web1-75c54b6579   1         1         1       47s
  9. NAME                        READY   STATUS    RESTARTS   AGE
  10. pod/web1-75c54b6579-lqf7z   1/1     Running   0          47s
  12. NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
  13. service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   24h
  14. service/web1         ClusterIP   10.103.152.45   <none>        80/TCP    44s
  16. user@ubuntu:~/hpa$

So we now have our deployment with a replica set, one pod and a service (created because of the --expose switch).

Now let’s create an HPA for our deployment:

  1. user@ubuntu:~/hpa$ kubectl autoscale deployment web1 --cpu-percent=50 --min=1 --max=5
  3. horizontalpodautoscaler.autoscaling/web1 autoscaled
  5. user@ubuntu:~/hpa$

Our HPA will now scale up when our set of pods exceed 50% of their desired CPU resources, in this case 100 mils. Right
now we have one pod with a request for 20% of a CPU, or 200 mils. Display the HPA. You may see the Targets for the pod
read for a few minutes until the autoscaler communicates with the metrics server:

  1. user@ubuntu:~/hpa$ kubectl get hpa
  3. NAME   REFERENCE         TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
  4. web1   Deployment/web1   0%/50%    1         5         1          63s
  6. user@ubuntu:~/hpa$

Now display the resources used by your pod:

  1. user@ubuntu:~/hpa$ kubectl top pod
  3. NAME                    CPU(cores)   MEMORY(bytes)   
  4. web1-75c54b6579-lqf7z   1m           8Mi     
  6. user@ubuntu:~/hpa$

Our pod is using 1 mil of its 200 mils. It will need to get to 100 mils before the HPA will add another pod to the
deployment. Let’s create some load!

Run an interactive busybox pod to hit the PHP server from:

  1. user@ubuntu:~/hpa$ kubectl run --generator=run-pod/v1 -it driver --image=busybox
  3. If you don't see a command prompt, try pressing enter.
  5. / #

Now start a loop that will make continuous requests of our 1mm square root service:

  1. / # while true; do wget -q -O- http://web1.default.svc.cluster.local; done
  2. OK!
  3. OK!
  4. OK!
  6. ...

While the requests are running, open a new terminal and check the pod utilization with top:

  1. user@ubuntu:~$ kubectl top pod
  3. NAME                    CPU(cores)   MEMORY(bytes)   
  4. web1-75c54b6579-lqf7z   1m           8Mi             
  6. user@ubuntu:~$ kubectl top pod
  8. NAME                    CPU(cores)   MEMORY(bytes)   
  9. driver                  7m           1Mi             
  10. web1-75c54b6579-lqf7z   928m         10Mi    
  12. user@ubuntu:~$

Now over 900 mils, that is well over the threshold of 100 mils, why don’t we have more pods reporting metrics? The HPA
will not respond to spikes because this would trash the cluster, creating and deleting pods wastefully. Rather the HPA
waits for a configurable period (defined by Controller Manager settings, defaulting to about 3 minutes) and then begins
to scale.

Run top several more times until you see scaling activity, display the HPA status now and then also:

  1. user@ubuntu:~$ kubectl top pod
  3. NAME                    CPU(cores)   MEMORY(bytes)   
  4. driver                  7m           1Mi             
  5. web1-75c54b6579-lqf7z   928m         10Mi       
  7. user@ubuntu:~$ kubectl get hpa
  9. NAME   REFERENCE         TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
  10. web1   Deployment/web1   83%/50%   1         5         5          3m11s
  12. user@ubuntu:~$ kubectl top pod
  14. NAME                    CPU(cores)   MEMORY(bytes)   
  15. driver                  10m          2Mi             
  16. web1-75c54b6579-gjss2   170m         10Mi            
  17. web1-75c54b6579-k5tqp   164m         10Mi            
  18. web1-75c54b6579-lqf7z   203m         10Mi            
  19. web1-75c54b6579-psm69   130m         10Mi            
  20. web1-75c54b6579-w6d4c   170m         10Mi     
  22. user@ubuntu:~$

You should see the HPA scale the pods out to the limit of 5 configured. Once those pods communicate with the metrics server, the start showing up on kubectl top output. Display your deployment:

  1. user@ubuntu:~$ kubectl get deploy
  3. NAME   READY   UP-TO-DATE   AVAILABLE   AGE
  4. web1   5/5     5            5           4m32s
  6. user@ubuntu:~$

Note that the only thing the HPA actually does is change the replica count on the deployment. The deployment and replica
set do the rest.

Use control C to stop the driver loop and exit the pod:

  2. ...
  4. OK!
  5. OK!
  7. ^C
  9. / # exit
  11. Session ended, resume using 'kubectl attach driver -c driver -i -t' command when the pod is running
  13. user@ubuntu:~/hpa$

Examine the metrics and HPA:

  1. user@ubuntu:~/hpa$ kubectl get hpa
  3. NAME   REFERENCE         TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
  4. web1   Deployment/web1   90%/50%   1         5         5          3m58s
  6. user@ubuntu:~/hpa$ kubectl top pod
  8. NAME                    CPU(cores)   MEMORY(bytes)   
  9. driver                  10m          2Mi             
  10. web1-75c54b6579-gjss2   122m         10Mi            
  11. web1-75c54b6579-k5tqp   182m         10Mi            
  12. web1-75c54b6579-lqf7z   202m         10Mi            
  13. web1-75c54b6579-psm69   211m         10Mi            
  14. web1-75c54b6579-w6d4c   192m         10Mi    
  16. user@ubuntu:~/hpa$

The HPA will wait to ensure traffic does not return for a few minutes before scaling down. While you
wait for the HPA to scale down, describe the resource:

  1. user@ubuntu:~/hpa$ kubectl describe hpa web1
  3. Name:                                                  web1
  4. Namespace:                                             default
  5. Labels:                                                <none>
  6. Annotations:                                           <none>
  7. CreationTimestamp:                                     Fri, 24 Jan 2020 16:28:17 -0800
  8. Reference:                                             Deployment/web1
  9. Metrics:                                               ( current / target )
  10.   resource cpu on pods  (as a percentage of request):  0% (1m) / 50%
  11. Min replicas:                                          1
  12. Max replicas:                                          5
  13. Deployment pods:                                       5 current / 5 desired
  14. Conditions:
  15.   Type            Status  Reason               Message
  16.   ----            ------  ------               -------
  17.   AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  18.   ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  19.   ScalingLimited  True    TooManyReplicas      the desired replica count is more than the maximum replica count
  20. Events:
  21.   Type     Reason                        Age    From                       Message
  22.   ----     ------                        ----   ----                       -------
  23.   Warning  FailedGetResourceMetric       4m28s  horizontal-pod-autoscaler  did not receive metrics for any ready pods
  24.   Warning  FailedComputeMetricsReplicas  4m28s  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: did not receive metrics for any ready pods
  25.   Normal   SuccessfulRescale             3m12s  horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  26.   Normal   SuccessfulRescale             2m57s  horizontal-pod-autoscaler  New size: 5; reason: cpu resource utilization (percentage of request) above target
  28. user@ubuntu:~/hpa$

Notice the scaling events are reported at the bottom of the description.

The HPA abides by a setting called horizontal-pod-autoscaler-downscale-stabilization which is passed as a flag to the
cluster’s kube-controller-manager. This argument prevents an HPA from scaling down for 5 minutes by default. After 5 minutes, the HPA
will scale the deployment back down to one pod.

Use watch to repeatedly run the kubectl get hpa and kubectl top pod commands:

  1. user@ubuntu:~/hpa$ watch 'kubectl get hpa && kubectl top pod'
  3. Every 2.0s: kubectl get hpa && kubectl top pod          Fri Jan 24 16:33:44 2020
  5. NAME   REFERENCE         TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
  6. web1   Deployment/web1   0%/50%    1         5         5          5m27s
  7. NAME                    CPU(cores)   MEMORY(bytes)
  8. driver                  0m           1Mi
  9. web1-75c54b6579-gjss2   1m           10Mi
  10. web1-75c54b6579-k5tqp   1m           10Mi
  11. web1-75c54b6579-lqf7z   1m           10Mi
  12. web1-75c54b6579-psm69   1m           10Mi
  13. web1-75c54b6579-w6d4c   1m           10Mi
  16. user@ubuntu:~/hpa$

After about five minutes, you will see the deployment scale down. Use Ctrl c to exit the watch when you see the scale
down event:

  1. Every 2.0s: kubectl get hpa && kubectl top pod          Fri Jan 24 16:37:47 2020
  3. NAME   REFERENCE         TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
  4. web1   Deployment/web1   0%/50%    1         5         5          9m30s
  5. NAME                    CPU(cores)   MEMORY(bytes)
  6. driver                  0m           1Mi
  7. web1-75c54b6579-lqf7z   1m           10Mi
  9. ^C
  11. user@ubuntu:~/hpa$

Describe the HPA once more and you will see a scaling event at the bottom that reports a new size and the reason:

  1. user@ubuntu:~/hpa$ kubectl describe hpa | grep Events -A10
  3. Events:
  4.   Type     Reason                        Age    From                       Message
  5.   ----     ------                        ----   ----                       -------
  6.   Warning  FailedGetResourceMetric       9m50s  horizontal-pod-autoscaler  did not receive metrics for any ready pods
  7.   Warning  FailedComputeMetricsReplicas  9m50s  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: did not receive metrics for any ready pods
  8.   Normal   SuccessfulRescale             8m34s  horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  9.   Normal   SuccessfulRescale             8m19s  horizontal-pod-autoscaler  New size: 5; reason: cpu resource utilization (percentage of request) above target
  10.   Normal   SuccessfulRescale             46s    horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
  12. user@ubuntu:~/hpa$

Your cluster is now capable of dynamically sizing its elements based on metrics measured from incoming workloads.

5. Clean up

List all of your resources:

  1. user@ubuntu:~/hpa$ kubectl get all -o=name
  3. pod/driver
  4. pod/web1-75c54b6579-lqf7z
  5. service/kubernetes
  6. service/web1
  7. deployment.apps/web1
  8. replicaset.apps/web1-75c54b6579
  9. horizontalpodautoscaler.autoscaling/web1
  11. user@ubuntu:~/hpa$

Now delete everything you created for this lab (but don’t delete the “kubernetes” service!!):

  1. user@ubuntu:~/hpa$ kubectl delete service/web1 deploy/web1 pod/driver hpa/web1
  3. service "web1" deleted
  4. deployment.apps "web1" deleted
  5. pod "driver" deleted
  6. horizontalpodautoscaler.autoscaling "web1" deleted
  8. user@ubuntu:~/hpa$

Congratulations, you have completed the lab!

Copyright (c) 2015-2020 RX-M LLC, Cloud Native Consulting, all rights reserved