k8s 1.8中完全移除heapster
Metrics-Server概念介绍
修改配置
安装metrics-server(v0.3.6)
- versionPriority: 100
- namespace: kube-system

k8s 1.8中完全移除heapster

从Heapster的GitHub地址：https://github.com/kubernetes-retired/heapster中可以看到，heapster 已经 RETIRED【过时】。从Kubernetes 1.12开始将从Kubernetes各种安装脚本中移除。Kubernetes推荐使用metrics-server。我们这里使用Helm来部署metrics-server。

Kubernetes 推荐使用 metrics-server。我们这里也使用helm来部署metrics-server。

:::warning 如果要部署prometheus，就不用部署了metrics了，否则冲突 :::

Metrics-Server概念介绍

Metrics-Server是集群核心监控数据的聚合器，在k8s早期版本中，对资源的监控使用的是heapster的资源监控工具。但是从 Kubernetes 1.8 开始，Kubernetes 通过 Metrics API 获取资源使用指标，例如容器 CPU 和内存使用情况。这些度量指标可以由用户直接访问，例如通过使用kubectl top 命令，或者使用集群中的控制器，,因为k8s的api-server将所有的数据持久化到了etcd中，显然k8s本身不能处理这种频率的采集，而且这种监控数据变化快且都是临时数据，因此需要有一个组件单独处理他们.
环境：K8s-v1.18、Docker-18.06.1-ce

修改配置

检查 API Server 是否开启了 Aggregator Routing：查看 API Server 是否具有 —enable-aggregator-routing=true 选项。

[root@k8s-master manifests]# ps -ef | grep apiserver
root      22008  21989  4 19:33 ?        00:06:37 kube-apiserver --advertise-address=192.168.181.142 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true  --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.1.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root      66924  57038  0 21:47 pts/0    00:00:00 grep --color=auto apiserver

修改每个 API Server 的 kube-apiserver.yaml 配置开启 Aggregator Routing：修改 manifests 配置后 API Server 会自动重启生效。

vim /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.10.253:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.10.253
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --enable-aggregator-routing=true #添加本行
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.1.0.0/16
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.18.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.10.253
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
status: {}

安装metrics-server(v0.3.6)

运行top命令

[root@adm-master ~]# kubectl top pods 
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

这是因为还缺少个组件，这个组件叫做metric-server。
这是一个K8S的资源监控工具，监控CPU和内存的使用情况。他是一款必须安装的组件，因为有其他许多组件依赖他，比如HPA,scheduler,kubectl top

下载yaml文件

wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml

GitHub地址：
https://github.com/kubernetes-sigs/metrics-server

修改compoents.yaml配置文件
其实，官网已经提示了，metric-server安装要满足2个条件

在master节点要能访问metrics server pod ip（kubeadm部署默认已经满足该条件，二进制部署需注意要在master节点也部署node组件）
apiserver启用聚合层支持（kubeadm默认已经启用，二进制部署需自己启用）yaml里增加两个参数：跳过证书验证和使用node ip通信
- —kubelet-insecure-tls
- —kubelet-preferred-address-types=InternalIP ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:aggregated-metrics-reader labels: rbac.authorization.k8s.io/aggregate-to-view: “true” rbac.authorization.k8s.io/aggregate-to-edit: “true” rbac.authorization.k8s.io/aggregate-to-admin: “true” rules:

apiGroups: [“metrics.k8s.io”] resources: [“pods”, “nodes”] verbs: [“get”, “list”, “watch”]

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: metrics-server:system:auth-delegator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects:

kind: ServiceAccount name: metrics-server namespace: kube-system

apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: metrics-server-auth-reader namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: extension-apiserver-authentication-reader subjects:

kind: ServiceAccount name: metrics-server namespace: kube-system

apiVersion: apiregistration.k8s.io/v1beta1 kind: APIService metadata: name: v1beta1.metrics.k8s.io spec: service: name: metrics-server namespace: kube-system group: metrics.k8s.io version: v1beta1 insecureSkipTLSVerify: true groupPriorityMinimum: 100

versionPriority: 100

apiVersion: v1 kind: ServiceAccount metadata: name: metrics-server

namespace: kube-system

apiVersion: apps/v1 kind: Deployment metadata: name: metrics-server namespace: kube-system labels: k8s-app: metrics-server spec: selector: matchLabels: k8s-app: metrics-server template: metadata: name: metrics-server labels: k8s-app: metrics-server spec: serviceAccountName: metrics-server volumes:

  # mount in tmp so we can safely use from-scratch images and/or read-only containers
  - name: tmp-dir
    emptyDir: {}
  containers:
  - name: metrics-server
    image: registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6 #修改为阿里云的镜像地址
    imagePullPolicy: IfNotPresent
    args:
      - --cert-dir=/tmp
      - --secure-port=4443
      - /metrics-server #新增
      - --kubelet-preferred-address-types=InternalIP #新增
      - --kubelet-insecure-tls #新增
    ports:
    - name: main-port
      containerPort: 4443
      protocol: TCP
    securityContext:
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
    volumeMounts:
    - name: tmp-dir
      mountPath: /tmp
  nodeSelector:
    kubernetes.io/os: linux
    kubernetes.io/arch: "amd64"

apiVersion: v1 kind: Service metadata: name: metrics-server namespace: kube-system labels: kubernetes.io/name: “Metrics-server” kubernetes.io/cluster-service: “true” spec: selector: k8s-app: metrics-server ports:

port: 443 protocol: TCP targetPort: main-port

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:metrics-server rules:

apiGroups:
- “” resources:
- pods
- nodes
- nodes/stats
- namespaces
- configmaps verbs:
- get
- list
- watch

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:metrics-server roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:metrics-server subjects:

kind: ServiceAccount name: metrics-server namespace: kube-system ```

安装

[root@k8s-master01 metrics-server]# kubectl apply -f components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

查看metrics-server服务状态

[root@k8s-master01 metrics-server]#  kubectl get pod -n kube-system | grep metrics-server
metrics-server-86f9d4c6-fn495              1/1     Running   0          6m28s

检查接口是否有异常

[root@k8s-master01 metrics-server]# kubectl describe apiservice v1beta1.metrics.k8s.io
Name:         v1beta1.metrics.k8s.io
Namespace:    
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"name":"v1beta1.metrics.k8s.io"},"spec":{"...
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2021-05-05T04:56:03Z
  Resource Version:    653577
  Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  UID:                 07761491-6436-49e8-8f5f-e4867199541d
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            metrics-server
    Namespace:       kube-system
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2021-05-05T04:56:06Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>

执行以下命令，检查节点占用性能情况。

[root@k8s-master01 metrics-server]# kubectl top nodes
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%     
k8s-node02     84m          4%     883Mi           46%         
k8s-master01   <unknown>                           <unknown>               <unknown>               <unknown>               
k8s-node01     <unknown>                           <unknown>               <unknown>               <unknown>               
k8s-node03     <unknown>                           <unknown>               <unknown>               <unknown>               
[root@k8s-master01 metrics-server]# kubectl top pods
NAME                             CPU(cores)   MEMORY(bytes)   
heapster-7dfc69b8d4-pxbqf        1m           17Mi            
ingress-nginx-controller-z46vb   2m           69Mi