启用 Thanos

Thanos

1. 部署 thanos-sidecar

修改 values.yaml：

## thanos 信息
thanos:
  image: quay.io/thanos/thanos:v0.17.2
  logLevel: debug
  objectStorageConfig:  # 后端对象存储服务配置
    key: thanos.yaml
    name: thanos-objstore-config

添加 objectStorageConfig 配置后，thanos-sidecar 会以 2h 为间隔，将 prometheus 产生的新数据块备份到对象存储中。
注意：开启该配置后，会禁用本地 prometheus 压缩，这意味着 thanos-compactor 是负责在全局对象存储级别上进行压缩的主要单例组件。

部署后结果：

thanos-sidecar:
    Args:
      sidecar
      --prometheus.url=http://127.0.0.1:9090/
      --tsdb.path=/prometheus
      --grpc-address=[$(POD_IP)]:10901
      --http-address=[$(POD_IP)]:10902
      --objstore.config=$(OBJSTORE_CONFIG)
      --log.level=debug
      --log.format=logfmt
    State:          Running
      Started:      Fri, 11 Dec 2020 16:03:00 +0800
    Ready:          True

2. 部署 thanos-ruler

thanos-ruler 可以跨多个 prometheus 实例，处理 recording/alerting 规则。
它需要至少一个 query endpoints，该 endpoint 指向 thanos-querier、prometheus。

...
apiVersion: monitoring.coreos.com/v1
kind: ThanosRuler
metadata:
  name: thanos-ruler-demo
  labels:
    example: thanos-ruler
  namespace: monitoring
spec:
  image: quay.io/thanos/thanos
  ruleSelector:  #select from prometheusrules(CRD)
    matchLabels:
      role: my-thanos-rules
  queryEndpoints:
    - dnssrv+_http._tcp.my-thanos-querier.monitoring.svc.cluster.local

3. 部署 thanos-query

deployment 部署 query，并创建 service, servicemonitor：

kubectl apply -f querier-deployment.yaml
kubectl apply -f querier-service.yaml
kubectl apply -f querier-servicemonitor.yaml

访问对应 svc 即可查看 thanos-query 统一控制台。

4. 部署 thanos-store

创建 cm 保存对象存储的详细配置，并以 statefulset 方式安装 store：

kubectl apply -f thanos-stroe.yaml

5. 部署 thanos-compact

①. thanos.yaml

type: s3
config:
  bucket: prom-remote-store
  endpoint: ${CEPH-RGW-SVC-IP}:${CEPH-RGW-SVC-PORT}

创建 secret，保存后端对象存储服务信息：

kubectl -n monitoring create secret generic thanos-objstore-config --from-file=thanos.yaml

②. thanos-store.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-store
  namespace: monitoring
  labels:
    app: thanos-store
    thanos-peer: "true"
spec:
  serviceName: "thanos-store"
  replicas: 1
  selector:
    matchLabels:
      app: thanos-store
      thanos-peer: "true"
  template:
    metadata:
      labels:
        app: thanos-store
        thanos-peer: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "10902"
    spec:
      containers:
        - name: thanos-store
          # Always use explicit image tags (release or master-<date>-sha) instead of ambigous `latest` or `master`.
          image: quay.io/thanos/thanos:v0.17.2
          args:
            - "store"
            - "--log.level=debug"
            - "--data-dir=/var/thanos/store"
            - "--objstore.config-file=/config/thanos-store.yml"
          ports:
            - name: http
              containerPort: 10902
            - name: grpc
              containerPort: 10901
            - name: cluster
              containerPort: 10900
          volumeMounts:
            - name: config
              mountPath: /config/
              readOnly: true
            - name: data
              mountPath: /var/thanos/store
      volumes:
        - name: data
          emptyDir: {}
        - name: config
          configMap:
            name: thanos-store-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-store-config
  namespace: monitoring
data:
  thanos-store.yml: |-
    type: S3
    config:
      bucket: prom-remote-store
      endpoint: "${CEPH-RGW-SVC-IP}:${CEPH-RGW-SVC-PORT}$():32276"
      region: ""
      access_key: "A33UKGTXRBGVT12343W0"
      insecure: false
      signature_version2: false
      secret_key: "zgH8BbFsoNyk6xJj6cd435Bbbu1uYhQ3HytvMLza"
      put_user_metadata: {}
      http_config:
        idle_conn_timeout: 1m30s
        response_header_timeout: 2m
        insecure_skip_verify: false
      trace:
        enable: false
      list_objects_version: ""
      part_size: 134217728
      sse_config:
        type: ""
        kms_key_id: ""
        kms_encryption_context: {}
        encryption_key: ""

④. thanos-query.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-query
  namespace: monitoring
  labels:
    app: thanos-query
    thanos-peers: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: thanos-query
      thanos-peers: "true"
  template:
    metadata:
      labels:
        app: thanos-query
        thanos-peers: "true"
    spec:
      containers:
        - name: thanos-query
          image: quay.io/thanos/thanos:v0.17.2
          args:
            - "query"
            - "--log.level=debug"
            - "--store=${PROM-SERVER-POD-IP}:10901"
            - "--store=${PROM-SERVER-POD-IP}:10901"
          ports:
            - name: http
              containerPort: 10902
            - name: grpc
              containerPort: 10901
            - name: cluster
              containerPort: 10900
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/path: /metrics
    prometheus.io/port: "10902"
    prometheus.io/scrape: "true"
  name: thanos-query
  labels:
    app: thanos-query
    release: prometheus-operator
    jobLabel: thanos
  namespace: monitoring
spec:
  selector:
    app: thanos-query
  ports:
    - port: 9090
      protocol: TCP
      targetPort: http
      name: http-query
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prom-thanos-query
  namespace: monitoring
  labels:
    release: prometheus
spec:
  jobLabel: jobLabel
  selector:
    matchLabels:
      app: thanos-query
  namespaceSelector:
    matchNames:
      - "monitoring"
  endpoints:
    - port: http-query
      interval: 15s