可乐去冰
文章 标签 分类 关于
可乐去冰

目录

kubernetes监控-Prometheus 监控GKE集群

 收录于 Kubernetes
   约 1211 字   预计阅读 3 分钟 

自建 Prometheus 监控 GKE 集群.

动机

谷歌 gke 集群不像国内云商用户体验这么好，监控也不好搞，而且他官方的文档全都是基于他自家云产品的用法，主要是用他家的产品价格也不便宜，于是我们用集群外自建 Prometheus 接入 gke 的方式，来实现监控告警。

要实现上述的方法监控 gke 集群，需要对 3 个地方的配置进行设置：

  • kube-state-metrics

  • cadvisor

  • node-exporter

sa token

创建 sa 账号，给予指定权限，我这里贪方便，用的 cluster-admin 的权限。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ksa-prometheus
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-admins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: ksa-prometheus
  namespace: default
EOF

kubernetes 新版本创建 serviceAccount 的时候已经不会自动生成 secret 了，如果确认需要 token ，需要我们自己手动创建 secret 让他自己填充。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

EOF
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: ksa-prometheus
  name: ksa-prometheus-token
  namespace: default
type: kubernetes.io/service-account-token
EOF

拿到 token 以后，后面的操作才能继续。

kube-state-metrics

kube-state-metrics 是获取 pod 相关指标的重要组件。

部署

git clone kube-state-metrics 的仓库下来，修改 examples/standard/service.yaml 为适合自己集群的负载方式，我这里用的 lb 。

1
2
3
4

git clone https://github.com/kubernetes/kube-state-metrics.git

cd kube-state-metrics
kubectl apply -f examples/standard

配置

Prometheus 配置。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

- job_name: 'kube-state-metrics'
  honor_timestamps: true
  scrape_interval: 60s
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: http
  static_configs:
    - targets: ['10.131.xxx.xxx:8080']
  metric_relabel_configs:
  - target_label: cluster
    replacement: gke

cadvisor

cadvisor 是获取容器相关指标的重要组成部分。

Prometheus 配置。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

- job_name: k8s-cadvisor
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: https://34.xxx.xxx.xxx
    role: node
    bearer_token_file: /etc/prometheus/gke.token
    tls_config:
      insecure_skip_verify: true
  bearer_token_file: /etc/prometheus/gke.token
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: ['__meta_kubernetes_node_label_kubernetes_io_hostname']
    target_label: node
  - separator: ;
    regex: __meta_kubernetes_node_label_(.+)
    replacement: $1
    action: labelmap
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: 34.xxx.xxx.xxx
    action: replace
  - source_labels: [__meta_kubernetes_node_name]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    action: replace
  metric_relabel_configs:
  - target_label: cluster
    replacement: gke

node-exporter

node-exporter 是获取宿主机监控指标的部分。

部署 node-exporter。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: gmp-public
  name: node-exporter
  labels:
    app.kubernetes.io/name: node-exporter
    app.kubernetes.io/version: 1.3.1
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: node-exporter
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 10%
  template:
    metadata:
      labels:
        app.kubernetes.io/name: node-exporter
        app.kubernetes.io/version: 1.3.1
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - arm64
                - amd64
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      containers:
      - name: node-exporter
        image: quay.io/prometheus/node-exporter:v1.3.1
        args:
        - --web.listen-address=:9100
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root
        - --no-collector.wifi
        - --no-collector.hwmon
        - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/pods/.+)($|/)
        - --collector.netclass.ignored-devices=^(veth.*)$
        - --collector.netdev.device-exclude=^(veth.*)$
        ports:
        - name: metrics
          containerPort: 9100
        resources:
          limits:
            memory: 180Mi
          requests:
            cpu: 102m
            memory: 180Mi
        volumeMounts:
        - mountPath: /host/sys
          mountPropagation: HostToContainer
          name: sys
          readOnly: true
        - mountPath: /host/root
          mountPropagation: HostToContainer
          name: root
          readOnly: true
      hostNetwork: true
      hostPID: true
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
      volumes:
      - hostPath:
          path: /sys
        name: sys
      - hostPath:
          path: /
        name: root

Prometheus 配置。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

- job_name: gke-node
  scheme: http
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /etc/prometheus/gke.token
  kubernetes_sd_configs:
  - role: node
    api_server: https://34.xxx.xxx.xxx
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/gke-mota.token
  relabel_configs:
    - source_labels: [__address__]
      regex: '(.*):10250'
      replacement: '${1}:9100'
      target_label: __address__
      action: replace
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
  metric_relabel_configs:
  - target_label: cluster
    replacement: gke

常用的告警规则

node

node 内存使用率大于90%

(node_memory_MemTotal_bytes{} - node_memory_MemAvailable_bytes{cluster="mota-gke"}) / node_memory_MemTotal_bytes{} * 100 > 90

node CPU使用率大于90%

sum by (cluster, instance) (avg by (cluster, mode, instance) (rate(node_cpu_seconds_total{mode!="idle"}[2m]))) * 100 > 90

node 每个CPU负载大于2

sum by (instance) (node_load1{}) / count by (instance) (node_cpu_seconds_total{mode="idle"}) > 2

node 磁盘使用率大于90%

(node_filesystem_size_bytes - (node_filesystem_avail_bytes{device=~"/dev/s.*"})) / node_filesystem_size_bytes * 100 > 90

node 磁盘 inode 使用率小于20%

node_filesystem_files_free{fstype!=""} / node_filesystem_files{fstype!=""} * 100 < 20

pod

pod 状态异常

min_over_time(sum by (cluster, namespace, pod, phase) (kube_pod_status_phase{phase=~"Pending|Unknown|Failed"})[5m:1m]) > 0

pod CPU使用率大于90%

(sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (cluster, pod, container) / sum(container_spec_cpu_quota{container!=""}/container_spec_cpu_period{container!=""}) by (cluster, pod, container) * 100) > 90

pod 内存使用率大于90%

(sum(container_memory_working_set_bytes{container!=""}) BY (cluster, container, pod) / sum(container_spec_memory_limit_bytes > 0) BY (cluster, container, pod) * 100) > 90

pod 出现OOM

((kube_pod_container_status_restarts_total{} - kube_pod_container_status_restarts_total{} offset 10m >= 1) and ignoring (reason) min_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m])) == 1

pod 频繁异常重启

delta(kube_pod_container_status_restarts_total{cluster="mota-gke"}[15m]) > 1

pod 长时间未就绪 min_over_time(sum by(namespace,host_ip,pod_ip,instance,pod,node)( kube_pod_info{} AND ON (pod, namespace) kube_pod_status_ready{condition!="true",pod!~"(kube-eventer-init|security-inspector).*"} == 1)[3m:1m])

deployment

Deployment 可用副本状态异常

kube_deployment_spec_replicas{} != kube_deployment_status_replicas_available{} != 0

总结

总的来说，没有 prometheus operator 方便，不过也难不倒我们，就多费一点心思而已，然后接下来就是配置监控大盘了。

后续有更新规则了再更新这里的告警规则，开箱即用，方便后续在其他地方使用，就无需每次都要去百度查了。

更新于 2024-06-06
阅读原始文档
 Kubernetes, Kubernetes监控, Prometheus
返回 | 主页