K8S集群之-ETCD集群监控
### 生产ETCD集群监控
核心指标
- etcd服务存活状态
up{job=~"kubernetes-etcd.*"}==0
说明：up==0代表服务挂掉
-
etcd是否有脱离情况
etcd_server_has_leader{job=~"kubernetes-etcd.*"}==0说明：每个instance，该值应该都为1，否则这个节点可能已经离开集群，最好在发生过半这样的情况前介入
-
etcd改变次数
increase(etcd_server_leader_changes_seen_total{job=~"kubernetes-etcd.*"}[1h]) >3
说明：这个指标metrics类型为counter，即它是单调递增的，可以监控该值的变化率，如果发现变化率高，说明集群的负载过高或者网络连接可能不稳定
-
leader选举失败
rate(etcd_server_proposals_failed_total{job=~"kubernetes-etcd.*"}[15m])!=0说明：该值的类型也是counter。proposal字面意思是“提案”，客户端的一个写操作可以认为是一个提案，提案需要集群内的Etcd实例来“表决”，如果上述值不为零，说明有proposal没有提交成功，如果经常这样，说明集群leader选举失败或者集群有过半节点离线
-
http访问5分钟内失败百分比（待定）
sum by(method) (rate(etcd_http_failed_total{job=~"kubernetes-etcd.*"}[5m])) / sum by(method) (rate(etcd_http_received_total{job=~"kubernetes-etcd.*"}[5m]))> 0.05
- etcd集群切主次数
changes(etcd_server_leader_changes_seen_total{job=~".*"}[1d])>1
- WAL文件顺序写入的持久化时间
histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~".*"}[5m]))>0.5
说明：Etcd的持久化保证依赖WAL和快照机制，这些全靠硬盘的IO表现。如果硬盘的性能不佳，在高负载情况下，将严重拖慢Etcd的处理速度，因此在生产环境中建议使用SSD来替代传统机械硬盘。可以通过监控
etcd_disk_backend_commit_duration_seconds_bucket的0.99分位数来衡量硬盘的表现情况 如果该值仅几个毫秒，说明你的Etcd比较健康
-
磁盘使用率
(etcd_mvcc_db_total_size_in_bytes{}/etcd_server_quota_backend_bytes{}) * 100>80
prometheus的yaml配置
- job_name: 'kubernetes-etcd-19'scheme: httpstls_config:cert_file: /usr/local/prometheus/ssl/kube-etcd-19.pemkey_file: /usr/local/prometheus/ssl/kube-etcd-19-key.peminsecure_skip_verify: truescrape_interval: 120sstatic_configs:- targets: ['110.152.117.19:2379']- job_name: 'kubernetes-etcd-20'scheme: httpstls_config:cert_file: /usr/local/prometheus/ssl/kube-etcd-20.pemkey_file: /usr/local/prometheus/ssl/kube-etcd-20-key.peminsecure_skip_verify: truescrape_interval: 120sstatic_configs:- targets: ['110.152.117.20:2379']- job_name: 'kubernetes-etcd-21'scheme: httpstls_config:cert_file: /usr/local/prometheus/ssl/kube-etcd-21.pemkey_file: /usr/local/prometheus/ssl/kube-etcd-21-key.peminsecure_skip_verify: truescrape_interval: 120sstatic_configs:- targets: ['110.152.117.21:2379']
prometheus的rules配置文件
groups:
- name: 公共事业部ETCD集群监控 #project name取公司名称rules:- alert: "ETCD服务存活状态活监控"expr: up{job=~"kubernetes-etcd.*"}==0for: 30slabels:severity: "重要"team: ops-gt-monitoralert_type: "ETCD告警"alert_host: "{{ $labels.service }}"alert_value: "{{ $value }}"alert_subject: "ETCD告警"annotations:summary: "ETCD集群监控"description: "ETCD集群已经离开集群，(资源信息:{{ $labels.instance }})，请尽快处理！"- alert: "ETCD是否有脱离监控"expr: etcd_server_has_leader{job=~"kubernetes-etcd.*"}==0for: 30slabels:severity: "重要"team: ops-gt-monitoralert_type: "ETCD告警"alert_host: "{{ $labels.service }}"alert_value: "{{ $value }}"alert_subject: "ETCD告警"annotations:summary: "ETCD集群监控"description: "ETCD集群宕机或掉线，(资源信息:{{ $labels.instance }})，请尽快处理！"- alert: "ETCD改变次数监控"expr: increase(etcd_server_leader_changes_seen_total{job=~"kubernetes-etcd.*"}[1h]) >3for: 30slabels:severity: "重要"team: ops-gt-monitoralert_type: "ETCD告警"alert_host: "{{ $labels.service }}"alert_value: "{{ $value }}"alert_subject: "ETCD告警"annotations:summary: "ETCD集群监控"description: "ETCD集群负载过高或者网络连接不稳定，(资源信息:{{ $labels.instance }})，请尽快处理！"- alert: "ETCD选举监控"expr: rate(etcd_server_proposals_failed_total{job=~"kubernetes-etcd.*"}[15m])!=0for: 30slabels:severity: "重要"team: ops-gt-monitoralert_type: "ETCD告警"alert_host: "{{ $labels.service }}"alert_value: "{{ $value }}"alert_subject: "ETCD告警"annotations:summary: "ETCD集群监控"description: "ETCD集群leader选举失败{{ $value }}，(资源信息:{{ $labels.instance }})，请尽快处理！" - alert: "ETCD切主次数监控"expr: changes(etcd_server_leader_changes_seen_total{job=~".*"}[1d])>1for: 30slabels:severity: "重要"team: ops-gt-monitoralert_type: "ETCD告警"alert_host: "{{ $labels.service }}"alert_value: "{{ $value }}"alert_subject: "ETCD告警"annotations:summary: "ETCD集群监控"description: "ETCD集群切主次数{{ $value }}，(资源信息:{{ $labels.instance }})，请尽快处理！"- alert: "ETCD集群WAL写入时间"expr: histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket{job=~".*"}[5m]))>0.5for: 30slabels:severity: "重要"team: ops-gt-monitoralert_type: "ETCD告警"alert_host: "{{ $labels.service }}"alert_value: "{{ $value }}"alert_subject: "ETCD告警"annotations:summary: "ETCD集群监控"description: "ETCD集群WAL文件顺序写入的持久化时间{{ $value }}，(资源信息:{{ $labels.instance }})，请尽快处理！"- alert: "ETCD集群磁盘使用率"expr: (etcd_mvcc_db_total_size_in_bytes{}/etcd_server_quota_backend_bytes{}) * 100>80for: 30slabels:severity: "重要"team: ops-gt-monitoralert_type: "ETCD告警"alert_host: "{{ $labels.service }}"alert_value: "{{ $value }}"alert_subject: "ETCD告警"annotations:summary: "ETCD集群监控"description: "ETCD集群磁盘使用率{{ $value }}，(资源信息:{{ $labels.instance }})，请尽快处理！"