摘要：，負(fù)責(zé)抓取存儲(chǔ)指標(biāo)信息，并提供查詢功能，本文重點(diǎn)使用它的告警功能。，負(fù)責(zé)將告警通知給相關(guān)人員。配置的告警觸發(fā)規(guī)則使用超過(guò)最大上限的機(jī)時(shí)間超過(guò)秒分鐘分鐘時(shí)間在最近分鐘里超過(guò)配置連接，配置。

原文地址

在前一篇文章中提到了如何使用Prometheus+Grafana來(lái)監(jiān)控JVM。本文介紹如何使用Prometheus+Alertmanager來(lái)對(duì)JVM的某些情況作出告警。

本文所提到的腳本可以在這里下載。

用到的工具：

Docker，本文大量使用了Docker來(lái)啟動(dòng)各個(gè)應(yīng)用。

Prometheus，負(fù)責(zé)抓取/存儲(chǔ)指標(biāo)信息，并提供查詢功能，本文重點(diǎn)使用它的告警功能。

Grafana，負(fù)責(zé)數(shù)據(jù)可視化（本文重點(diǎn)不在于此，只是為了讓讀者能夠直觀地看到異常指標(biāo)）。

Alertmanager，負(fù)責(zé)將告警通知給相關(guān)人員。

JMX exporter，提供JMX中和JVM相關(guān)的metrics。

Tomcat，用來(lái)模擬一個(gè)Java應(yīng)用。

先講一下大致步驟：

利用JMX exporter，在Java進(jìn)程內(nèi)啟動(dòng)一個(gè)小型的Http server

配置Prometheus抓取那個(gè)Http server提供的metrics。

配置Prometheus的告警觸發(fā)規(guī)則

heap使用超過(guò)最大上限的50%、80%、90%

instance down機(jī)時(shí)間超過(guò)30秒、1分鐘、5分鐘

old gc時(shí)間在最近5分鐘里超過(guò)50%、80%

配置Grafana連接Prometheus，配置Dashboard。

配置Alertmanager的告警通知規(guī)則

告警的大致過(guò)程如下：

Prometheus根據(jù)告警觸發(fā)規(guī)則查看是否觸發(fā)告警，如果是，就將告警信息發(fā)送給Alertmanager。

Alertmanager收到告警信息后，決定是否發(fā)送通知，如果是，則決定發(fā)送給誰(shuí)。

1) 新建一個(gè)目錄，名字叫做prom-jvm-demo。

2) 下載JMX exporter到這個(gè)目錄。

3) 新建一個(gè)文件simple-config.yml內(nèi)容如下：

---
lowercaseOutputLabelNames: true
lowercaseOutputName: true
whitelistObjectNames: ["java.lang:type=OperatingSystem"]
rules:
 - pattern: "java.lang<>((?!process_cpu_time)w+):"
   name: os_$1
   type: GAUGE
   attrNameSnakeCase: true

4) 運(yùn)行以下命令啟動(dòng)3個(gè)Tomcat，記得把替換成正確的路徑（這里故意把-Xmx和-Xms設(shè)置的很小，以觸發(fā)告警條件）：

docker run -d 
  --name tomcat-1 
  -v :/jmx-exporter 
  -e CATALINA_OPTS="-Xms32m -Xmx32m -javaagent:/jmx-exporter/jmx_prometheus_javaagent-0.3.1.jar=6060:/jmx-exporter/simple-config.yml" 
  -p 6060:6060 
  -p 8080:8080 
  tomcat:8.5-alpine

docker run -d 
  --name tomcat-2 
  -v :/jmx-exporter 
  -e CATALINA_OPTS="-Xms32m -Xmx32m -javaagent:/jmx-exporter/jmx_prometheus_javaagent-0.3.1.jar=6060:/jmx-exporter/simple-config.yml" 
  -p 6061:6060 
  -p 8081:8080 
  tomcat:8.5-alpine

docker run -d 
  --name tomcat-3 
  -v :/jmx-exporter 
  -e CATALINA_OPTS="-Xms32m -Xmx32m -javaagent:/jmx-exporter/jmx_prometheus_javaagent-0.3.1.jar=6060:/jmx-exporter/simple-config.yml" 
  -p 6062:6060 
  -p 8082:8080 
  tomcat:8.5-alpine

5) 訪問(wèn)http://localhost:8080|8081|8082看看Tomcat是否啟動(dòng)成功。

6) 訪問(wèn)對(duì)應(yīng)的http://localhost:6060|6061|6062看看JMX exporter提供的metrics。

備注：這里提供的simple-config.yml僅僅提供了JVM的信息，更復(fù)雜的配置請(qǐng)參考JMX exporter文檔。

1) 在之前新建目錄prom-jvm-demo，新建一個(gè)文件prom-jmx.yml，內(nèi)容如下：

scrape_configs:
  - job_name: "java"
    static_configs:
    - targets:
      - ":6060"
      - ":6061"
      - ":6062"

# alertmanager的地址
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - ":9093"

# 讀取告警觸發(fā)條件規(guī)則
rule_files:
  - "/prometheus-config/prom-alert-rules.yml"

2) 新建文件prom-alert-rules.yml，該文件是告警觸發(fā)規(guī)則：

# severity按嚴(yán)重程度由高到低：red、orange、yello、blue
groups:
  - name: jvm-alerting
    rules:

    # down了超過(guò)30秒
    - alert: instance-down
      expr: up == 0
      for: 30s
      labels:
        severity: yellow
      annotations:
        summary: "Instance {{ $labels.instance }} down"
        description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 30 seconds."

    # down了超過(guò)1分鐘
    - alert: instance-down
      expr: up == 0
      for: 1m
      labels:
        severity: orange
      annotations:
        summary: "Instance {{ $labels.instance }} down"
        description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes."

    # down了超過(guò)5分鐘
    - alert: instance-down
      expr: up == 0
      for: 5m
      labels:
        severity: red
      annotations:
        summary: "Instance {{ $labels.instance }} down"
        description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."

    # 堆空間使用超過(guò)50%
    - alert: heap-usage-too-much
      expr: jvm_memory_bytes_used{job="java", area="heap"} / jvm_memory_bytes_max * 100 > 50
      for: 1m
      labels:
        severity: yellow
      annotations:
        summary: "JVM Instance {{ $labels.instance }} memory usage > 50%"
        description: "{{ $labels.instance }} of job {{ $labels.job }} has been in status [heap usage > 50%] for more than 1 minutes. current usage ({{ $value }}%)"

    # 堆空間使用超過(guò)80%
    - alert: heap-usage-too-much
      expr: jvm_memory_bytes_used{job="java", area="heap"} / jvm_memory_bytes_max * 100 > 80
      for: 1m
      labels:
        severity: orange
      annotations:
        summary: "JVM Instance {{ $labels.instance }} memory usage > 80%"
        description: "{{ $labels.instance }} of job {{ $labels.job }} has been in status [heap usage > 80%] for more than 1 minutes. current usage ({{ $value }}%)"
    
    # 堆空間使用超過(guò)90%
    - alert: heap-usage-too-much
      expr: jvm_memory_bytes_used{job="java", area="heap"} / jvm_memory_bytes_max * 100 > 90
      for: 1m
      labels:
        severity: red
      annotations:
        summary: "JVM Instance {{ $labels.instance }} memory usage > 90%"
        description: "{{ $labels.instance }} of job {{ $labels.job }} has been in status [heap usage > 90%] for more than 1 minutes. current usage ({{ $value }}%)"

    # 在5分鐘里，Old GC花費(fèi)時(shí)間超過(guò)30%
    - alert: old-gc-time-too-much
      expr: increase(jvm_gc_collection_seconds_sum{gc="PS MarkSweep"}[5m]) > 5 * 60 * 0.3
      for: 5m
      labels:
        severity: yellow
      annotations:
        summary: "JVM Instance {{ $labels.instance }} Old GC time > 30% running time"
        description: "{{ $labels.instance }} of job {{ $labels.job }} has been in status [Old GC time > 30% running time] for more than 5 minutes. current seconds ({{ $value }}%)"

    # 在5分鐘里，Old GC花費(fèi)時(shí)間超過(guò)50%        
    - alert: old-gc-time-too-much
      expr: increase(jvm_gc_collection_seconds_sum{gc="PS MarkSweep"}[5m]) > 5 * 60 * 0.5
      for: 5m
      labels:
        severity: orange
      annotations:
        summary: "JVM Instance {{ $labels.instance }} Old GC time > 50% running time"
        description: "{{ $labels.instance }} of job {{ $labels.job }} has been in status [Old GC time > 50% running time] for more than 5 minutes. current seconds ({{ $value }}%)"

    # 在5分鐘里，Old GC花費(fèi)時(shí)間超過(guò)80%
    - alert: old-gc-time-too-much
      expr: increase(jvm_gc_collection_seconds_sum{gc="PS MarkSweep"}[5m]) > 5 * 60 * 0.8
      for: 5m
      labels:
        severity: red
      annotations:
        summary: "JVM Instance {{ $labels.instance }} Old GC time > 80% running time"
        description: "{{ $labels.instance }} of job {{ $labels.job }} has been in status [Old GC time > 80% running time] for more than 5 minutes. current seconds ({{ $value }}%)"

3) 啟動(dòng)Prometheus：

docker run -d 
  --name=prometheus 
  -p 9090:9090 
  -v :/prometheus-config 
  prom/prometheus --config.file=/prometheus-config/prom-jmx.yml

4) 訪問(wèn)http://localhost:9090/alerts應(yīng)該能看到之前配置的告警規(guī)則：

如果沒(méi)有看到三個(gè)instance，那么等一會(huì)兒再試。

參考使用Prometheus+Grafana監(jiān)控JVM

1) 新建一個(gè)文件alertmanager-config.yml：

global:
  smtp_smarthost: ""
  smtp_from: ""
  smtp_auth_username: ""
  smtp_auth_password: ""

# The directory from which notification templates are read.
templates: 
- "/alertmanager-config/*.tmpl"

# The root route on which each incoming alert enters.
route:
  # The labels by which incoming alerts are grouped together. For example,
  # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  # be batched into a single group.
  group_by: ["alertname", "instance"]

  # When a new group of alerts is created by an incoming alert, wait at
  # least "group_wait" to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first 
  # notification.
  group_wait: 30s

  # When the first notification was sent, wait "group_interval" to send a batch
  # of new alerts that started firing for that group.
  group_interval: 5m

  # If an alert has successfully been sent, wait "repeat_interval" to
  # resend them.
  repeat_interval: 3h 

  # A default receiver
  receiver: "user-a"

# Inhibition rules allow to mute a set of alerts given that another alert is
# firing.
# We use this to mute any warning-level notifications if the same alert is 
# already critical.
inhibit_rules:
- source_match:
    severity: "red"
  target_match_re:
    severity: ^(blue|yellow|orange)$
  # Apply inhibition if the alertname and instance is the same.
  equal: ["alertname", "instance"]
- source_match:
    severity: "orange"
  target_match_re:
    severity: ^(blue|yellow)$
  # Apply inhibition if the alertname and instance is the same.
  equal: ["alertname", "instance"]
- source_match:
    severity: "yellow"
  target_match_re:
    severity: ^(blue)$
  # Apply inhibition if the alertname and instance is the same.
  equal: ["alertname", "instance"]

receivers:
- name: "user-a"
  email_configs:
  - to: ""

修改里面關(guān)于smtp_*的部分和最下面user-a的郵箱地址。

備注：因?yàn)閲?guó)內(nèi)郵箱幾乎都不支持TLS，而Alertmanager目前又不支持SSL，因此請(qǐng)使用Gmail或其他支持TLS的郵箱來(lái)發(fā)送告警郵件，見(jiàn)這個(gè)issue，這個(gè)問(wèn)題已經(jīng)修復(fù)，下面是阿里云企業(yè)郵箱的配置例子：

smtp_smarthost: "smtp.qiye.aliyun.com:465"
smtp_hello: "company.com"
smtp_from: "username@company.com"
smtp_auth_username: "username@company.com"
smtp_auth_password: password
smtp_require_tls: false

2) 新建文件alert-template.tmpl，這個(gè)是郵件內(nèi)容模板：

{{ define "email.default.html" }}
Summary
  
{{ .CommonAnnotations.summary }}

Description

{{ .CommonAnnotations.description }}
{{ end}}

3） 運(yùn)行下列命令啟動(dòng)：

docker run -d 
  --name=alertmanager 
  -v :/alertmanager-config 
  -p 9093:9093 
  prom/alertmanager:master --config.file=/alertmanager-config/alertmanager-config.yml

4) 訪問(wèn)http://localhost:9093，看看有沒(méi)有收到Prometheus發(fā)送過(guò)來(lái)的告警(如果沒(méi)有看到稍等一下)：

等待一會(huì)兒（最多5分鐘）看看是否收到郵件。如果沒(méi)有收到，檢查配置是否正確，或者docker logs alertmanager看看alertmanager的日志，一般來(lái)說(shuō)都是郵箱配置錯(cuò)誤導(dǎo)致。