Alertmanager通过命令行标志和配置文件进行配置。 虽然命令行标志配置了不可变的系统参数，但配置文件定义了禁止规则，通知路由和通知接收器。

可视化编辑器可以帮助构建路由树。

要查看所有可用的命令行标志，请运行alertmanager -h。

Alertmanager可以在运行时重新加载其配置。 如果新配置格式不正确，则不会应用更改并记录错误。 通过向进程发送SIGHUP或向/-/reload端点发送HTTP POST请求来触发配置重新加载。

一、配置文件

指定要加载的配置文件，使用--config.file标志

  1. ./alertmanager --config.file=simple.yml

该文件以YAML格式写入，由下面描述的方案定义。括号表示参数是可选的。对于非列表参数，该值设置为指定的默认值。

通用占位符定义如下：

  • <duration>：与正则表达式匹配的持续时间[0-9]+(ms|[smhdwy])
  • <labelname>：与正则表达式匹配的字符串[a-zA-Z _][a-zA-Z0-9 _]*
  • <labelvalue>：一串unicode字符
  • <filepath>：当前工作目录中的有效路径
  • <boolean>：一个可以取值为true或false的布尔值
  • <string>：常规字符串
  • <secret>：一个秘密的常规字符串，例如密码
  • <tmpl_string>：在使用前进行模板扩展的字符串
  • <tmpl_secret>：在使用之前进行模板扩展的字符串，它是一个秘密 其他占位符是单独指定的。

可以在此处找到有效的示例文件。

全局配置指定在所有其他配置上下文中有效的参数。它们还可用作其他配置节的默认值。

  1. global:
  2.   # ResolveTimeout is the time after which an alert is declared resolved
  3.   # if it has not been updated.
  4.   [ resolve_timeout: <duration> | default = 5m ]
  6.   # The default SMTP From header field.
  7.   [ smtp_from: <tmpl_string> ]
  8.   # The default SMTP smarthost used for sending emails, including port number.
  9.   # Port number usually is 25, or 587 for SMTP over TLS (sometimes referred to as STARTTLS).
  10.   # Example: smtp.example.org:587
  11.   [ smtp_smarthost: <string> ]
  12.   # The default hostname to identify to the SMTP server.
  13.   [ smtp_hello: <string> | default = "localhost" ]
  14.   [ smtp_auth_username: <string> ]
  15.   # SMTP Auth using LOGIN and PLAIN.
  16.   [ smtp_auth_password: <secret> ]
  17.   # SMTP Auth using PLAIN.
  18.   [ smtp_auth_identity: <string> ]
  19.   # SMTP Auth using CRAM-MD5. 
  20.   [ smtp_auth_secret: <secret> ]
  21.   # The default SMTP TLS requirement.
  22.   [ smtp_require_tls: <bool> | default = true ]
  24.   # The API URL to use for Slack notifications.
  25.   [ slack_api_url: <secret> ]
  26.   [ victorops_api_key: <secret> ]
  27.   [ victorops_api_url: <string> | default = "https://alert.victorops.com/integrations/generic/20131114/alert/" ]
  28.   [ pagerduty_url: <string> | default = "https://events.pagerduty.com/v2/enqueue" ]
  29.   [ opsgenie_api_key: <secret> ]
  30.   [ opsgenie_api_url: <string> | default = "https://api.opsgenie.com/" ]
  31.   [ hipchat_api_url: <string> | default = "https://api.hipchat.com/" ]
  32.   [ hipchat_auth_token: <secret> ]
  33.   [ wechat_api_url: <string> | default = "https://qyapi.weixin.qq.com/cgi-bin/" ]
  34.   [ wechat_api_secret: <secret> ]
  35.   [ wechat_api_corp_id: <string> ]
  37.   # The default HTTP client configuration
  38.   [ http_config: <http_config> ]
  40. # Files from which custom notification template definitions are read.
  41. # The last component may use a wildcard matcher, e.g. 'templates/*.tmpl'.
  42. templates:
  43.   [ - <filepath> ... ]
  45. # The root node of the routing tree.
  46. route: <route>
  48. # A list of notification receivers.
  49. receivers:
  50.   - <receiver> ...
  52. # A list of inhibition rules.
  53. inhibit_rules:
  54.   [ - <inhibit_rule> ... ]
二、<route>

路由块定义路由树中的节点及其子节点。 如果未设置，则其可选配置参数将从其父节点继承。

每个警报都在配置的顶级路由中进入路由树，该路由必须匹配所有警报（即没有任何已配置的匹配器）。 然后它遍历子节点。 如果将continue设置为false，则在第一个匹配的子项后停止。 如果匹配节点上的continue为true，则警报将继续与后续兄弟节点匹配。 如果警报与节点的任何子节点都不匹配（没有匹配的子节点，或者不存在），则根据当前节点的配置参数处理警报。

  1. [ receiver: <string> ]
  2. # The labels by which incoming alerts are grouped together. For example,
  3. # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  4. # be batched into a single group.
  5. #
  6. # To aggregate by all possible labels use the special value '...' as the sole label name, for example:
  7. # group_by: ['...'] 
  8. # This effectively disables aggregation entirely, passing through all 
  9. # alerts as-is. This is unlikely to be what you want, unless you have 
  10. # a very low alert volume or your upstream notification system performs 
  11. # its own grouping.
  12. [ group_by: '[' <labelname>, ... ']' ]
  14. # Whether an alert should continue matching subsequent sibling nodes.
  15. [ continue: <boolean> | default = false ]
  17. # A set of equality matchers an alert has to fulfill to match the node.
  18. match:
  19.   [ <labelname>: <labelvalue>, ... ]
  21. # A set of regex-matchers an alert has to fulfill to match the node.
  22. match_re:
  23.   [ <labelname>: <regex>, ... ]
  25. # How long to initially wait to send a notification for a group
  26. # of alerts. Allows to wait for an inhibiting alert to arrive or collect
  27. # more initial alerts for the same group. (Usually ~0s to few minutes.)
  28. [ group_wait: <duration> | default = 30s ]
  30. # How long to wait before sending a notification about new alerts that
  31. # are added to a group of alerts for which an initial notification has
  32. # already been sent. (Usually ~5m or more.)
  33. [ group_interval: <duration> | default = 5m ]
  35. # How long to wait before sending a notification again if it has already
  36. # been sent successfully for an alert. (Usually ~3h or more).
  37. [ repeat_interval: <duration> | default = 4h ]
  39. # Zero or more child routes.
  40. routes:
  41.   [ - <route> ... ]

例子：

  1. # The root route with all parameters, which are inherited by the child
  2. # routes if they are not overwritten.
  3. route:
  4.   receiver: 'default-receiver'
  5.   group_wait: 30s
  6.   group_interval: 5m
  7.   repeat_interval: 4h
  8.   group_by: [cluster, alertname]
  9.   # All alerts that do not match the following child routes
  10.   # will remain at the root node and be dispatched to 'default-receiver'.
  11.   routes:
  12.   # All alerts with service=mysql or service=cassandra
  13.   # are dispatched to the database pager.
  14.   - receiver: 'database-pager'
  15.     group_wait: 10s
  16.     match_re:
  17.       service: mysql|cassandra
  18.   # All alerts with the team=frontend label match this sub-route.
  19.   # They are grouped by product and environment rather than cluster
  20.   # and alertname.
  21.   - receiver: 'frontend-pager'
  22.     group_by: [product, environment]
  23.     match:
  24.       team: frontend
三、<inhibit_rule>

当存在与另一组匹配器匹配的警报（源）时，禁止规则将匹配一组匹配器的警报（目标）静音。 目标和源警报必须具有相同列表中标签名称的equal标签值。

从语义上讲，缺少标签和具有空值的标签是equal的。 因此，如果源和目标警报中都缺少所有相同的标签名称，则禁用规则将适用。

为了防止警报抑制自身，禁止规则将永远不会禁止与规则的目标和源侧匹配的警报。 但是，我们建议以警报永远不会匹配双方的方式选择目标和源匹配器。 理由更容易，并且不会触发这种特殊情况。

  1. # Matchers that have to be fulfilled in the alerts to be muted.
  2. target_match:
  3.   [ <labelname>: <labelvalue>, ... ]
  4. target_match_re:
  5.   [ <labelname>: <regex>, ... ]
  7. # Matchers for which one or more alerts have to exist for the
  8. # inhibition to take effect.
  9. source_match:
  10.   [ <labelname>: <labelvalue>, ... ]
  11. source_match_re:
  12.   [ <labelname>: <regex>, ... ]
  14. # Labels that must have an equal value in the source and target
  15. # alert for the inhibition to take effect.
  16. [ equal: '[' <labelname>, ... ']' ]
四、<http_config>

http_config允许配置接收器用于与基于HTTP的API服务通信的HTTP客户端。

  1. # Note that `basic_auth`, `bearer_token` and `bearer_token_file` options are
  2. # mutually exclusive.
  4. # Sets the `Authorization` header with the configured username and password.
  5. # password and password_file are mutually exclusive.
  6. basic_auth:
  7.   [ username: <string> ]
  8.   [ password: <secret> ]
  9.   [ password_file: <string> ]
  11. # Sets the `Authorization` header with the configured bearer token.
  12. [ bearer_token: <secret> ]
  14. # Sets the `Authorization` header with the bearer token read from the configured file.
  15. [ bearer_token_file: <filepath> ]
  17. # Configures the TLS settings.
  18. tls_config:
  19.   [ <tls_config> ]
  21. # Optional proxy URL.
  22. [ proxy_url: <string> ]
五、<tls_config>

tls_config允许配置TLS连接。

  1. # CA certificate to validate the server certificate with.
  2. [ ca_file: <filepath> ]
  4. # Certificate and key files for client cert authentication to the server.
  5. [ cert_file: <filepath> ]
  6. [ key_file: <filepath> ]
  8. # ServerName extension to indicate the name of the server.
  9. # http://tools.ietf.org/html/rfc4366#section-3.1
  10. [ server_name: <string> ]
  12. # Disable validation of the server certificate.
  13. [ insecure_skip_verify: <boolean> | default = false]
六、<receiver>

Receiver是一个或多个通知集成的命名配置。

我们没有主动添加新的接收器，我们建议通过webhook接收器实现自定义通知集成。

  1. # The unique name of the receiver.
  2. name: <string>
  4. # Configurations for several notification integrations.
  5. email_configs:
  6.   [ - <email_config>, ... ]
  7. hipchat_configs:
  8.   [ - <hipchat_config>, ... ]
  9. pagerduty_configs:
  10.   [ - <pagerduty_config>, ... ]
  11. pushover_configs:
  12.   [ - <pushover_config>, ... ]
  13. slack_configs:
  14.   [ - <slack_config>, ... ]
  15. opsgenie_configs:
  16.   [ - <opsgenie_config>, ... ]
  17. webhook_configs:
  18.   [ - <webhook_config>, ... ]
  19. victorops_configs:
  20.   [ - <victorops_config>, ... ]
  21. wechat_configs:
  22.   [ - <wechat_config>, ... ]
七、<email_config>
  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = false ]
  4. # The email address to send notifications to.
  5. to: <tmpl_string>
  7. # The sender address.
  8. [ from: <tmpl_string> | default = global.smtp_from ]
  10. # The SMTP host through which emails are sent.
  11. [ smarthost: <string> | default = global.smtp_smarthost ]
  13. # The hostname to identify to the SMTP server.
  14. [ hello: <string> | default = global.smtp_hello ]
  16. # SMTP authentication information.
  17. [ auth_username: <string> | default = global.smtp_auth_username ]
  18. [ auth_password: <secret> | default = global.smtp_auth_password ]
  19. [ auth_secret: <secret> | default = global.smtp_auth_secret ]
  20. [ auth_identity: <string> | default = global.smtp_auth_identity ]
  22. # The SMTP TLS requirement.
  23. [ require_tls: <bool> | default = global.smtp_require_tls ]
  25. # TLS configuration.
  26. tls_config:
  27.   [ <tls_config> ]
  29. # The HTML body of the email notification.
  30. [ html: <tmpl_string> | default = '{{ template "email.default.html" . }}' ]
  31. # The text body of the email notification.
  32. [ text: <tmpl_string> ]
  34. # Further headers email header key/value pairs. Overrides any headers
  35. # previously set by the notification implementation.
  36. [ headers: { <string>: <tmpl_string>, ... } ]
八、<hipchat_config>

HipChat通知使用Build Your Own集成。

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = false ]
  4. # The HipChat Room ID.
  5. room_id: <tmpl_string>
  6. # The auth token.
  7. [ auth_token: <secret> | default = global.hipchat_auth_token ]
  8. # The URL to send API requests to.
  9. [ api_url: <string> | default = global.hipchat_api_url ]
  11. # See https://www.hipchat.com/docs/apiv2/method/send_room_notification
  12. # A label to be shown in addition to the sender's name.
  13. [ from:  <tmpl_string> | default = '{{ template "hipchat.default.from" . }}' ]
  14. # The message body.
  15. [ message:  <tmpl_string> | default = '{{ template "hipchat.default.message" . }}' ]
  16. # Whether this message should trigger a user notification.
  17. [ notify:  <boolean> | default = false ]
  18. # Determines how the message is treated by the alertmanager and rendered inside HipChat. Valid values are 'text' and 'html'.
  19. [ message_format:  <string> | default = 'text' ]
  20. # Background color for message.
  21. [ color:  <tmpl_string> | default = '{{ if eq .Status "firing" }}red{{ else }}green{{ end }}' ]
  23. # The HTTP client's configuration.
  24. [ http_config: <http_config> | default = global.http_config ]
九、<pagerduty_config>

PagerDuty通知通过PagerDuty API发送。 PagerDuty提供了有关如何在此集成的文档。

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The following two options are mutually exclusive.
  5. # The PagerDuty integration key (when using PagerDuty integration type `Events API v2`).
  6. routing_key: <tmpl_secret>
  7. # The PagerDuty integration key (when using PagerDuty integration type `Prometheus`).
  8. service_key: <tmpl_secret>
  10. # The URL to send API requests to
  11. [ url: <string> | default = global.pagerduty_url ]
  13. # The client identification of the Alertmanager.
  14. [ client:  <tmpl_string> | default = '{{ template "pagerduty.default.client" . }}' ]
  15. # A backlink to the sender of the notification.
  16. [ client_url:  <tmpl_string> | default = '{{ template "pagerduty.default.clientURL" . }}' ]
  18. # A description of the incident.
  19. [ description: <tmpl_string> | default = '{{ template "pagerduty.default.description" .}}' ]
  21. # Severity of the incident.
  22. [ severity: <tmpl_string> | default = 'error' ]
  24. # A set of arbitrary key/value pairs that provide further detail
  25. # about the incident.
  26. [ details: { <string>: <tmpl_string>, ... } | default = {
  27.   firing:       '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
  28.   resolved:     '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
  29.   num_firing:   '{{ .Alerts.Firing | len }}'
  30.   num_resolved: '{{ .Alerts.Resolved | len }}'
  31. } ]
  33. # Images to attach to the incident.
  34. images:
  35.   [ <image_config> ... ]
  37. # Links to attach to the incident.
  38. links:
  39.   [ <link_config> ... ]
  41. # The HTTP client's configuration.
  42. [ http_config: <http_config> | default = global.http_config ]
9.1 <image_config>

这些字段记录在PagerDuty API文档中。

  1. source: <tmpl_string>
  2. alt: <tmpl_string>
  3. text: <tmpl_string>
9.2 <link_config>

这些字段记录在PagerDuty API文档中。

  1. href: <tmpl_string>
  2. text: <tmpl_string>
十、<pushover_config>

推送通知通过Pushover API发送。

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The recipient user’s user key.
  5. user_key: <secret>
  7. # Your registered application’s API token, see https://pushover.net/apps
  8. token: <secret>
  10. # Notification title.
  11. [ title: <tmpl_string> | default = '{{ template "pushover.default.title" . }}' ]
  13. # Notification message.
  14. [ message: <tmpl_string> | default = '{{ template "pushover.default.message" . }}' ]
  16. # A supplementary URL shown alongside the message.
  17. [ url: <tmpl_string> | default = '{{ template "pushover.default.url" . }}' ]
  19. # Priority, see https://pushover.net/api#priority
  20. [ priority: <tmpl_string> | default = '{{ if eq .Status "firing" }}2{{ else }}0{{ end }}' ]
  22. # How often the Pushover servers will send the same notification to the user.
  23. # Must be at least 30 seconds.
  24. [ retry: <duration> | default = 1m ]
  26. # How long your notification will continue to be retried for, unless the user
  27. # acknowledges the notification.
  28. [ expire: <duration> | default = 1h ]
  30. # The HTTP client's configuration.
  31. [ http_config: <http_config> | default = global.http_config ]
十一、<slack_config>

Slack通知通过Slack webhooks发送。 通知包含附件。

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = false ]
  4. # The Slack webhook URL.
  5. [ api_url: <secret> | default = global.slack_api_url ]
  7. # The channel or user to send notifications to.
  8. channel: <tmpl_string>
  10. # API request data as defined by the Slack webhook API.
  11. [ icon_emoji: <tmpl_string> ]
  12. [ icon_url: <tmpl_string> ]
  13. [ link_names: <boolean> | default = false ]
  14. [ username: <tmpl_string> | default = '{{ template "slack.default.username" . }}' ]
  15. # The following parameters define the attachment.
  16. actions:
  17.   [ <action_config> ... ]
  18. [ callback_id: <tmpl_string> | default = '{{ template "slack.default.callbackid" . }}' ]
  19. [ color: <tmpl_string> | default = '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' ]
  20. [ fallback: <tmpl_string> | default = '{{ template "slack.default.fallback" . }}' ]
  21. fields:
  22.   [ <field_config> ... ]
  23. [ footer: <tmpl_string> | default = '{{ template "slack.default.footer" . }}' ]
  24. [ pretext: <tmpl_string> | default = '{{ template "slack.default.pretext" . }}' ]
  25. [ short_fields: <boolean> | default = false ]
  26. [ text: <tmpl_string> | default = '{{ template "slack.default.text" . }}' ]
  27. [ title: <tmpl_string> | default = '{{ template "slack.default.title" . }}' ]
  28. [ title_link: <tmpl_string> | default = '{{ template "slack.default.titlelink" . }}' ]
  29. [ image_url: <tmpl_string> ]
  30. [ thumb_url: <tmpl_string> ]
  32. # The HTTP client's configuration.
  33. [ http_config: <http_config> | default = global.http_config ]
11.1 <action_config>

这些字段记录在Slack API文档中。

  1. type: <tmpl_string>
  2. text: <tmpl_string>
  3. url: <tmpl_string>
  4. [ style: <tmpl_string> [ default = '' ]
11.2 <field_config>

这些字段记录在Slack API文档中。

  1. title: <tmpl_string>
  2. value: <tmpl_string>
  3. [ short: <boolean> | default = slack_config.short_fields ]
十二、<opsgenie_config>

OpsGenie通知通过OpsGenie API发送。

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The API key to use when talking to the OpsGenie API.
  5. [ api_key: <secret> | default = global.opsgenie_api_key ]
  7. # The host to send OpsGenie API requests to.
  8. [ api_url: <string> | default = global.opsgenie_api_url ]
  10. # Alert text limited to 130 characters.
  11. [ message: <tmpl_string> ]
  13. # A description of the incident.
  14. [ description: <tmpl_string> | default = '{{ template "opsgenie.default.description" . }}' ]
  16. # A backlink to the sender of the notification.
  17. [ source: <tmpl_string> | default = '{{ template "opsgenie.default.source" . }}' ]
  19. # A set of arbitrary key/value pairs that provide further detail
  20. # about the incident.
  21. [ details: { <string>: <tmpl_string>, ... } ]
  23. # Comma separated list of team responsible for notifications.
  24. [ teams: <tmpl_string> ]
  26. # Comma separated list of tags attached to the notifications.
  27. [ tags: <tmpl_string> ]
  29. # Additional alert note.
  30. [ note: <tmpl_string> ]
  32. # Priority level of alert. Possible values are P1, P2, P3, P4, and P5.
  33. [ priority: <tmpl_string> ]
  35. # The HTTP client's configuration.
  36. [ http_config: <http_config> | default = global.http_config ]
十三、<victorcops_config>

VictorOps通知通过VictorOps API发送出去

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The API key to use when talking to the VictorOps API.
  5. [ api_key: <secret> | default = global.victorops_api_key ]
  7. # The VictorOps API URL.
  8. [ api_url: <string> | default = global.victorops_api_url ]
  10. # A key used to map the alert to a team.
  11. routing_key: <tmpl_string>
  13. # Describes the behavior of the alert (CRITICAL, WARNING, INFO).
  14. [ message_type: <tmpl_string> | default = 'CRITICAL' ]
  16. # Contains summary of the alerted problem.
  17. [ entity_display_name: <tmpl_string> | default = '{{ template "victorops.default.entity_display_name" . }}' ]
  19. # Contains long explanation of the alerted problem.
  20. [ state_message: <tmpl_string> | default = '{{ template "victorops.default.state_message" . }}' ]
  22. # The monitoring tool the state message is from.
  23. [ monitoring_tool: <tmpl_string> | default = '{{ template "victorops.default.monitoring_tool" . }}' ]
  25. # The HTTP client's configuration.
  26. [ http_config: <http_config> | default = global.http_config ]
十四、<webhook_config>

webhook接收器允许配置通用接收器。

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = true ]
  4. # The endpoint to send HTTP POST requests to.
  5. url: <string>
  7. # The HTTP client's configuration.
  8. [ http_config: <http_config> | default = global.http_config ]

Alertmanager将以下列JSON格式将HTTP POST请求发送到配置的端点：

  1. {
  2.   "version": "4",
  3.   "groupKey": <string>,    // key identifying the group of alerts (e.g. to deduplicate)
  4.   "status": "<resolved|firing>",
  5.   "receiver": <string>,
  6.   "groupLabels": <object>,
  7.   "commonLabels": <object>,
  8.   "commonAnnotations": <object>,
  9.   "externalURL": <string>,  // backlink to the Alertmanager.
  10.   "alerts": [
  11.     {
  12.       "status": "<resolved|firing>",
  13.       "labels": <object>,
  14.       "annotations": <object>,
  15.       "startsAt": "<rfc3339>",
  16.       "endsAt": "<rfc3339>",
  17.       "generatorURL": <string> // identifies the entity that caused the alert
  18.     },
  19.     ...
  20.   ]
  21. }

有一个与此功能集成的列表。

十五、<wechat_config>

微信通知通过微信API发送。

  1. # Whether or not to notify about resolved alerts.
  2. [ send_resolved: <boolean> | default = false ]
  4. # The API key to use when talking to the WeChat API.
  5. [ api_secret: <secret> | default = global.wechat_api_secret ]
  7. # The WeChat API URL.
  8. [ api_url: <string> | default = global.wechat_api_url ]
  10. # The corp id for authentication.
  11. [ corp_id: <string> | default = global.wechat_api_corp_id ]
  13. # API request data as defined by the WeChat API.
  14. [ message: <tmpl_string> | default = '{{ template "wechat.default.message" . }}' ]
  15. [ agent_id: <string> | default = '{{ template "wechat.default.agent_id" . }}' ]
  16. [ to_user: <string> | default = '{{ template "wechat.default.to_user" . }}' ]
  17. [ to_party: <string> | default = '{{ template "wechat.default.to_party" . }}' ]
  18. [ to_tag: <string> | default = '{{ template "wechat.default.to_tag" . }}' ]``