Reading time: ~12 minutes Audience: Homelabbers who want alerts when services fail

What Is Prometheus Alertmanager?

Overview

Prometheus Alertmanager handles alerts sent by client applications such as Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, Slack, Telegram, PagerDuty, or webhook. It also handles silencing and inhibition of alerts.

Key concepts: - Alerting rules: Prometheus evaluates expressions and fires alerts - Alertmanager: Receives fired alerts, groups them, and routes to receivers - Grouping: Multiple related alerts are sent as one notification - Inhibition: If a critical alert is firing, suppress lower-priority alerts for the same service - Silencing: Mute alerts for a maintenance window

Why It Matters

A dashboard that shows a red dot is useless if you are asleep. Alertmanager turns metrics into actionable notifications. When your Nextcloud container exits, when your NAS disk hits 90%, or when your Proxmox node goes down, you need to know immediately.

Prerequisites

Hardware Requirements

Component Minimum Recommended
CPU 1 core 2 cores
RAM 512 MB 1 GB
Storage 1 GB 5 GB (for alert history)
Network 1 GbE 1 GbE

Software Requirements

  • Docker and Docker Compose installed
  • Prometheus already running (or deploy together)
  • A notification target: Slack webhook, Telegram bot token, or SMTP server

Knowledge Prerequisites

  • Basic YAML syntax
  • Understanding of Prometheus scrape configs and recording rules
  • Familiarity with your notification platform (Slack, Telegram, etc.)

Step 1: Deploy Alertmanager with Docker Compose

Objective

Run Alertmanager as a container with a persistent configuration volume.

Step-by-Step Instructions

  1. Create a directory for Alertmanager:
mkdir -p ~/alertmanager/config
cd ~/alertmanager
  1. Create config/alertmanager.yml:
global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'your-app-password'
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
  telegram_api_url: 'https://api.telegram.org'
  resolve_timeout: 5m

route:
  receiver: 'default'
  group_by: ['alertname', 'job', 'severity']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  routes:
    - match:
        severity: critical
      receiver: 'critical-alerts'
      continue: true
    - match:
        severity: warning
      receiver: 'warning-alerts'

receivers:
  - name: 'default'
    slack_configs:
      - channel: '#homelab-alerts'
        send_resolved: true
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

  - name: 'critical-alerts'
    telegram_configs:
      - api_url: 'https://api.telegram.org'
        bot_token: 'YOUR_BOT_TOKEN'
        chat_id: 'YOUR_CHAT_ID'
        message: 'CRITICAL: {{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
        parse_mode: 'Markdown'
    email_configs:
      - to: '[email protected]'
        send_resolved: true
        headers:
          Subject: 'CRITICAL: {{ .GroupLabels.alertname }}'

  - name: 'warning-alerts'
    slack_configs:
      - channel: '#homelab-warnings'
        send_resolved: true

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'job']
  1. Create docker-compose.yml:
services:
  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    restart: unless-stopped
    ports:
      - "9093:9093"
    volumes:
      - ./config:/etc/alertmanager
      - alertmanager-data:/alertmanager
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
      - '--web.external-url=http://alertmanager.yourdomain.com'

volumes:
  alertmanager-data:
  1. Start the container:
docker compose up -d
  1. Verify the UI is accessible at http://your-server-ip:9093

Step 2: Configure Prometheus to Send Alerts

Objective

Add alerting rules to Prometheus and point it at Alertmanager.

Step-by-Step Instructions

  1. Edit your Prometheus prometheus.yml:
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - alertmanager:9093

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
  1. Create alerting rules in rules/alerts.yml:
groups:
  - name: homelab-alerts
    interval: 30s
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% for more than 5 minutes."

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Disk space low on {{ $labels.instance }}"
          description: "Less than 10% disk space remaining on {{ $labels.device }}."

      - alert: NodeDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Node {{ $labels.instance }} is down"
          description: "Prometheus has been unable to scrape {{ $labels.instance }} for 1 minute."

      - alert: ContainerHighMemory
        expr: (container_memory_usage_bytes / container_spec_memory_limit_bytes) > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container memory high"
          description: "Container {{ $labels.name }} is using >85% of its memory limit."
  1. Reload Prometheus configuration:
curl -X POST http://localhost:9090/-/reload

Step 3: Set Up Telegram Bot Notifications

Objective

Create a Telegram bot and configure Alertmanager to send critical alerts to it.

Step-by-Step Instructions

  1. Message @BotFather on Telegram and create a new bot. Save the token.

  2. Get your chat ID:

curl -s https://api.telegram.org/botYOUR_TOKEN/getUpdates | grep -o '"chat":{"id":[0-9]*' | grep -o '[0-9]*'

  1. Update alertmanager.yml with your real token and chat ID.

  2. Restart Alertmanager:

docker compose restart alertmanager
  1. Test by firing a manual alert:
curl -X POST http://localhost:9093/api/v1/alerts \
  -H 'Content-Type: application/json' \
  -d '[{"labels":{"alertname":"TestAlert","severity":"critical"},"annotations":{"summary":"This is a test"},"generatorURL":"http://test"}]'

Step 4: Configure Slack Notifications

Objective

Create a Slack webhook and send alerts to a channel.

Step-by-Step Instructions

  1. In Slack, go to Settings & Administration > Manage Apps > Incoming Webhooks
  2. Add a new webhook to your #homelab-alerts channel
  3. Copy the webhook URL
  4. Update alertmanager.yml with the webhook URL
  5. Restart Alertmanager
  6. Test with the same curl command as above

Step 5: Add Silencing and Maintenance Windows

Objective

Prevent alert spam during planned maintenance.

Step-by-Step Instructions

  1. Use the Alertmanager UI (http://your-server:9093) to create silences manually
  2. Or use the API for automated maintenance scripts:
curl -X POST http://localhost:9093/api/v1/silences \
  -H 'Content-Type: application/json' \
  -d '{
    "matchers": [
      {"name": "alertname", "value": "NodeDown", "isRegex": false}
    ],
    "startsAt": "2026-06-10T02:00:00Z",
    "endsAt": "2026-06-10T04:00:00Z",
    "createdBy": "admin",
    "comment": "Scheduled maintenance window"
  }'

Pro Tips

Tip 1: Group Alerts to Avoid Spam

The group_by field in the route configuration groups alerts by labels. If 10 containers on the same host are down, you get one notification, not 10. Use group_by: ['alertname', 'job'] for service-level grouping and group_by: ['alertname', 'instance'] for host-level grouping.

Tip 2: Use Inhibition for Cascade Failures

If your router is down, every service behind it will also appear down. The inhibition rule suppresses “NodeDown” for dependent services when the router is already critical. Define dependencies explicitly:

inhibit_rules:
  - source_match:
      severity: 'critical'
      alertname: 'RouterDown'
    target_match:
      severity: 'critical'
    equal: ['datacenter']

Tip 3: Keep Alertmanager Config in Git

Store alertmanager.yml in a Git repository. Use a CI pipeline to validate the YAML and reload the container on push. This prevents configuration drift and gives you an audit trail.

Troubleshooting Common Issues

Problem 1: Alerts Not Firing

Check: Prometheus alerting rules page (http://prometheus:9090/alerts). If the rule is green, it is not firing. If yellow, it is pending. If red, it is firing but may not be reaching Alertmanager.

Fix: Verify the alerting block in prometheus.yml points to the correct Alertmanager host and port. If using Docker, ensure both containers are on the same network.

Problem 2: Telegram Messages Not Received

Check: Verify the bot token and chat ID. Test with:

curl -s "https://api.telegram.org/botTOKEN/sendMessage?chat_id=CHAT_ID&text=Test"

Fix: If the bot is not in the group, add it. If the group is private, use the numeric chat ID, not the group name.

Problem 3: Duplicate Alerts

Check: Ensure Prometheus is not scraping multiple instances that report the same metrics. If you have two Prometheus servers, configure external labels to distinguish them.

Conclusion

Summary

Prometheus Alertmanager turns metrics into actionable notifications. With proper grouping, routing, and inhibition, you get timely alerts without noise. Telegram and Slack are the most popular homelab receivers. Use silencing for maintenance windows and keep your config in version control.

Next Steps

  1. Add more alerting rules for your specific services (Nextcloud, Jellyfin, Plex)
  2. Configure PagerDuty or webhook receivers for critical infrastructure
  3. Set up a Grafana OnCall or Uptime Kuma as a secondary alerting path
  4. Review and tune alert thresholds monthly

Affiliate Opportunities

  • VPS: Hetzner for offsite Alertmanager (referral)
  • Notification hardware: Grafana OnCall, PagerDuty (referral)

Internal Linking Strategy

  • prometheus-setup → guide: “prometheus-monitoring-homelab.md”
  • grafana-setup → guide: “grafana-docker-compose.md”
  • docker-monitoring → guide: “docker-monitoring-grafana-prometheus.md”
  • node-exporter → guide: “prometheus-monitoring-homelab.md”

CTA

  • [comment] What notification channel do you use for homelab alerts? Telegram, Slack, or something else?
  • [newsletter] Subscribe for our monitoring stack templates and alerting rule packs.
  • [internal_link] Need to set up Prometheus first? Read our Prometheus monitoring guide next.