Reading time: ~14 minutes Audience: Homelabbers wanting observability for their infrastructure

What Is Prometheus?

Overview

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It collects metrics from configured targets at given intervals, stores them in a time-series database, and allows querying via PromQL. It is the de facto standard for cloud-native monitoring and works perfectly for homelab observability.

Key Benefits

Benefit Detail
Pull-based Prometheus scrapes targets — no agent push needed
Time-series DB Efficient storage of numeric metrics over time
PromQL Powerful query language for alerts and dashboards
Exporters Hundreds of community exporters for hardware, apps, and services
Alertmanager Route alerts to email, Slack, Telegram, or webhooks
Service discovery Auto-detect Docker containers, Kubernetes pods, or DNS entries
Native Grafana First-class integration with Grafana dashboards

Prerequisites

Hardware Requirements

  • Any Docker host (mini PC, server, or VM)
  • 1GB RAM for Prometheus + Alertmanager + Node Exporter
  • 10GB+ storage for metrics retention (varies by scrape frequency and targets)

Software Requirements

  • Docker Engine 24.x+ and Docker Compose v2+
  • Linux host with persistent storage
  • Basic understanding of YAML and networking

Knowledge Prerequisites

  • Docker Compose basics
  • Understanding of metrics and time-series data
  • Basic YAML editing

Step 1: Deploy Prometheus Stack with Docker Compose

Objective

Run Prometheus, Node Exporter, and Alertmanager in Docker with persistent storage.

Step-by-Step Instructions

  1. Create the project directory:
mkdir -p ~/docker/prometheus && cd ~/docker/prometheus
mkdir -p prometheus_data alertmanager_data
  1. Create prometheus.yml configuration:
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - alertmanager:9093

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

  - job_name: 'docker'
    static_configs:
      - targets: ['docker-exporter:9323']
  1. Create alertmanager.yml:
route:
  receiver: 'default'
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: 'default'
    email_configs:
      - to: '[email protected]'
        from: '[email protected]'
        smarthost: 'smtp.gmail.com:587'
        auth_username: '[email protected]'
        auth_password: 'your-app-password'
        require_tls: true
  1. Create docker-compose.yml:
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./rules:/etc/prometheus/rules:ro
      - ./prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
    networks:
      - monitoring

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    restart: unless-stopped
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - ./alertmanager_data:/alertmanager
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    networks:
      - monitoring

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    privileged: true
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge
  1. Create alert rules directory and file:
mkdir -p rules
cat << 'EOF' > rules/homelab.yml
groups:
  - name: homelab
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% for more than 5 minutes."

      - alert: LowDiskSpace
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Disk space is below 10% on {{ $labels.mountpoint }}."

      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is above 85% for more than 5 minutes."
EOF
  1. Deploy:
docker compose up -d
  1. Verify:
  2. Prometheus UI: http://your-server-ip:9090
  3. Alertmanager: http://your-server-ip:9093
  4. Targets: http://your-server-ip:9090/targets

Step 2: Add Proxmox VE Exporter

Objective

Monitor Proxmox host metrics including VM/LXC status, storage, and node resources.

Step-by-Step Instructions

  1. On Proxmox host, create a monitoring user:
pveum user add monitoring@pve --password "Change...3!"
pveum aclmod / -user monitoring@pve -role PVEAuditor
  1. Add to prometheus.yml:
  - job_name: 'proxmox'
    static_configs:
      - targets: ['proxmox.example.com:9221']
    metrics_path: /pve
    params:
      module: [default]
  1. Deploy Proxmox exporter container:
  proxmox-exporter:
    image: prompve/prometheus-pve-exporter:latest
    container_name: proxmox-exporter
    restart: unless-stopped
    environment:
      - PVE_USER=monitoring@pve
      - PVE_PASSWORD=Change...3!
      - PVE_HOST=https://192.168.1.10:8006
    ports:
      - "9221:9221"
    networks:
      - monitoring

Step 3: Configure Docker Daemon Metrics

Objective

Enable Docker’s built-in metrics endpoint for Prometheus.

Step-by-Step Instructions

  1. Edit /etc/docker/daemon.json:
{
  "metrics-addr": "0.0.0.0:9323",
  "experimental": true
}
  1. Restart Docker:
systemctl restart docker
  1. Prometheus already scrapes this target via the docker job in prometheus.yml.

Pro Tips

Tip 1: Use Recording Rules for Complex Queries

Recording rules pre-compute expensive PromQL queries. Add them to rules/homelab.yml:

  - name: recording
    rules:
      - record: instance:cpu_usage
        expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Tip 2: Limit Retention for Storage

The default 30-day retention is usually enough for homelab. If storage is tight, reduce to 15 days:

command:
  - '--storage.tsdb.retention.time=15d'

Tip 3: Backup Prometheus Data

Prometheus data is in ./prometheus_data. Back it up with:

cd ~/docker/prometheus
tar czvf prometheus-backup-$(date +%F).tar.gz prometheus_data

Tip 4: Use Grafana for Visualization

Prometheus’s built-in UI is for debugging. Install Grafana for dashboards. See our Grafana Docker Compose guide for setup.

Troubleshooting Common Issues

Problem 1: “Targets Show as Down”

Cause: Network connectivity or container name resolution issues.

Fix:

# Test connectivity from Prometheus container
docker exec -it prometheus wget -qO- http://node-exporter:9100/metrics

# Check Prometheus logs
docker logs prometheus

Problem 2: “No Data in Queries”

Cause: Scrape interval too long, or target not configured.

Fix: - Reduce scrape_interval to 15s - Verify targets at http://your-server-ip:9090/targets - Check exporter is running: docker ps | grep exporter

Problem 3: “Alertmanager Not Sending Emails”

Cause: SMTP settings or authentication issues.

Fix: - Use an app-specific password (not your main password) for Gmail - Test SMTP with swaks or msmtp - Check Alertmanager logs: docker logs alertmanager

Conclusion

Summary

Prometheus gives you powerful observability over your homelab. With Node Exporter, cAdvisor, and the Proxmox exporter, you can monitor CPU, memory, disk, container, and VM metrics. Combined with Alertmanager, you’ll know immediately when something breaks.

Next Steps

  1. Install Grafana for beautiful dashboards
  2. Add Loki for log aggregation
  3. Compare with InfluxDB for alternative storage

Affiliate Opportunities

  • Samsung 990 EVO: Fast NVMe for Prometheus storage
  • Beelink Mini S12 Pro: Mini PC for monitoring stack

Internal Linking Strategy

  • intro → grafana-docker-compose for dashboard visualization
  • tip-4 → grafana-prometheus-setup for integrated setup
  • conclusion → prometheus-vs-influxdb for storage comparison

CTA

  • [comment] What’s your homelab monitoring stack? Prometheus, InfluxDB, or something else?
  • [newsletter] Subscribe for weekly observability and homelab monitoring guides.