Running Prometheus, Loki and Alertmanager on one VM: configuration, retention and cost

Two configuration patterns that circulate for this stack do not load. Prometheus retention is not a setting inside prometheus.yml, and Alertmanager performs no environment-variable expansion, so a receiver carrying $PAGERDUTY_SERVICE_KEY sends that literal string as the routing key. This runbook covers a Prometheus, Loki, Alertmanager and Grafana deployment on a single Linux VM that starts on the first attempt, the disk-sizing formula that decides the machine class, the constraints that decide whether one VM is the right shape at all, and the cost model that decides whether to run it instead of buying a platform.

Prerequisites

• One Linux VM with Docker Engine and the Compose plugin, plus a mounted block device for the time-series database and the Loki chunk cache. Size it with the formula below before choosing the machine.
• An S3-compatible object storage bucket with credentials. Loki needs it for chunks and the compactor needs it for the delete-request store.
• node_exporter running on every host to be scraped, with port 9100 reachable from the monitoring host and from nowhere else.
• A reverse proxy or tunnel terminating TLS in front of Grafana. Prometheus and Alertmanager support TLS and basic authentication through --web.config.file, but neither has user management or authorisation, so bind their ports to the loopback interface and put access control in front of them.
• An external probe living outside this VM. A monitoring stack cannot alert on its own death.
• A paging destination whose lifecycle you have checked. Grafana OnCall (OSS) entered maintenance mode on 2025-03-11 per the project README, and the repository is now archived and read-only; Atlassian states Opsgenie REST APIs work only until 2027-04-05. Wiring either as the escalation layer in 2026 buys a migration.

Retention is a flag, not a config key

The Prometheus storage documentation sets retention with two command-line flags: --storage.tsdb.retention.time, defaulting to 15d when neither flag is set, and --storage.tsdb.retention.size, defaulting to 0 and therefore disabled. When both are set, whichever triggers first wins. The configuration file's storage section accepts only the tsdb and exemplars blocks, so a storage.tsdb.retention.time key in prometheus.yml is a startup failure rather than a policy.

services:
  prometheus:
    image: prom/prometheus:${PROM_VERSION}   # pin an exact tag, never latest
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=60d
      - --storage.tsdb.retention.size=30GB
      - --web.enable-lifecycle
      - --web.enable-admin-api    # required for the snapshot call in Rollback
    volumes:
      - /srv/mon/prometheus:/etc/prometheus:ro
      - tsdb:/prometheus
    ports:
      - "127.0.0.1:9090:9090"
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:${AM_VERSION}
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
    volumes:
      - /srv/mon/alertmanager:/etc/alertmanager:ro
      - /srv/mon/secrets:/run/secrets:ro    # mode 0400, owned by the container uid
    ports:
      - "127.0.0.1:9093:9093"
    restart: unless-stopped

volumes:
  tsdb:

Compose interpolates ${PROM_VERSION} from the .env file next to it. Alertmanager does no equivalent inside its own configuration, and that asymmetry produces the broken receiver described below.

Size the disk before choosing the VM

The Prometheus storage documentation publishes the capacity formula directly: needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample, with roughly 1 to 2 bytes per sample on average. Worked example: thirty hosts running node_exporter at roughly 1,000 active series each is 30,000 series, and at a 30-second scrape interval that is 1,000 samples per second. Sixty days is 5,184,000 seconds. At the pessimistic 2 bytes per sample, 5,184,000 x 1,000 x 2 = 10.4 GB. The metrics side is rarely what forces a bigger machine.

Do not estimate the series count. Query prometheus_tsdb_head_series on a running instance and substitute the real figure. Then add headroom: the write-ahead log runs in 128 MB segments with a minimum of three files, and compaction produces blocks spanning up to 10 percent of the retention window or 31 days, whichever is smaller. The 30GB size cap in the Compose block is headroom over that 10.4 GB estimate; time is the primary retention control, and the size flag guards against unplanned cardinality growth filling the volume, so set it above the computed figure rather than at it.

Log volume lands elsewhere. Loki writes chunks to the object store in the configuration below, so daily ingest does not accumulate on the block device: the local volume carries the time-series database, the Loki index and the chunk cache, which together stay in the tens of GB at this estate size. Daily ingest sizes the bucket and the object-storage line in the cost model. Loki writes compressed chunks, so raw ingest overstates stored bytes by a margin that depends on the log format; rather than apply an assumed ratio, run for a week and read stored bytes off the bucket.

What breaks the single-VM shape is throughput rather than capacity. If the compactor cannot finish a cycle within its interval at the estate's ingest rate, or if the index plus chunk cache outgrows the largest volume the provider will attach to one machine, the single VM is the wrong answer and the rest of this runbook stops applying.

Alert rules that survive an on-call rota

The Prometheus alerting-practices document is blunt about the design: alert on symptoms associated with user pain rather than every possible cause, and aim for as few alerts as possible. Two rules from the common templates need fixing first. CPU utilisation belongs on rate. irate reads only the last two samples in the window and behaves erratically under an averaging aggregation with a long for clause. The disk-prediction rule fires on read-only mounts and on any large transient write to a mostly empty filesystem unless it is guarded.

groups:
  - name: infrastructure
    rules:
      - alert: HostDown
        expr: up{job="node"} == 0
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "Host {{ $labels.instance }} has not been scraped for 5m"

      - alert: HighCPU
        expr: 100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) > 90
        for: 15m
        labels: { severity: warning }

      - alert: DiskFillingUp
        expr: |
          predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.*"}[6h], 24*3600) < 0
          and on(instance, device) node_filesystem_readonly == 0
          and on(instance, device)
            node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.3
        for: 30m
        labels: { severity: warning }
        annotations:
          summary: "{{ $labels.device }} on {{ $labels.instance }} trends to full within 24h"

The readonly == 0 guard and the 30 percent utilisation floor together remove the largest single source of false pages in this rule set. Every rule that pages should carry a runbook annotation, because a page without a documented response trains people to ignore pages.

Alertmanager routing with the secrets outside the config

Alertmanager's configuration reference provides file-based variants for exactly this problem: service_key_file and routing_key_file for PagerDuty, api_url_file and app_token_file for Slack. Mount the secret, reference the path, and keep the credential out of the file that ends up in version control.

route:
  receiver: chat
  group_by: [alertname, service]
  group_wait: 30s        # documented default
  group_interval: 5m     # documented default
  repeat_interval: 4h    # documented default
  routes:
    - matchers: [ severity = critical ]
      receiver: page
      repeat_interval: 1h
    - matchers: [ severity = warning ]
      receiver: chat
      repeat_interval: 12h

receivers:
  - name: page
    pagerduty_configs:
      - routing_key_file: /run/secrets/pagerduty_routing_key
  - name: chat
    slack_configs:
      - api_url_file: /run/secrets/slack_webhook
        channel: "#alerts"
        send_resolved: true

Loki keeps logs forever until told otherwise

The Loki retention documentation is explicit that with no retention configured, logs sent to Loki live forever. Retention runs in the compactor and needs retention_enabled: true plus a delete_request_store. The global window lives in limits_config.retention_period; per-stream overrides use a selector with a priority and a documented minimum period of 24 hours.

compactor:
  retention_enabled: true
  delete_request_store: s3
  retention_delete_delay: 2h

limits_config:
  retention_period: 744h            # 31 days global
  retention_stream:
    - selector: '{job="nginx-access"}'
      priority: 1
      period: 168h                  # 7 days for high-volume access logs

The delete delay exists so index gateways can pull the modified index file before chunks disappear. Shortening it to zero to reclaim disk faster produces query failures during the window.

Expected output

The Prometheus management API documents /-/healthy as always returning 200 and /-/ready as returning 200 once the instance can serve queries. Check status codes rather than response bodies, which vary across versions.

for target in 9090/-/ready 9093/-/healthy 3100/ready 3000/api/health; do
  printf '%s ' "$target"
  curl -s -o /dev/null -w '%{http_code}\n' "http://127.0.0.1:${target%%/*}/${target#*/}"
done

curl -sG 'http://127.0.0.1:9090/api/v1/query' \
  --data-urlencode 'query=count(up == 1)' | grep -o '"value":\[[^]]*\]'

Expected output is four lines ending in 200 and a value array whose second element equals the number of scrape targets currently up. If that count is lower than the host inventory, the gap is the set of hosts believed to be monitored and not being monitored, which is the most useful number this stack produces on day one.

What it costs

Compute the self-hosted side rather than quoting it, because provider list prices move and a stale figure is an error the reader inherits. The cash line is: VM class + (volume GB from the sizing formula x provider per-GB rate) + (object storage GB x rate) + egress + paging seats. Read each rate off the provider's current price list on the day you build the model.

The platform side can be computed from a published list. Datadog's pricing page, read on 2026-07-28, lists Infrastructure Pro at USD 15 per host per month on an annual commitment and USD 18 on demand, APM at USD 31 per host per month with infrastructure attached, log ingest at USD 0.10 per ingested or scanned GB, and indexing at USD 1.70 per million log events per month at 15-day retention, with 30-day and longer windows quoted separately. Note that the 31-day Loki window configured above is not the tier that rate covers. For the 30-host estate above: infrastructure only is 30 x 15 = USD 450 per month annually committed. Infrastructure plus APM is 30 x 46 = USD 1,380 per month. Twenty GB of logs per day is roughly 600 GB per month, so USD 60 of ingest before any indexing.

The term that actually decides the question is the one nobody measures. Write it out: self_hosted_total = infra_cash + (operator_hours_per_month x loaded_hourly_rate). Track the hours for one quarter using the same ticket queue that carries the alerts, then compare against the platform total for the same estate shape. Any break-even quoted as a host count without those two inputs is a guess wearing a number. Routing alerts into a ticket queue so the hours are visible at all is the practical part, and it is the same integration described under workflow engineering and alert routing.

For scale reference on the machine class: the ITSailor platform runs Directus and n8n on a single Hetzner CPX32 with backups to a Storage Box, and that footprint bills at roughly EUR 20.49 per month as of 2026-07-28. That is the size of machine this runbook targets, and no monitoring workload has been benchmarked on it.

Side effects

• Every scraped host gains a listening port that publishes host inventory, kernel version and filesystem layout to anything that can reach it. Restrict 9100 to the monitoring host.
• Setting --storage.tsdb.retention.size deletes the oldest blocks without a confirmation step. A disk that fills during compaction can leave a time-series database that will not start.
• Enabling Loki retention deletes data on a schedule. A selector with a wrong label match silently expires a stream after the 24-hour minimum, and the delete delay hides the loss for a while.
• Alertmanager holds the first notification for a new group for group_wait, 30 seconds by default. No page from this stack is instantaneous; incident timelines should reflect that.
• Grafana on the same VM competes with Prometheus for page cache during a heavy dashboard query, which is precisely when an incident is in progress.
• Enabling --web.enable-admin-api for the snapshot call also exposes delete_series and clean_tombstones on the same unauthenticated port, which is a further reason 9090 stays bound to the loopback interface.
• Copying an active data directory is not a restorable backup. Use the admin snapshot API, then copy the snapshot. The wider pattern is covered under backup and disaster recovery.

Rollback

Before any upgrade, silence the estate so the failure mode is not a page storm: amtool silence add alertname=~".+" --duration=1h --comment="mon upgrade". Take a TSDB snapshot with curl -XPOST http://127.0.0.1:9090/api/v1/admin/tsdb/snapshot, which the Prometheus HTTP API documentation states requires --web.enable-admin-api, then copy the resulting snapshots/<datetime>-<rand> directory off the host. Keep the previous image tags in .env, which is why the tags are pinned there rather than written into the Compose file.

To roll back: docker compose down (which preserves named volumes), restore the previous values of PROM_VERSION and AM_VERSION, docker compose up -d, re-run the four health checks, confirm count(up == 1) matches its pre-upgrade value, then run amtool silence expire. Expiring the silence before that count is confirmed pages for healthy hosts whose scrapes have not restarted.

A tag revert is safe within a minor series. Across a major version the previous binary may refuse to read blocks the newer one wrote into the same named volume, in which case the rollback is restoring the snapshot into an empty data directory rather than reverting the tag.

To remove the stack entirely: expire the silences, delete the external probe, stop and remove node_exporter on every host and close port 9100, then docker compose down -v to destroy the volumes. Exporters left listening after a decommission are the usual residue, publishing host detail to a segment with nothing left that reads it.

Limitations

One VM is one failure domain. Running two Prometheus instances on that host does not create availability; genuine redundancy needs a second host and Alertmanager clustering via --cluster.peer, without which the two instances page twice for the same event. This runbook does not cover that topology.

The advice stops applying when the time-series database, the Loki index and the chunk cache together exceed the largest volume the provider will attach to one machine, or when log ingest exceeds what a single VM can compact within a compaction interval. Above that point the answer is a distributed deployment or a managed platform.

Queries spanning the full 31-day retention against object storage on a single small VM take minutes, and a longer retention_period makes that worse unless a query frontend, chunk caching and split_queries_by_interval are configured. Treat the cold tier as an investigation surface only.

The Kubernetes case is out of scope. Annotation-based scrape discovery is legacy, and current estates use the ServiceMonitor and PodMonitor custom resources from kube-prometheus-stack, which replaces this configuration model rather than extending it.

No claim is made here about how many of a given estate's incidents these rules catch. Establishing that requires a labelled incident history, which most organisations do not have until they build one. Record which alerts preceded which incidents for a quarter, then tune against that record.

The Datadog figures are USD list prices read on the date below. A negotiated contract, a different region, or currency conversion changes them, and a purchasing decision should be rebuilt against a quote.

Last verified 2026-07-28. Retention flags, the sizing formula, block and write-ahead-log behaviour, the health endpoints, the web authentication statement and the snapshot endpoint's flag requirement were checked against the Prometheus storage, configuration, HTTPS, alerting-practices, management-API and HTTP-API documentation. File-based receiver secrets and grouping defaults were checked against the Alertmanager configuration reference, retention behaviour against the Loki retention operations guide, vendor lifecycle dates against the archived Grafana OnCall repository and Atlassian's Opsgenie shutdown page, and prices against the Datadog pricing page.