Playbook

Start Monitoring Stack

Prometheus, Node Exporter and Grafana in Docker, listening on the server only.

Category

Monitoring

Time limit per server

30 minutes

Temporary failures

Retried automatically, up to 2 times

Version

1.0

Supported systems

Ubuntu 22.04+, Debian 12+, RHEL-compatible 8+ (AlmaLinux, Rocky Linux, RHEL, CentOS Stream, Oracle Linux), Amazon Linux 2023+. Still runs on Ubuntu 20.04 and Debian 11, which are past the end of standard support.

When to use it

When you want detailed, long-lived metrics and your own dashboards for a server, beyond the CPU, memory and disk readings the platform keeps. Run Install Docker first, with the Compose plugin; the Monitoring Server template runs both.

This playbook has no options.

What it does on the server

  1. Checks that Docker and Docker Compose are installed and that ports 9090, 9100 and 3000 are free.
  2. Writes a Compose project in /opt/iap-monitoring with pinned image versions: Prometheus v3.13.3, Node Exporter v1.12.1 and Grafana 13.0.8.
  3. Generates Grafana's administrator password on the server, the first time only, and stores it in /opt/iap-monitoring/.env, readable only by root. It is never shown in the run or sent to the platform.
  4. Configures Prometheus to scrape this server's Node Exporter and keep 15 days of metrics, and provisions Grafana with Prometheus as its data source.
  5. Starts the stack and waits until all three answer.

All three listen on 127.0.0.1 only: Prometheus on 9090, Node Exporter on 9100, Grafana on 3000. Nothing is exposed to the internet.

After the run

Open Grafana through an SSH tunnel from your computer. For a server at 203.0.113.10 that you sign in to as root:

ssh -L 3000:127.0.0.1:3000 root@203.0.113.10

Use your server's own address and account. Then browse to http://localhost:3000 and sign in as admin. Read the password on the server with grep GRAFANA_ADMIN_PASSWORD /opt/iap-monitoring/.env. Import a Node Exporter dashboard from Grafana's library to see the server's metrics.

To reach Grafana over HTTPS instead, put Configure Reverse Proxy in front of http://127.0.0.1:3000 with a domain of your own.

Common problems

"The monitoring stack runs on Docker with Compose, and they are not both installed on this server." — Run Install Docker with Install the Compose plugin on, then this playbook again.

"Port … is already in use on this server" — Something else listens on 9090, 9100 or 3000, often an existing Prometheus or Grafana. Stop it, or keep your existing installation instead of this one.

"The stack was started, but not every service was answering after two minutes." — The container states and recent logs are in the output. On a server with little memory Grafana can be slow to start; run the playbook again once, which reuses what is there.