Monitoring

Auto-Fix

Many outages have simple causes: a service stopped and did not restart, a PHP-FPM pool hung, or package caches filled the disk. When a monitor attached to a server fails, the platform tries a safe fix first and records what it did.

Four safe fixes

Restart stopped services. Web server, PHP, database and application services that are stopped or failed are started. Services that are already running are not touched.

Reload PHP-FPM. A graceful reload that lets requests in progress finish.

Free disk space. Clears package caches, trims the system journal to one week and removes the platform’s own temporary files.

Reload the web server. After testing its configuration, so a certificate already renewed on disk is actually served.

Fixes matched to the failure

The fix depends on what the check saw. A site that does not respond may have a stopped service. A 502, 503 or 504 from a gateway may point to a failed PHP pool. A failing disk check gets the disk fix, and a failing SSL check gets a web server reload.

A 500 or 404 gets no fix. Those responses come from an application that is running, and restarting services underneath it would not address the cause. When no fix applies, the alert is sent straight away.

What Auto-Fix never does

It never reboots a server, deletes your data, changes firewall rules or rolls back a deployment.

Each of these can turn a short outage into a longer one, and each is a decision that should be made by a person.

Alerts are delayed, not suppressed

While a fix runs, which takes about a minute, the down alert waits. If the check still fails afterwards, the alert is sent with the fix that was attempted. If the fix works, the incident records what was done.

Auto-Fix runs at most three times per monitor per day. After that, the platform stops trying and sends the alert, because a failure that keeps coming back needs a person to investigate.

Frequently asked questions

Which monitors can use Auto-Fix?
Monitors attached to one of your servers, because a fix needs a server to run on. Auto-Fix is on by default for those monitors and can be turned off for any of them.
Will Auto-Fix reboot my server?
No. The platform never reboots a server on its own, and it never deletes your data, changes firewall rules or rolls back a deployment.
Will I still be alerted?
Yes, whenever a fix does not resolve the problem. The alert waits about a minute while the fix runs, then includes what was attempted. Successful fixes are recorded on the incident.
What stops it restarting a crashing service in a loop?
A limit of three automatic fixes per monitor per day. After that, the platform stops trying and sends the alert.
What does the disk space fix delete?
Package manager caches, system journal entries older than a week, and temporary files the platform created. Your files, logs and databases are not touched.

Key points

  • Restarts services, reloads PHP-FPM, frees disk space, reloads the web server
  • Fix chosen from the failure the check detected
  • Never reboots, deletes data, changes firewall rules or rolls back
  • Alert sent if the fix does not work
  • At most three fixes per monitor per day

Related features