Preflight checks on every server
Before any change, each selected server is checked: reachable, able to act as root when the run needs it, supported by the playbook, and with enough free disk space for installs. Servers that fail are marked as blocked with the reason and left untouched.
Preflight checks run on several servers at a time rather than one after another.
A canary server first
With the canary on, one server runs the change by itself first. If it fails, the remaining servers are skipped and the run stops. If it succeeds, the run pauses until someone confirms, with the canary server’s output available to review.
Dry runs skip the canary, because they change nothing.
Batches with a failure limit
After confirmation, the remaining servers run in batches of 1 to 50. Each batch starts only when the previous one has finished.
You set how many failures to tolerate. The default is zero: once failures exceed the limit, no further batches start and the remaining servers are left unchanged.
Retry failed servers
When a run finishes, every server has a status: succeeded, failed, blocked, skipped or cancelled. A retry starts a new run for every server that did not succeed, linked to the original, so servers that succeeded are never run twice.
Example: updating 50 servers
Preflight. All 50 servers are checked. Two are out of disk space and marked as blocked.
Canary. One of the remaining 48 servers is updated successfully.
Confirmation. You review the canary server’s output and continue.
Batches. The remaining servers update in batches of five. One fails after a package mirror times out twice.
Retry. After freeing disk space on the two blocked servers, you retry every server that did not succeed.
Frequently asked questions
- Can I turn off the canary?
- Yes, for runs where waiting for confirmation is not useful. It is on by default, and it matters most for commands, which have no dry run.
- How long does a run wait for confirmation?
- Up to 24 hours after the canary succeeds. Nothing else runs in the meantime, and if nobody confirms, the run is cancelled and the other servers are not touched.
- What does a failure limit of zero mean?
- The first failure after the canary stops further batches from starting. Servers already running in the current batch finish, and the rest are left untouched.
- How large can a batch be?
- Between 1 and 50 servers.
- Does the preflight change anything on the server?
- No. It connects and reads what the server runs and what the account can do. A server that fails the preflight is left exactly as it was.
Key points
- Preflight checks on every server before any change
- One canary server, then a person confirms
- Batches of 1 to 50 servers
- Stops when failures exceed your limit
- Retry only the servers that did not succeed