Multi-node runbook: updates, node removal and failover

This runbook is for administrators of a self-hosted multi-node installation described in . It covers day-to-day operations (checking the cluster, updating nodes one at a time, taking a node out of the pool, adding or restoring a node) and what to do when a node, the database, Valkey or the load balancer fails.

Prerequisites: the cluster is built as in the high-availability guide, you can reach every node over SSH, and you know which backup you would restore from (). Examples use node-a.internal and node-b.internal for the app nodes, app.example.com for the public name, and Docker Compose commands; use your process manager's equivalents if you do not run containers.

1. Check the state of the cluster

Run this before and after every operation.

Each node answers on its own address:

curl -s http://node-a.internal:3000/api/health

curl -s http://node-b.internal:3000/api/health

If HEALTH_TOKEN is set, add -H "X-Health-Token: <token>" to both. Expect HTTP 200 and "status":"ok".

Compare the build on every node:

for h in node-a.internal node-b.internal; do echo -n "$h: "; curl -s http://$h:3000/api/health | grep -o '"buildId":"[^"]*"'; done

Every line must show the same buildId.

Database replication, on the primary:

SELECT application_name, state, sync_state,
       pg_size_pretty(pg_wal_lsn_diff(sent_lsn, replay_lsn)) AS lag
FROM pg_stat_replication;

Expect streaming and a lag near zero.

Valkey, on the primary:

valkey-cli -h valkey-primary.example.internal -a "$VK_PASS" --no-auth-warning INFO replication

Expect role:master (and connected_slaves:1 if you run a replica).

In the application, open System > Health > Jobs: every schedule should be healthy, and nodes that ran work should be listed by their NODE_ID.

Check: all nodes are ok, all buildId values are equal, replication is streaming.

2. Update the nodes one at a time (rolling update)

A rolling update keeps at least one node serving at all times. Never update two nodes at once.

Read the release notes. If the release says it is not compatible with the previous version, do not use this procedure: take a maintenance window, stop all nodes, update and start them together.

Take a fresh backup of PostgreSQL ().

Fetch the new build onto every node first, without restarting anything. With the bundled stack that is a pull or a load of the new image. A failed download is then found before any node is touched.

If your nodes run with SKIP_MIGRATIONS=1, apply the migrations once, from one place, with the new image:

docker run --rm -e DATABASE_URL="$DATABASE_URL" --entrypoint node ${NEW_IMAGE} /app/docker/migrate.mjs

Expect [migrate] up to date. If you let the nodes migrate on start, the first updated node does it.

For each node in turn, do steps 6 to 10.

Take the node out of the pool (section 3). This is optional for a clean stop, because a stopping node closes its listener and the balancer retries elsewhere, but it avoids a burst of retried requests.

Stop the node and wait for it to exit. The drain takes up to 12 seconds.

docker compose stop app

Start the new version.

docker compose up -d app

Wait until the node is healthy. A cold start can take a minute or more, so allow three.

until curl -fsS http://node-a.internal:3000/api/health >/dev/null; do sleep 5; done

10.

Put the node back into the pool and watch the error logs and System > Health > Jobs for a few minutes before touching the next node.

11.

After the last node, run the build comparison from section 1.

Check: every node reports the same new buildId, and System > Health > Jobs shows no failing schedule.

Careful: if a node does not become healthy, stop. Do not update the next node. The other node is still serving the old version. Fix or roll back the failed node first. If the failed update stops midway, a node can keep serving an old build while its health check stays green, so always compare buildId at the end.

Careful: a node that was updated and then fails must not go back into the pool on an older build than its neighbours. Compare buildId before re-adding it.

If you changed any value under System > Config since the last restart, the nodes that did not save it still use the old value until they restart. The rolling update restarts them, so that is the natural moment to confirm the new value is in effect on every node.

3. Take a node out of the pool and put it back

In nginx, add down to the node's server line in every upstream block (app_pool, app_sheet, app_mcp in the example of the high-availability guide), on every balancer, and leave a dated comment saying why:

server 10.0.0.12:3000 max_fails=1 fail_timeout=10s down;   # node-b, maintenance, YYYY-MM-DD

nginx -t

systemctl reload nginx

With hash-based routing only the users of that node move to the others. Their editors reconnect, and their spreadsheet and MCP sessions restart on the new node.

To put it back, remove down, test the configuration and reload again.

Check: the access log (with $upstream_addr) no longer shows the node, and /api/health on the node still answers if the node is running.

Careful: when you copy a balancer configuration to a second balancer, check whether the file contains a down marker. Copying it carries the marker over and silently removes the same node on the second balancer.

4. Add a node, or restore a node that was lost

Adding a new node and restoring a failed one are the same procedure, because nodes keep no state of their own.

Provision the host with the container runtime, the same image as the other nodes and time synchronisation.

Give it the same environment as the other nodes: the same AUTH_SECRET, CONNECTOR_ENCRYPTION_KEY, DATABASE_URL, VALKEY_URL and PUBLIC_APP_URL, and a new, unique NODE_ID. Set NEXT_MANUAL_SIG_HANDLE=true and a stop grace period of at least 20 seconds.

Allow the balancer to reach port 3000 on the node, and the node to reach PostgreSQL, Valkey and the object storage.

Start the node and wait for /api/health to answer ok (section 2, step 9).

Confirm its buildId equals the other nodes' (section 1).

Add it to the upstream blocks of every balancer, keeping the server order identical, then reload.

Check: the node appears in System > Health > Jobs under its NODE_ID within a few minutes, and the access log shows requests served by it.

Careful: do not copy a data directory, container volume or .env file from another node to create a node and keep its NODE_ID. A duplicated NODE_ID makes two nodes look like one, and they can renew each other's job leases.

5. When an app node fails

What happens by itself:

•

The balancer stops sending traffic to the node after its health check fails (or after a connection fails) and retries on another node.

•

Editors connected to that node lose their WebSocket and reconnect through the balancer to another node. That node rebuilds each document from the last PostgreSQL snapshot plus the update log in Valkey, so the content is the same as it was, except for the last fraction of a second of typing that the failed node had not yet written to Valkey (updates are batched for about 100 ms).

•

Jobs that the node was running keep their lease. When the lease (5 minutes) lapses, another node takes them over. On a clean stop they are handed back immediately.

•

MCP clients get a 404 for their old session and must start a new one. Spreadsheet editors reconnect.

What you do:

Look at /api/health on the node and at its logs. A node that reports degraded names the failing dependency under checks.

If the cause is the node, restart it. If it is the host, follow section 4.

Compare buildId and check System > Health > Jobs after it returns.

A node that loses its connection to Valkey fails its health check within seconds and is removed from the pool, and a node cannot start without a working Valkey. The reason for both is that a node with a broken message channel would serve stale documents and could overwrite newer content with its older copy when it writes its next snapshot.

6. When the database primary fails (manual failover)

These steps are for self-managed streaming replication. With a managed database, use the provider's failover and continue at step 6.

Careful: the main risk is two writable primaries ("split brain"). Writes then go to two different databases and cannot be merged afterwards. Do not promote the standby until you are sure the old primary cannot accept writes.

Confirm that the primary is really down, from the app nodes and from the standby, and not just unreachable from your own machine.

Fence the old primary: power the host off, or block its PostgreSQL port at the network level. If you cannot reach it, isolate it through your hosting provider's console. Do not rely on it staying down by itself.

On the standby, check how far it got:

SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp();

With asynchronous replication the last few seconds of committed transactions may be missing. Write down the time shown, because it is the point you recovered to.

Promote the standby:

SELECT pg_promote();

Confirm that it is writable:

SELECT pg_is_in_recovery();

Expect f.

Point the nodes at the new primary. If DATABASE_URL uses a name or address that you can move to the new primary, move it. Otherwise change DATABASE_URL on each node (and the Log DB address, if you use a separate Log DB) and restart the nodes one at a time as in section 2.

Check /api/health on every node and System > Health > Jobs.

Rebuild the old primary as a standby of the new one. Do not start it as it is. Remove its data directory (after keeping a copy if you want to inspect it) and take a fresh base backup from the new primary, with the same command as in section 4.1 of the high-availability guide.

Take a new backup, and update monitoring to the new roles.

Check: pg_stat_replication on the new primary shows the rebuilt standby as streaming, and a test edit in the application survives a restart of a node.

Careful: a replication slot that nobody is reading keeps WAL on the primary indefinitely (up to max_slot_wal_keep_size). After a failover, drop the slot on the new primary that belonged to the old standby, or the disk fills. SELECT pg_drop_replication_slot('<slot name>');. A standby whose slot was dropped must be rebuilt from a new base backup.

7. When Valkey fails

Valkey carries live collaboration messages, the recent update log of documents being edited, locks and slot claims. Losing it does not lose saved content, which is in PostgreSQL. It can lose the edits made since the last document snapshot (every 200 updates or 10 minutes) if no editor's browser still holds them.

If the Valkey process or host is down:

Every node fails its health check and leaves the pool within seconds. Users see errors until Valkey is back or you switch.

If you have a replica, fence the old primary (stop it, or block its port), then promote the replica:

valkey-cli -h valkey-replica.example.internal -a "$VK_PASS" --no-auth-warning REPLICAOF NO ONE

Make VALKEY_URL reach the new primary. Move the name or address, or change the variable on each node.

Restart the app nodes one at a time (section 2, steps 6 to 10). Do not skip this: messages published while a node was disconnected are gone, and a node's in-memory copy of a document may have missed them. Restarting makes it rebuild from PostgreSQL and the log.

When the old primary comes back, turn it into a replica of the new one before it takes any traffic:

valkey-cli -h valkey-old.example.internal -a "$VK_PASS" --no-auth-warning REPLICAOF valkey-replica.example.internal 6379

Check: INFO replication shows the expected roles, every node is healthy, and two browsers on different nodes see each other's edits.

Careful: two writable Valkey primaries divide the nodes into two groups that cannot see each other's collaboration messages. Documents then diverge between nodes, and each node's next snapshot may overwrite the other's. Always make sure the old primary is stopped or demoted before nodes use the new one.

Careful: if CONNECTOR_ENCRYPTION_KEY is not set in the environment, the key is kept in Valkey and in a file in the app's working directory. A fresh, empty Valkey on a node without that file generates a new key, and stored connector credentials stop decrypting. Set the key in the environment on every node, as the high-availability guide requires.

8. When the load balancer fails

If you run two balancers, move the public address (a floating IP, a DNS record or your provider's equivalent) to the healthy one. The standby balancer needs the same configuration, the same node list and a valid certificate for app.example.com and any wildcard host. Keep certificates synchronised when they renew.

Check: curl -sI https://app.example.com/ returns the expected response from the standby, and WebSockets connect.

DNS-based switching is as slow as the record's time to live. Lower the TTL in advance if you rely on it.

9. Data-loss and split-brain cautions

•

Asynchronous database replication can lose the last few committed transactions at failover. Know your recovery point and tell users when you used a standby.

•

Never have two writable PostgreSQL primaries or two writable Valkey primaries. Fence first, then promote.

•

Never start a former primary without rebuilding it from the current one.

•

Replication is not a backup. A deleted project is deleted on the standby too. Test restoring from .

•

Do not drop a replication slot to "free the disk" unless you accept rebuilding that standby.

•

Do not rotate AUTH_SECRET or CONNECTOR_ENCRYPTION_KEY on some nodes only.

•

A node that was offline for a long time may be on an old build. Compare buildId before it rejoins the pool.

•

Do not run docker compose down -v, or any command that removes volumes, on a node that also hosts the database or Valkey data.

10. After any incident

All nodes ok, all buildId equal (section 1).

Replication streaming and lag near zero, free disk space on the database and Valkey hosts is healthy.

System > Health > Jobs has no failing schedules, and background jobs are being completed.

Take a fresh backup.

Write down what happened, how long it took and which recovery point you reached. Adjust the checklist in if something was missing.

11. Rehearse it

Failover steps that have never been run fail when they are needed. At least once a quarter, on a copy of the setup or in a quiet window:

•

stop one app node under light load and watch the balancer, the editors and the jobs;

•

do a rolling update with a no-op release;

•

promote the database standby, repoint the nodes and rebuild the old primary;

•

restore the latest backup to a scratch database and open a project from it.

Check: each rehearsal ended with all nodes healthy and equal in build, and the time each step took is written down.