High availability and multi-node deployment

This guide is for administrators of a self-hosted installation who want to run the application on two or more nodes behind a load balancer, so that one node can fail or be updated without an outage.

It covers the reference architecture, what the application needs in order to be safe on several nodes, the load balancer settings, the health endpoint, and a checklist. The procedures for updating nodes, taking one out of the pool and recovering from failures are in .

You need a working single-node installation first (), and a place to run a PostgreSQL server, a Valkey (or Redis-compatible) server and an S3-compatible bucket that is not on one of the app nodes.

1. Reference architecture

                 users
                   |
        app.example.com  ->  203.0.113.10
                   |
          +--------+---------+
          |  load balancer   |   TLS, WebSocket upgrade, health checks
          |  (nginx, HAProxy |
          |   or a cloud LB) |
          +----+--------+----+
               |        |
        +------+--+  +--+------+
        | node-a  |  | node-b  |    identical app nodes, port 3000
        | app     |  | app     |    (add node-c, ... the same way)
        +--+---+--+  +--+---+--+
           |   |        |   |
           |   +---+----+   |
           |       |        |
   +-------+--+ +--+-------+ +----+---------------+
   | Postgres | |  Valkey  |     | S3-compatible   |
   | primary  | |  primary |     | object storage  |
   |   + standby | + replica |   | (uploads)       |
   +----------+ +----------+     +-----------------+

The application does not need a shared filesystem between nodes, and it has no leader election of its own. Coordination goes through PostgreSQL (job claims, advisory locks) and Valkey (collaboration, slot claims, locks).

2. What the application does on several nodes

These behaviours decide how you configure the cluster.

3. Configuration every app node needs

All nodes get the same values except NODE_ID.

Generate the secrets once and copy them to every node:

openssl rand -base64 48 | tr -d '\n'

openssl rand -base64 32 | tr -d '\n'

The first command gives an AUTH_SECRET, the second a CONNECTOR_ENCRYPTION_KEY.

Careful: do not generate secrets separately on each node, and do not rotate AUTH_SECRET or CONNECTOR_ENCRYPTION_KEY on one node only. Rotating the encryption key without re-entering the credentials makes stored connector credentials unreadable.

Example for a container-based node (Docker Compose). It uses only the application service of the bundled stack: remove the bundled db and valkey services and point the variables at your shared services.

services:
  app:
    image: ${CODEDOCS_IMAGE}
    restart: unless-stopped
    ports:
      - "10.0.0.11:3000:3000"        # private interface only; the load balancer reaches it here
    stop_grace_period: 20s            # the application needs up to 12 s to drain
    environment:
      DATABASE_URL: postgres://app:<password>@db.example.internal:5432/app
      VALKEY_URL: redis://:<password>@valkey.example.internal:6379
      PUBLIC_APP_URL: https://app.example.com
      AUTH_SECRET: ${AUTH_SECRET}
      CONNECTOR_ENCRYPTION_KEY: ${CONNECTOR_ENCRYPTION_KEY}
      NODE_ID: node-a
      NEXT_MANUAL_SIG_HANDLE: "true"
      SKIP_MIGRATIONS: "1"            # migrations are run separately, see below
    volumes:
      - traces:/app/traces
volumes:
  traces:

Migrations. The bundled container entrypoint applies pending migrations on every start unless SKIP_MIGRATIONS=1 is set. Choose one approach and keep it:

Careful: while the first updated node migrates, the nodes still running the old version keep serving against the new schema. Read the release notes before a rolling update. If a release is marked as incompatible with the previous version, stop all nodes and update them together.

Check: on each node, curl -s http://127.0.0.1:3000/api/health returns "status":"ok" and the node field shows that node's own NODE_ID.

4. Shared services

4.1 PostgreSQL

Use a managed PostgreSQL service with automatic failover if you can. The extensions vector and pg_trgm must be available on it, because the schema needs them.

If you run it yourself, set up one primary and one streaming standby. On the primary, in postgresql.conf:

listen_addresses = '*'
wal_level = replica
max_wal_senders = 10
max_replication_slots = 10
hot_standby = on
max_slot_wal_keep_size = 20GB

Create the replication role and allow the standby to connect (adjust the address and the authentication method to your policy):

CREATE ROLE replicator WITH REPLICATION LOGIN PASSWORD '<strong password>';

host  replication  replicator  198.51.100.21/32  scram-sha-256

On the standby, with an empty data directory and the same PostgreSQL major version and extension packages as the primary:

pg_basebackup -h db-primary.example.internal -U replicator -D /var/lib/postgresql/data -R -X stream -C -S standby_1

Check on the primary:

SELECT application_name, state, sync_state,
       pg_size_pretty(pg_wal_lsn_diff(sent_lsn, replay_lsn)) AS lag
FROM pg_stat_replication;

Expect state = streaming. Check on the standby:

SELECT pg_is_in_recovery();

Expect t.

Connection budget: each node opens a main pool of 20 connections (PG_POOL_MAX) and a pool of up to 10 for advisory locks (PG_LOCK_POOL_MAX), plus a few small auxiliary connections. If you use a separate Log DB (set under System > Config > Storage & Logs > Log DB configuration), its address must be reachable from every node: never use 127.0.0.1 there. Size max_connections for all nodes together with headroom.

Careful: replication is asynchronous by default. If the primary is lost, the last few committed transactions may not be on the standby. Synchronous replication removes that gap at the cost of waiting for the standby on every commit. Decide on purpose.

Careful: a replication slot makes the primary keep WAL until the standby has read it. A standby that is offline for long fills the primary's disk. max_slot_wal_keep_size caps this; a slot that exceeds the cap is invalidated and the standby must be rebuilt. Monitor the slot and the free disk space ().

Replication protects against losing a server, not against deleted or corrupted data. Backups are a separate duty: .

4.2 Valkey

Run Valkey with persistence enabled (append-only file), as the bundled stack does, and without a key-eviction policy (maxmemory-policy noeviction) so that locks and logs are never evicted. Protect it with a password and keep it off public networks.

A replica is optional and is for fast recovery. In its configuration:

replicaof valkey-primary.example.internal 6379
masterauth <password>
requirepass <password>

Check on the primary:

valkey-cli -h valkey-primary.example.internal -a "$VK_PASS" --no-auth-warning INFO replication

Expect role:master and connected_slaves:1. Check on the replica:

valkey-cli -h valkey-replica.example.internal -a "$VK_PASS" --no-auth-warning INFO replication

Expect role:slave and master_link_status:up.

Careful: replication copies stored data but not published messages. A subscriber receives live collaboration messages only from the Valkey it is connected to. Applications must always connect to the current primary, never to a replica.

4.3 Object storage

Check: upload an image in a project and open the same image URL through each node's own address (port 3000). Both return the image.

Switching the provider affects new uploads only. To move existing files, use the Migrate storage panel on the same tab: copy, verify, then switch over.

Careful: do not add a second node while Local filesystem is active. A file uploaded through one node would be missing on the other.

5. Load balancer

The balancer must:

Which routes need which kind of routing:

Example nginx configuration. Replace the addresses with your nodes' private addresses. Keep the server lists identical on every balancer: the hash-based methods give different answers if the lists differ.

map $http_upgrade $connection_upgrade {
    default upgrade;
    ""      "";
}

upstream app_pool {
    hash $cookie_session consistent;
    server 10.0.0.11:3000 max_fails=1 fail_timeout=10s;   # node-a
    server 10.0.0.12:3000 max_fails=1 fail_timeout=10s;   # node-b
    keepalive 32;
}

upstream app_sheet {
    hash $uri consistent;
    server 10.0.0.11:3000 max_fails=1 fail_timeout=10s;
    server 10.0.0.12:3000 max_fails=1 fail_timeout=10s;
    keepalive 32;
}

upstream app_mcp {
    ip_hash;
    server 10.0.0.11:3000 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:3000 max_fails=3 fail_timeout=30s;
}

server {
    listen 443 ssl http2;
    server_name app.example.com;
    # ssl_certificate / ssl_certificate_key ...

    client_max_body_size 50m;
    proxy_connect_timeout 2s;
    proxy_read_timeout 86400s;
    proxy_send_timeout 86400s;
    proxy_next_upstream error timeout http_502 http_503 http_504;

    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_buffering off;

    location ~ ^/api/projects/[^/]+/documents/[^/]+/sheet-collab$ {
        proxy_pass http://app_sheet;
    }

    location /api/mcp {
        proxy_pass http://app_mcp;
    }

    location / {
        proxy_pass http://app_pool;
    }
}

Notes on these settings:

Careful: if you run two balancers, give them the same configuration with the same node list. A balancer that lists a node as itself plus a public address, instead of the other node, quietly sends everything to one node.

Check: add $upstream_addr to the balancer's log_format, send a few requests with different session cookie values, and confirm in the access log that they are spread over both nodes while one cookie value always goes to the same node. Requests to /api/health through the public name come back with the node field of whichever node served them (when HEALTH_TOKEN is set, send the token).

6. Health endpoint

GET /api/health is the single health check for balancers and monitoring. It runs a database SELECT 1, a Valkey PING and a connection-pool check, each with a 3-second limit, so it answers even when a dependency hangs.

Example healthy body (shortened):

{
  "status": "ok",
  "node": "node-a",
  "pid": 7,
  "uptime_s": 30532,
  "checks": { "db": "ok", "valkey": "ok", "pool": "ok" },
  "version": { "sha": "a1b2c3d", "buildTime": "2026-01-01T00:00:00Z", "buildId": "..." }
}

version.buildId differs on every build. Compare it across nodes after an update to prove that every node runs the same build. version.sha and version.buildTime are filled in only if you pass BUILD_SHA and BUILD_TIME when you build the image; otherwise they read unknown.

The token can be sent as an X-Health-Token header, as Authorization: Bearer <token>, or as a ?token= query parameter. Prefer the header: a query parameter is written to the balancer's access log on every probe.

Careful: the container image of the bundled stack has its own health check that calls /api/health without a token. If you set HEALTH_TOKEN, that check gets 401 and the container is reported unhealthy. Override the container health check to send the header, and give your balancer's health checks the token too. If you cannot send a header, leave HEALTH_TOKEN unset and restrict access to port 3000 by firewall instead.

Point the balancer at each node's own address and port 3000, not at the public name. Alerts and dashboards for the same endpoint are described in .

Besides the endpoint, System > Health > Jobs shows each background schedule, whether it runs once per cluster or on every node, and which node ran it last. After you add a node, it should appear there within a few minutes.

7. How a node stops

When the process receives SIGTERM or SIGINT (a container stop, a service stop, a process-manager restart) it drains instead of dropping its work. The whole sequence has a 12-second limit:

A second signal, or running out of time, exits immediately. For this to work your process manager must wait longer than 12 seconds before it kills the process (stop_grace_period: 20s in Compose, TimeoutStopSec=20 in systemd, kill_timeout: 15000 in PM2) and NEXT_MANUAL_SIG_HANDLE=true must be set. A process killed without the drain loses unsaved edits made since the last flush and leaves its jobs waiting for the 5-minute lease to lapse.

The application also stops itself if its event loop is blocked for more than 5 seconds (a built-in watchdog; the first 60 seconds after start are excused). Your process manager must restart it automatically (restart: unless-stopped, Restart=always).

8. Checklist

Before going live with more than one node:

Check: sign in as two different users in two browsers and open the same document. Use the balancer's access log (with $upstream_addr) to see which node each browser's WebSocket went to, and repeat with other users until the two are on different nodes. Type in one browser: the edit appears in the other within a second. Then stop one node as described in the runbook: the page keeps working and the editors reconnect.