Observability

Observability

For: the administrator of a self-hosted Juxtaboard server. This guide covers self-hosted installations only. Covers: exporting traces, metrics and logs over OpenTelemetry (OTLP), serving metrics for Prometheus, the ready-made Grafana dashboard, and how to check that it works. You need: a running server, a platform account that may open System → Config (the system right manage_jobs), and a collector or a Prometheus server you operate yourself.

The server can send traces, metrics and logs over OTLP to any OpenTelemetry collector (for example Grafana Alloy or the OpenTelemetry Collector, with Tempo, Loki and Mimir behind it). It can also serve metrics on a Prometheus scrape port.

It is off by default. While it is off, none of the telemetry code is loaded; the only cost is a flag check at each instrumented point.

The in-app traces page (System → Traces, /system/traces) is a separate feature. It lists LLM agent execution traces stored by the server, and it works whether or not OpenTelemetry is on.

1. Switch it on

1.

Sign in with a platform account and open System → Config.

2.

Open the Observability tab.

3.

Fill in at least one destination: a collector under Collector (OTLP), or Serve metrics for Prometheus under Prometheus (see section 2).

4.

Turn on the Enable OpenTelemetry switch at the top of the panel.

5.

Press Save.

6.

Restart the application on every node. Settings are read once, when a node starts.

For the Docker installation the restart is:

docker compose restart app

Check: reload the Observability tab. The status block lists each node as running since .... If it says off with a reason, or Runs different settings from the saved ones - restart this node to apply, read the reason and see the table at the end of section 7.

How settings are stored and applied:

Item

Detail

Where

In the server's configuration table. The settings JSON holds everything on the panel except the OTLP headers

OTLP headers

One key=value per line. Write-only: once saved the panel shows only that they are set. To replace them, type new ones. To delete them, tick Remove the saved headers

When it applies

When a node starts. Exporters, metric filters and instrumentations are fixed at start

Kill switch

The environment variable OTEL_SDK_DISABLED=true on a node keeps telemetry off on that node whatever is saved. The panel shows when it is set

Standard variables

The standard OTEL_* variables (such as OTEL_EXPORTER_OTLP_ENDPOINT) are not read. The panel is the only source

Start failure

If the settings cannot be read when a node starts (for example the database is not up yet), that node runs without telemetry and the panel says so

Careful: the Docker Compose file shipped with the installation passes a fixed list of variables to the app container. To use OTEL_SDK_DISABLED there, add it to the app service's environment section.

The status block refreshes every 30 seconds and shows every node that reported recently. A node that stops reporting drops off the list within 90 seconds. Each node's line shows:

•

running since a time, or off and why;

•

the Prometheus address it listens on, or why the port could not be bound;

•

the last SDK error (a failed export, a refused scrape);

•

a restart warning when the settings that node runs differ from the saved ones.

2. Destinations

OTLP

Under Collector (OTLP), enter the collector's base URL in OTLP endpoint, for example http://collector:4318. For HTTP, /v1/traces, /v1/metrics and /v1/logs are added for you. A URL that already ends in one of them is refused.

OTLP protocol

Typical port

Notes

HTTP / protobuf (default)

4318

Lighter

gRPC

4317

Loads the gRPC library, the heaviest dependency here, only when chosen

•

Leave OTLP endpoint empty to use Prometheus only.

•

Traces and logs can only go over OTLP.

•

Metrics go over OTLP unless Send metrics over OTLP is off. They can go to Prometheus as well.

•

OTLP metrics are pushed every 30 seconds.

Prometheus

Under Prometheus, turn on Serve metrics for Prometheus. Each node then listens on its own scrape port.

Field

Default

Prometheus bind address

0.0.0.0

Prometheus port

9464

Path

/metrics (only GET /metrics answers; any other path returns 404)

Allowed IP masks

127.0.0.1/32 and ::1/128

Scrape every node directly, by its own address. Never scrape through the load balancer: each scrape would reach a random node.

Allowed IP masks work like this:

•

One mask per line, in CIDR form (10.0.0.0/8) or as a single address.

•

Every connection is checked on arrival against the connecting address itself, not a forwarded header, so it cannot be spoofed. A client outside the masks is disconnected before any HTTP is read. The first refusal, and then every 100th, is logged and shown on the panel as the last error.

•

An IPv4 client that arrives on a dual-stack socket (::ffff:10.0.0.5) matches its IPv4 masks.

•

An empty list admits nobody. Saving with Prometheus on and no valid mask is refused.

•

A mask with prefix /0 admits everyone. The panel asks you to tick a confirmation before it saves.

Careful: the masks are defence in depth, not a replacement for a firewall. Keep a firewall rule that allows the scrape port only from your Prometheus server.

Docker installation: the Compose file shipped with the installation does not publish port 9464. To scrape from outside the host, publish it from the app service in a Compose override, bound to a private address, or run Prometheus on the same Docker network. Add the address the application actually sees for the scraper (the Docker network gateway or the Prometheus container address) to Allowed IP masks.

Example prometheus.yml, one target per node:

scrape_configs:
  - job_name: juxtaboard
    static_configs:
      - targets: ['203.0.113.11:9464', '203.0.113.12:9464']

3. What is collected

The What to collect table has one row per group, with separate Traces and Metrics switches. A dash means the group cannot produce that signal.

Defaults: traces on for incoming requests, outgoing HTTP, SQL, background jobs and AI calls; metrics on for every group except incoming requests; Schedulers traces off; Node.js runtime traces not available.

Entry groups start traces: Incoming requests, Background jobs and Schedulers (marked "starts traces" on the panel). Switching an entry group's traces off drops the whole trace, including the SQL, HTTP and AI spans inside it.

Other groups only add detail to a trace that an entry group started. They never start a trace. An outgoing call made in a timer, or an AI call outside any job or request, produces no trace.

Group

Traces

Metrics (Prometheus names)

Incoming requests (entry)

Request spans named GET /route, plus route handler and render spans. /_next/static/, /_next/image, /_next/webpack-hmr, /favicon.ico and your Ignored request paths are never traced

None

Outgoing HTTP

One client span per outgoing HTTP call. Every query value is replaced by REDACTED and credentials are removed

http_client_request_duration

SQL (main pool)

One span per statement inside a traced operation, named like SELECT documents or INSERT rows, with db.operation.name, db.collection.name, db.query.text (capped at 2 KB) and the PostgreSQL error code as error.type. Each transaction is one span named transaction, and the statements inside it are not traced separately

db_client_operation_duration (by operation and error), juxtaboard_db_transaction_duration (outcome commit or rollback), juxtaboard_db_operations_in_flight, juxtaboard_db_transactions_open, db_client_connection_max

Background jobs (entry)

A root span job <type> with juxtaboard.job.type, juxtaboard.job.id, juxtaboard.job.attempt and juxtaboard.job.outcome

juxtaboard_job_duration (job type, outcome success, retry, failed, requeued), juxtaboard_jobs_active

Schedulers (entry)

A root span scheduler <id> with juxtaboard.scheduler.id, created only on the node that did the tick

juxtaboard_scheduler_tick_duration (scheduler id, outcome; its _count is the tick count), juxtaboard_scheduler_enabled

AI calls

AI SDK spans such as ai.streamText, ai.generateText and ai.toolCall, with model and token usage. ai.telemetry.functionId names the feature: chat.agent, portal.agent, portal.completion, agent.run, agent.data_run, agent.data_extract, agent.proxy, automation.agent_preview, inbox.analysis, image.describe, text_extract.ocr, analysis.ai_doc, and others

gen_ai_client_token_usage (model, role, token type input or output), gen_ai_client_operation_duration, juxtaboard_ai_calls_total (role, status), juxtaboard_ai_pool_active, juxtaboard_ai_pool_max

Node.js runtime

None

nodejs_eventloop_delay_*, nodejs_eventloop_utilization, v8js_memory_*, v8js_gc_duration, and juxtaboard_mcp_sessions_open (MCP sessions held open by the process)

Metric names above are the Prometheus form. Histograms also carry _bucket, _sum and _count, and some sources add a unit suffix such as _seconds (see section 4).

Reading scheduler outcomes

Every process wakes at the same time slots, and one of them claims each slot in Valkey. So ran for one scheduler is at most (60 / interval in seconds) per minute across the whole cluster, however many processes run it.

Outcome

Meaning

ran

This process won the slot and the lock, and did the work

skipped

It won the lock, but the period was already handled

slot_taken

Another process claimed the slot. The normal outcome on every other process. It costs no database work

lock_busy

It won the slot, but the previous tick still holds the lock: a tick that outlived its interval

claim_error

Valkey did not answer in time, so the tick fell back to the lock alone. Ticks may run more often while this lasts

error

The work failed

The schedulers cross-table-stats and traces-local-sweep run on every process and record only ran or error.

juxtaboard_scheduler_enabled is 1 on every process that runs cluster schedulers (the SCHEDULERS environment variable is not none). Summed, it is the number of processes competing for each slot.

Logs

Under Logs, turn on Send logs to mirror console output to the collector as OTLP log records. This needs an OTLP endpoint and is off by default.

•

Minimum level is debug, info (default), warn or error.

•

Each record carries the trace_id and span_id of whatever was running.

•

A [Component] prefix at the start of a line becomes the code.namespace attribute.

•

A log body is capped at 8,192 characters.

•

Lines the server's log-noise filter drops are not exported.

Pool starvation

Watch juxtaboard_db_operations_in_flight against db_client_connection_max, and juxtaboard_db_transactions_open. Query duration includes the time spent waiting for a free connection.

AI metrics

AI metrics are recorded for each metered AI call, the same place that feeds usage and billing. The AI code-analysis and AI-documentation features are traced but are not counted in the AI metrics, just as they are not counted in usage. OCR and image descriptions are counted per model call.

Request span names

Every request span is named by its route template, for example POST /projects/[projectId]/navigator or GET /api/mcp/route, never by the concrete URL. Tools that group spans by name would otherwise create one series per URL.

•

If the framework did not record the route, the server matches the path against the build's own route list. A path that matches nothing is named <METHOD> (unmatched).

•

http.target keeps the concrete path with every query value redacted.

•

In a development build there is no complete route list, so spans are named by the path without its query.

Streaming requests

Every request span carries juxtaboard.request.streaming (true or false). It is true for MCP connections, server-sent event streams (.../stream), streamed AI responses (chat, portal agent, agent proxy) and downloads (.../download). Their duration is how long the client stayed connected, not latency. The Grafana dashboard's p95 and slow-request panels leave them out. Request and error rates count everything.

4. Grafana dashboard

The Observability tab generates a dashboard for Grafana.

1.

In the Grafana dashboard section, choose How metrics reach Prometheus (see the table below).

2.

Press Download JSON, or Copy JSON.

3.

In Grafana open Dashboards → New → Import, paste or upload the file.

4.

At the top of the dashboard, pick your Prometheus, Loki and Tempo data sources.

Rows: overview, requests (TraceQL metrics from Tempo), PostgreSQL including pool saturation and transaction time, background jobs, schedulers, AI (tokens, calls, latency, provider pool, spans by call site), outgoing HTTP by host, Node.js runtime, and failed or slow traces and logs. Trace and log panels use the service name juxtaboard.

The same metric has a different name depending on how it reaches Prometheus, so the choice matters:

How metrics reach Prometheus

Example series

Service label

OTLP → Collector / Alloy → Prometheus

db_client_operation_duration_seconds_count

exported_job

OTLP straight into Prometheus / Mimir

db_client_operation_duration_seconds_count

job

Prometheus scrapes the app

db_client_operation_duration_count (no unit suffix)

job

If panels are empty, the dashboard was probably generated for a different choice than the pipeline you actually use. Generate it again with the right one.

5. Sampling and volume

•

Trace sample ratio (0 to 1, default 1) applies to new traces. Everything inside a kept trace is kept.

•

Ignored request paths: one path prefix per line. The default is /api/health. Each must start with /.

•

No trace context is propagated. No traceparent header is sent to AI providers, webhooks or connectors, which would hand trace ids to third parties. None is accepted from clients either, because a client could otherwise force its requests to be sampled.

•

SQL span cap. At most 500 SQL spans per parent span. When a loop runs more, the first 500 are recorded and the rest are counted in the attribute juxtaboard.db.spans_dropped on the parent. The duration metric stays complete.

•

Attribute size. An attribute value is cut at 16,384 characters.

•

Export buffers. The span queue and the log queue each hold 4,096 entries and drop anything beyond that. If the collector is down, memory stays bounded and the panel shows the export error.

•

Loss on restart. Up to about 5 seconds of buffered spans and 30 seconds of OTLP metrics can be lost when a node restarts. A graceful stop tries to flush them, but do not rely on it.

If load is a concern, reduce in this order:

1.

Record prompt, response and tool-call text on AI spans. Off by default. When on, the whole conversation is serialised at every agent step.

2.

SQL (main pool) traces. They are the largest number of spans. Switch the traces off and keep the metrics.

3.

Logs. Raise Minimum level.

4.

Schedulers traces. Off by default, and the noisiest source when on.

6. Privacy

•

AI prompts, responses and tool text are not recorded unless Record prompt, response and tool-call text on AI spans is on. They may contain personal data.

•

Metric labels carry no user ids, no project ids and no customer-named AI provider. AI metrics are labelled by model and role only.

•

Outgoing URLs are redacted: parameter names are kept, every value becomes REDACTED, and credentials are removed.

•

Span attributes can carry job ids, scheduler ids and table names. SQL text is the parameterised statement, with values bound separately, except where the code inlines a value.

•

A trace sent to a collector is data you now store somewhere else. Apply the retention and access rules of that system.

7. Verify it works

1.

Status. After the restart, System → Config → Observability shows each node as running since ... with no red error line.

2.

Startup log. The application logs one line when telemetry starts:

docker compose logs app | grep "\[telemetry\]"

Check: you see [telemetry] Started: traces=... otlpMetrics=... prometheus=... logs=... followed by the collector address, and, with Prometheus on, Prometheus metrics on http://...:9464/metrics, allowed: ....

3.

Prometheus port. From a machine that matches Allowed IP masks:

curl -s http://203.0.113.11:9464/metrics | grep -E "db_client_connection_max|juxtaboard_db_operations_in_flight"

Check: both series are listed. From a machine outside the masks the connection is reset.

4.

Traces. Open any page of the application, then search your tracing backend for the service juxtaboard. Check: a trace named like GET /projects/[projectId]/navigator appears within a few seconds. Requests to /api/health are ignored by default and will not appear.

5.

Metrics over OTLP. Allow up to a minute. Check: juxtaboard_job_duration_seconds_count (or the name your pipeline produces) grows after background jobs run.

If something is missing:

Symptom

Likely cause

Status says off with Enabled, but no destination (OTLP endpoint or Prometheus) is set

No OTLP endpoint and Prometheus is off

Status says OTEL_SDK_DISABLED=true in the environment

The kill switch is set on that node

Warning Runs different settings from the saved ones

The node was not restarted after Save

Red Prometheus port: error naming an address in use

Another process holds the port. Change Prometheus port and restart

Red Prometheus: refused connection from ...

The scraper's address is not in Allowed IP masks

Red export errors

The collector is unreachable or rejects the headers. Check OTLP endpoint, OTLP protocol and OTLP headers

Empty Grafana panels

The dashboard was generated for another How metrics reach Prometheus choice (section 4)

8. Not covered

•

Worker threads: no telemetry inside them.

•

Valkey commands and real-time collaboration (WebSocket) traffic.

•

Individual statements inside a transaction. The transaction is one span.

•

Browser-side monitoring.

•

Linking a background job's trace to the request that queued it. Jobs are always trace roots.

•

•

•