Skip to content

Observability

Axium records what happens to every request that goes through a proxy app, at two levels:

  • Daily statistics · always on

    Aggregates per proxy app, day, model and API key in the main database (stats.proxy_daily): requests, tokens and cost. They power the console's dashboards and GET /v1/stats/proxy.

  • Per-request traces · opt-in

    One record per request with identity, model, tokens, cost, timing, result, policies and client tags. Sent to the destination you choose: PostgreSQL or an HTTP endpoint.

    Destinations

Plus plain-text process logs on stdout, described in Request logging.

Turning traces on

Traces are off by default. One variable selects the destination, the rest configure it. Configuration is global to the installation and lives only in environment variables; there is no per-app, console or API switch.

OBSERVABILITY_TYPE=postgres
OBSERVABILITY_DB_POSTGRESQL=postgresql://axium:secret@db:5432/axium-api

Writes to telemetry.proxy_traces and makes traces queryable through GET /v1/observability/traces. Details

OBSERVABILITY_TYPE=http
OBSERVABILITY_HTTP_URL=https://collector.example.com/axium
OBSERVABILITY_HTTP_HEADERS='{"Authorization":"Bearer xxx"}'

POSTs {"version":1,"data":[…]} batches. Details

OBSERVABILITY_TYPE=http
OBSERVABILITY_HTTP_URL=https://clickhouse.example.com/?query=INSERT%20INTO%20axium.traces%20FORMAT%20JSONEachRow
OBSERVABILITY_HTTP_FORMAT=ndjson
OBSERVABILITY_HTTP_HEADERS='{"X-ClickHouse-Key":"xxx"}'

One JSON object per line, ready for ClickHouse JSONEachRow, BigQuery loads, Vector, Fluent Bit or a Lambda. Details

At startup the log confirms the result:

[observability] enabled (postgres)
[observability] enabled (http, ndjson)
[observability] disabled: <reason>

GET /v1/diagnostics (authenticated) reports the same under observability.

What a trace contains

Metadata only, never content:

Recorded Not recorded
Proxy app, API key, client IP, client user id and tags you send Prompts and responses, even truncated
Provider, region, model requested and model used, surface, upstream path Client headers other than x-axium-*
Tokens (in, out, cached, reasoning), cost in and out Credentials of any kind
Timing, streaming flag, max tokens
Status, HTTP status, error origin, category and provider message
Policies applied, denial reason, DLP findings (entity kinds, not values)

The exact schema, with types and the values each field can take, is in Formats & event schema.

How delivery works

sequenceDiagram
    participant R as Request
    participant B as In-memory buffer<br/>(per core process)
    participant D as Destination
    R->>B: add(trace) · synchronous, never throws
    Note over B: flush every 2 s, or as soon as 50 traces are queued
    B->>D: batch of ≤ 50 traces
    alt write fails
        Note over B: batch dropped, counted as lost,<br/>logged at most once per 5 min
    end
    Note over B: cap 10,000 traces · oldest dropped beyond that
  • Recording never slows down or fails a request.
  • Batches are not retried. A destination outage loses the batches sent during it; the log says how many: [observability] write failed, N trace(s) lost so far: <error>.
  • On SIGTERM the buffer is flushed in one attempt per batch before exit.
  • The buffer is per process. A replica that is killed loses up to 2 seconds of traces, or more if the destination was slow.

If you need at-least-once delivery, point the HTTP destination at a local collector (Vector, Fluent Bit, an OTel Collector with an HTTP receiver) that queues on disk and forwards.