PostgreSQL destination¶

OBSERVABILITY_TYPE=postgres
OBSERVABILITY_DB_POSTGRESQL=postgresql://axium:secret@db:5432/axium-api

The DSN can point at the main Axium database or at a separate one on any PostgreSQL 13+ server. The core opens its own pool for it (max 8 connections, 10 s to acquire one) so trace writes never compete with the request path for connections.

Schema management¶

At every start the core checks the telemetry schema in the trace database with the same logic it uses for its own schema:

Found	Action	Log
Nothing	Creates the schema, tables and indexes in one transaction	`[observability] schema version 1 installed`
Current version	Nothing	`[observability] schema version 1 is current`
Older version	Applies bundled migrations	`[observability] schema migrated a -> b`
`telemetry` schema without a version table	Refuses; observability stays disabled for this process	`[observability] disabled: …`

Several replicas can start at once; an advisory lock serialises the install. The user in the DSN needs CREATE on the database the first time.

Tables¶

`telemetry.proxy_traces`¶

One row per request. Columns match the event schema one to one, plus three generated columns:

Column	Definition
`latency_ms`	`(EXTRACT(EPOCH FROM ended_at - started_at) * 1000)::INTEGER`
`tokens_total`	`tokens_in + tokens_out` (NULL if either is NULL)
`cost`	`cost_in + cost_out` (NULL if either is NULL)

Indexes:

Index	Columns	Purpose
`idx_telemetry_proxy_traces_app`	`(proxy_app_id, provider_type, model_used, started_at)`	Per-app dashboards
`idx_telemetry_proxy_traces_started_at`	`(started_at)`	Time ranges, retention jobs
`idx_telemetry_proxy_traces_trace_id`	`(trace_id) WHERE trace_id IS NOT NULL`	Lookup by the `x-axium-trace-id` a client reports
`idx_telemetry_proxy_traces_tags`	GIN on `tags`	Filter by client tags (`tags @> '{"team":"sales"}'`)

Full DDL

CREATE SCHEMA IF NOT EXISTS telemetry;

CREATE TABLE telemetry.schema_version (
    version    INTEGER     NOT NULL,
    applied_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    install_id UUID        NOT NULL DEFAULT gen_random_uuid(),
    singleton  BOOLEAN     NOT NULL DEFAULT true UNIQUE CHECK (singleton)
);
INSERT INTO telemetry.schema_version (version) VALUES (1);

CREATE TABLE telemetry.schema_migrations (
    version    INTEGER     PRIMARY KEY,
    applied_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE IF NOT EXISTS telemetry.proxy_traces (
    trace_id            UUID,
    span_id             UUID,
    started_at          TIMESTAMPTZ,
    ended_at            TIMESTAMPTZ,
    latency_ms          INTEGER GENERATED ALWAYS AS ((EXTRACT(EPOCH FROM (ended_at - started_at)) * 1000)::INTEGER) STORED,
    source_module       TEXT,
    proxy_app_id        UUID,
    api_key_id          UUID,
    client_ip           INET,
    client_country      TEXT,
    client_user_id      TEXT,
    provider_id         UUID,
    provider_type       TEXT,
    provider_region     TEXT,
    model_id            UUID,
    model_used          TEXT,
    model_requested     TEXT,
    dialect             TEXT,
    surface             TEXT,
    upstream_path       TEXT,
    tokens_in           INTEGER,
    tokens_out          INTEGER,
    tokens_cache_read   INTEGER,
    tokens_cache_write  INTEGER,
    tokens_reasoning    INTEGER,
    tokens_saved        INTEGER,
    tokens_total        INTEGER GENERATED ALWAYS AS (tokens_in + tokens_out) STORED,
    cost_in             DECIMAL(14,6),
    cost_out            DECIMAL(14,6),
    cost                DECIMAL(14,6) GENERATED ALWAYS AS (cost_in + cost_out) STORED,
    cost_saved          DECIMAL(14,6),
    stream              BOOLEAN,
    max_tokens          INTEGER,
    error_origin        TEXT,
    status              TEXT,
    http_status         SMALLINT,
    error_category      TEXT,
    error_message       TEXT,
    cache_hit           BOOLEAN,
    fallback_used       BOOLEAN,
    policies_applied    TEXT[] NOT NULL DEFAULT '{}',
    denial_reason       TEXT,
    meta_tag            TEXT,
    tags                JSONB NOT NULL DEFAULT '{}',
    attributes          JSONB NOT NULL DEFAULT '{}'
);

CREATE INDEX IF NOT EXISTS idx_telemetry_proxy_traces_app        ON telemetry.proxy_traces (proxy_app_id, provider_type, model_used, started_at);
CREATE INDEX IF NOT EXISTS idx_telemetry_proxy_traces_started_at ON telemetry.proxy_traces (started_at);
CREATE INDEX IF NOT EXISTS idx_telemetry_proxy_traces_trace_id   ON telemetry.proxy_traces (trace_id) WHERE trace_id IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_telemetry_proxy_traces_tags       ON telemetry.proxy_traces USING GIN (tags);

Writes¶

Each batch (up to 50 traces, at most 2 s apart) becomes one multi-row INSERT. A failed insert drops the batch; see delivery.

Reading traces through the API¶

GET /v1/observability/traces (permission observability:read) queries the table. It exists only with this destination; with http or with observability off it answers 409 CONFLICT.

Query parameter	Type	Notes
`trace_id`	uuid	Exact match.
`proxy_app_id`	uuid
`status`	`success` · `error` · `denied` · `timeout` · `cancelled`
`from`, `to`	ISO-8601 with offset (`2026-01-01T00:00:00Z`)	Inclusive on `started_at`; `from` ≤ `to`.
`limit`	integer 1 to 1000	Default 100. Above 1000 is a `400`, not clipped.

Rows come back newest first with the generated columns included:

{ "status": true, "data": [ { "trace_id": "…", "latency_ms": 1360, "tokens_total": 155, "cost": 0.000039, "…": "…" } ], "total": 1 }

There is no offset. To page, pass the oldest started_at you received as to on the next call.

Useful queries¶

-- Cost per proxy app, last 7 days
SELECT proxy_app_id, model_used, sum(cost) AS cost, count(*) AS requests
FROM telemetry.proxy_traces
WHERE started_at >= now() - interval '7 days' AND status = 'success'
GROUP BY 1, 2 ORDER BY cost DESC;

-- p95 latency per model, today
SELECT model_used, percentile_cont(0.95) WITHIN GROUP (ORDER BY latency_ms) AS p95_ms
FROM telemetry.proxy_traces
WHERE started_at >= date_trunc('day', now()) AND status = 'success'
GROUP BY 1;

-- Why requests were denied
SELECT denial_reason, count(*) FROM telemetry.proxy_traces
WHERE status = 'denied' AND started_at >= now() - interval '1 day'
GROUP BY 1 ORDER BY 2 DESC;

-- Everything a given user did (x-axium-user-id)
SELECT started_at, surface, model_used, tokens_total, cost, status
FROM telemetry.proxy_traces WHERE client_user_id = 'user-42' ORDER BY started_at DESC;

Retention¶

Axium never deletes traces. Add your own job. The started_at index makes range deletes cheap:

DELETE FROM telemetry.proxy_traces WHERE started_at < now() - interval '365 days';

For large volumes convert the table to monthly partitions (pg_partman or native declarative partitioning) and drop old partitions instead. Axium only runs INSERTs against the table name, so partitioning is transparent to it.

client_ip is personal data under most privacy regimes; include the table in your data-retention and subject-access procedures, or null the column after a short period:

UPDATE telemetry.proxy_traces SET client_ip = NULL
WHERE started_at < now() - interval '30 days' AND client_ip IS NOT NULL;