Troubleshooting¶

Symptoms first, then the cause and the fix. Start every investigation with:

curl -s http://<core>/v1/install/status
docker logs axium-core 2>&1 | grep -E '^\[(install|db|license|observability|errorHandler)\]'

Startup¶

The container exits immediately with an environment error

envalid prints the offending variable. Typical cases:

MASTER_ENCRYPTION_KEY: must be 64 hex characters (32 bytes): a trailing newline, a key of the wrong length, or a Base64 value. Generate with openssl rand -hex 32 | tr -d '\n'.
DATABASE_URL: ... missing: not passed at all.
AXIUM_METASERVER or PORT invalid.

[db] waiting for postgres... for 30 seconds, then the process dies

The core cannot open a connection. Check, in order: the host in DATABASE_URL resolves from inside the container (axium-db on the same Docker network, not localhost), the port, the credentials, and that PostgreSQL accepts the connection (pg_hba.conf, Cloud SQL attachment, sslmode). A Cloud SQL DSN needs the @localhost/db?host=/cloudsql/… form.

/v1/install/status returns not_installed after startup

The automatic install failed. grep '\[install\]' in the logs. The usual message is permission denied for database: the user cannot create schemas.

GRANT CREATE ON DATABASE "axium-api" TO axium;

Then restart the core, or call POST /v1/install.

/v1/install/status returns partial

Axium schemas exist without the core.schema_version table: an installation died halfway, or someone restored a partial dump. On a fresh installation, drop the schemas and restart:

DROP SCHEMA core, llm, rm, stats CASCADE;

On a database with data, do not drop anything; contact support with the logs.

The database is at schema version N but this build ships M

You started an older image against a database that a newer version had already migrated. Deploy the newer image. Schema never downgrades.

[install] migration failed

The process exits on purpose. Restore the backup taken before the upgrade, roll the image back, and send the log line to support.

[observability] disabled: …

The destination could not be set up. Causes: unreadable DSN, the trace database user lacks CREATE, a telemetry schema left half-created (partial), an unparseable OBSERVABILITY_HTTP_HEADERS. Fix the value and restart; observability does not retry during the process's life.

Console¶

The login page shows a network error

The browser cannot reach API_URL. Open the browser's developer tools:

Mixed content blocked: the ui is https:// and API_URL is http://. Use https://.
Name not resolved: API_URL uses an internal name (http://axium-core:3000/v1). It must be a URL the browser can reach.
CORS error: usually a proxy in front of the core stripping headers or answering the pre-flight itself. The core allows all origins; let OPTIONS requests through.
404 on login: API_URL is missing the /v1 suffix or has a trailing slash.

Invalid email or password with the initial credentials

The user was already changed, or the database was restored from another installation. Reset with an administrator account, or, if none is available, with SQL against core.users (ask support for the procedure).

Providers fail after a restore or a redeploy

MASTER_ENCRYPTION_KEY differs from the one the credentials were encrypted with. Recover the original key. If it is gone, edit each provider and enter its credentials again.

Runtime (SDK traffic)¶

401 from the gateway

The key is unknown, inactive or expired; all three answer the same. Check the key in the console (GET /v1/rm/api-keys), its expires_at, and that the SDK actually sends it: the OpenAI SDK as Authorization: Bearer, Google's as x-goog-api-key. A ?key= query parameter is ignored.

403 from the gateway

Read the error message and the trace's denial_reason:

Message / reason	Fix
key of another app, scope mismatch	Use a key created for this proxy app.
`app_not_ready`	Publish the app (`PATCH /v1/proxy-apps/:id/publish`) or reactivate it.
`model_not_allowed`	Add the model to the app's allow-list, or fix the name.
`provider_inactive`	Activate the provider.
`firewall_denied`	Client IP not allowed. Behind a proxy, set `TRUST_PROXY=true` or every client looks like the proxy.
`dlp_blocked`	The prompt contained data the policy blocks.

404 Route not found from the Vertex gateway in Java

The SDK entered Vertex express mode because it was given .apiKey() alone and changed the path to /v1/publishers/…. Pass the key as a static credential instead; see SDK examples.

429 from the gateway

Either the provider's own rate limit relayed verbatim (error_origin: provider) or the app's budget (denial_reason: budget_exhausted). Raise budget_limit, wait for the period to roll over, or check the provider's quota.

Streaming arrives all at once at the end

Something in front buffers the response. Disable proxy buffering (nginx proxy_buffering off, Caddy flush_interval -1). The core already sends X-Accel-Buffering: no.

Uploads fail with 413 or a connection reset

Three limits apply: the reverse proxy's body size, MAX_UPLOAD_SIZE (default 50 MB), and the container's memory, since uploads are buffered in RAM. Align all three.

Gemini resumable upload: the second request goes to the wrong host

Set PUBLIC_BASE_URL=https://axium-api.your-domain.com and TRUST_PROXY=true so the upload URL handed back points through Axium with the right scheme.

403 PERMISSION_DENIED … aiplatform.endpoints.predict on Google Cloud

The core's service account lacks roles/aiplatform.user in the project configured on the provider. Re-run gcp.sh or update.sh with VERTEX_PROJECT=<that project>, or grant the role by hand.

Cost is null in traces and statistics

The model has no prices configured. Set input and output price per million tokens on the model.

Collecting information for support¶

curl -s http://<core>/v1/diagnostics -H "Authorization: Bearer <token>"
docker logs --since 1h axium-core 2>&1 | grep -vE '^\[(req|runtime|usage)\]'

Plus the x-axium-trace-id of a failing request, which identifies its trace.