Troubleshooting Observability

Start with the producer, then verify ingest authentication and organization scope, and finally account for dashboard polling.

Quick diagnostic path

1. Confirm observability is enabled for the SDK or product.
2. Confirm the producer uses the expected backend base URL.
3. End or flush the current plan trace.
4. Check for shipper warnings without printing the API key or payload.
5. Confirm the API key belongs to the organization selected in the dashboard.
6. Wait for the next 30-second dashboard poll or refresh the page.

No sessions or traces are visible

Check these conditions in order:

• The producer has a configured API key.
• The API key is bound to an organization.
• The SDK session has not set enabled: false.
• The active trace has ended. In-flight traces are retained and are not shipped as completed batches.
• The backend base URL is correct. SDK shippers append /observability/spans themselves.
• The dashboard is showing the same organization as the API key.
• The requested period contains the telemetry. The current dashboard uses 7 days.

For an SDK session, call flushObservability() or flush_observability() to end and ship the active plan trace, then call close() at the end of the session.

Wrong organization or API key

The ingest endpoint resolves organization scope from the API-key record. The request body cannot choose another organization.

Symptoms include:

• Organization context not found from API key;
• telemetry visible in a different organization;
• an empty dashboard despite successful agent operations;
• 401 or 403 responses in shipper warnings.

Create or select an API key for the intended organization. Do not log the key while debugging. It is enough to log a safe key identifier or prefix already stored by your application.

The product label is wrong

The backend prefers apiKey.product over the producer's payload.product. A product-pinned API key can therefore label SDK telemetry as another product even when the payload is correct.

Inspect the API-key record or create a key with the intended product association. Do not change the payload to work around a server-side pinned product.

Telemetry appears after the turn

This is expected for product bridges:

• ArmorCodex accumulates hook records on disk and assembles the trace at Stop.
• ArmorClaude ends the current turn trace at Stop so its SDK shipper can send it.
• The dashboard polls every 30 seconds.

Allow for the turn to finish, shipping and retry time, and one polling interval. This is polling, not true streaming.

Agent name is missing

The dashboard resolves a display name from agentId. If no registry match exists, it shows the raw ID; if the producer sent no ID, it shows --.

• Set agentId on ArmorIQClient for TypeScript.
• Set agent_id on ArmorIQClient for Python.
• Product bridges use their configured product agent ID.

agentId is a free-form string. Unlike session and user IDs, it does not need to be a UUID.

Tokens are missing

Token totals come only from generation spans. Policy and tool spans do not infer model usage.

• In Python, call session.record_generation(...) with provider-reported usage.
• In TypeScript, call the exported recordGeneration(recorder, trace, attrs) helper using session.observabilityRecorder and session.activePlanTraceContext.
• Confirm the generation is recorded before the plan trace is flushed or closed.
• Check that token fields are non-negative integers.
• ArmorClaude's current bridge records policy and tool activity but does not create generation spans.

ArmorCodex derives per-turn generation usage from its transcript at Stop. A missing or unreadable transcript can therefore produce a trace without tokens.

The backend initializes trace and session aggregate counters to zero. The dashboard can therefore show 0 even when no generation span was reported. Open Trace Detail and inspect the raw span list to distinguish missing usage from a measured zero.

Cost is zero or not reported

Cost is best-effort. It requires a generation span and either a producer-supplied cost or a model name present in the SDK's static price table.

An unknown model currently produces numeric costUsd: 0, and the dashboard renders that as $0.00. Verify the generation's model, token fields, and SDK price table before treating zero as a measured cost.

If the raw trace has no generation span, usage was not reported even if an aggregate view shows numeric zero.

The Python and TypeScript model tables are versioned independently. A model can be priced in one SDK version before it appears in another.

Shipper failures and retries

The TypeScript and Python shippers use the same retry and drop policy:

• 4xx: log and drop immediately because retrying the same rejected request is unlikely to help;
• network error or 5xx: retry three times after 500 ms, 1 second, and 2 seconds;
• exhausted retries: log and drop the affected batches;

Request timeout depends on the transport:

• a normal TypeScript session reuses the SDK client's HTTP instance, whose default timeout is 30 seconds;
• a standalone TypeScript ObservabilityRecorder that creates its own HTTP client uses 10 seconds;
• Python applies a 10-second timeout to each observability POST.

Telemetry exceptions do not change a policy decision or raise into the agent operation. An awaited explicit flush or product boundary hook can still wait for the request timeout and retries, so an unavailable endpoint can add shutdown or Stop latency.

Look for warnings such as shipper got 401, shipper got 400, or shipper failed after 3 retries. Do not enable HTTP logging that prints authorization headers or full telemetry bodies.

Observability is disabled

TypeScript SDK

Check SessionOptions.observability.enabled. There is no dedicated TypeScript ARMORIQ_OBSERVABILITY environment toggle in the current SDK.

Python SDK

Check both configuration and environment:

• ObservabilityConfig(enabled=False) disables it.
• ARMORIQ_OBSERVABILITY=false, 0, no, or off disables implicit observability when no config object overrides the flag.

Because ObservabilityConfig.enabled defaults to True, passing any config object makes that value win over ARMORIQ_OBSERVABILITY. Set enabled=False explicitly when you pass the object and want observability off.

ArmorCodex

Set ARMORCODEX_OBSERVABILITY_DISABLED=true. The configuration bridge also honors a forwarded disable_observability plugin option, although the current packaged plugin metadata may not expose that option in a configuration UI. The environment variable is the reliable customer-facing opt-out. Observability is also off when no API key is configured.

ArmorClaude

Set the disable_observability plugin option or ARMORIQ_OBSERVABILITY_DISABLED=true. Observability normally requires an API key or local-mock mode.

Local endpoint configuration

Pass a backend base URL, not the ingest path.

For example, use http://127.0.0.1:8080, not http://127.0.0.1:8080/observability/spans.

Dashboard refresh delay

The dashboard issues a new query every 30 seconds. It does not subscribe to a live stream. If direct API reads show a trace but the page does not, wait for the next poll or reload the page.

The current UI also has first-page limits and no Load more control:

• Overview requests 50 sessions.
• Trace Explorer requests 100 traces.

Use the cursor-paginated API when you need older rows beyond those first pages.

Duplicate trace handling

A duplicate trace ID normally means the shipper retried a request that the backend had already committed.

The backend counts the batch under duplicate, does not count it as rejected, and does not reapply spans or session totals. This is safe and does not require cleanup.

If distinct operations reuse the same trace UUID, fix the producer so every new trace mints a new UUID.

Missing or partial tool input and output

Inputs and outputs are optional, may be truncated by a producer, and can be omitted for privacy. ArmorCodex caps serialized attributes for safe atomic file appends. SDK applications decide what they pass as tool input, output, prompt, and completion.

Do not assume missing content is a transport failure if the trace, status, and span metadata are present.

Safe debugging

Log only:

• producer name and version;
• backend host without query parameters;
• trace, session, and span IDs;
• HTTP status;
• accepted, rejected, and duplicate counts;
• queue size or number of batches;
• exception class and a bounded message.

Do not log:

• API keys, JWTs, authorization headers, cookies, or passwords;
• full prompts, completions, tool inputs, or tool outputs;
• raw credentials files;
• full ingest payloads in shared logs.

When you need to inspect a payload, reproduce with synthetic values and use the data model and API reference as the expected contract.