Troubleshooting Observability

This page is for developers and operators who see no trace, a late trace, or a trace that is not complete. Start with the quick path. Then use the section that matches what you see.

Quick diagnostic path

1. Check that you run the current SDK: armoriq-sdk 0.8.17 or @armoriq/sdk 0.8.23. Older SDKs use the legacy recorder.
2. Check that tracing is on. See Tracing is switched off.
3. Open Settings > Observability. Check that the capture mode is not Off, and read the health signals.
4. Check where the SDK exports. See No-provider mode.
5. Check that the trace ended. See Incomplete traces.
6. Wait for the next 30-second dashboard poll, or reload the page.

Check the health signals

ArmorIQ gives you three places to look. None of them shows a secret.

In the dashboard

Open Settings > Observability and find Usage and pipeline health. See Organization controls.

The counters are accepted, rejected, duplicate, and mapping-failure spans, and failed requests. Duplicate spans are normal after a retry. See Duplicate spans.

You can open a trace to see more. A Rejected spans (trace-specific) value, a Pipeline health value of delayed, or the banner Telemetry arrived late (the data arrived more than one minute after the run ended) all point to a delivery problem for that trace.

To be told early, create alert rules for Exporter ingest rejection rate and Missing telemetry in the same settings page.

In the SDK log

The Python SDK logs a warning for a problem that it can report. These are the messages and what they mean.

The TypeScript SDK does not log these warnings. When external or provider export has an invalid setting, it logs nothing. It exports nothing and the agent continues. Check your settings with Choose who exports your traces.

In the answer from ArmorIQ

If you send OTLP yourself or through a Collector, these answers tell you what happened.

Retries are safe. See Duplicate spans.

No traces are visible

Check these in order:

1. SDK version. Use armoriq-sdk 0.8.17 or @armoriq/sdk 0.8.23.
2. Tracing is on. See Tracing is switched off.
3. Capture mode. If the organization mode is Off, or the SDK cannot read the policy, the SDK creates no spans.
4. Export destination. With external or provider, spans go to your destination, not to ArmorIQ. Spans go to ArmorIQ only with the default armoriq export, or through a Collector that you set up.
5. API key and organization. ArmorIQ takes the organization from the API key. The body of the request cannot choose another one. If the dashboard shows another organization, you see no data. A wrong key gives 401. A key with no organization gives 400 Organization context not found from API key.
6. The session ended. If you use the session API without an adapter, close the session. An open session does not export its root span.
7. Quota. A full monthly quota gives 429. Check Usage and pipeline health.
8. Permission. Without the content read permission, a trace page shows Permission required. Metadata still shows in the trace and session lists.
9. Time. The dashboard polls every 30 seconds. It does not stream.

Never print the API key, an authorization header, or a full payload while you debug.

The first spans of a new process are missing

The SDK reads your organization policy before it records data.

• TypeScript. With the armoriq and provider exports, the SDK does not wait for the first policy answer. Spans that start before it arrives are not recorded. A short script can lose all its spans. Wait once, at start-up, with await session.refreshObservabilityPolicy(). Do not call it inside a request hook. With the external export, the SDK waits up to 1.4 seconds for the first answer. If the answer is late, external export stays off for that session.
• Python. The first span waits for the policy answer, for up to 5 seconds, and it waits again after each policy expiry. If the policy service is slow, one request can wait. If it is down, the SDK records nothing, and tries again after about 60 seconds.

Incomplete traces

An incomplete trace is a trace that is missing spans or has not ended. The dashboard shows "This execution story is incomplete" and the reason. ArmorIQ gives each trace a completeness state:

A sweep runs every 15 minutes to set the timed-out, dropped, and unknown states. If a span arrives later, ArmorIQ recomputes the state and marks it as resumed.

Common causes and fixes:

• The root or plan span never ended. The SDK exports only spans that ended. A bare session that you never closed, a process that stopped before close(), or a framework that sent no end event leaves the trace in progress. Close the session, or call the close method of the adapter. See Framework support.
• The process stopped early. A TypeScript process drains ended spans on a normal exit, up to a fixed time. The SDK does not handle SIGINT, SIGTERM, or process.exit(). Close sessions before the process stops.
• Spans still open at close. In Python, child spans that are still open when the session closes are not exported.
• A parent is missing. The parent span was dropped, rejected by quota, or has not arrived yet. Wait for late data. Then check the rejected counters.
• The view is capped. The page says "Showing X of Y plan items" or "The span tree is limited to N captured spans." The data is stored. The view shows only part of it.
• The trace was sampled. If your organization uses sampling, a trace can be marked sampled out. ArmorIQ always keeps traces with errors, denials, approvals, delegations, or policy decisions.

Missing content is not the same as an incomplete trace. If prompts or tool results are empty but the spans are there, read the reason on the span. See Why a content field can be empty.

No-provider mode

No-provider mode means the SDK has nowhere to send spans, so it sends none. The agent runs as normal. This happens in these cases:

The SDK never falls back to localhost. It never falls back to ArmorIQ. It never picks the next endpoint setting if the first one is invalid. This is on purpose. A silent change of destination could send data to the wrong place.

To fix it, pass a provider, or set a valid endpoint, as in Choose who exports your traces. To confirm that traces can reach ArmorIQ, set ARMORIQ_OTEL_EXPORTER=armoriq and run the agent again.

Other things to know in provider mode:

• Your provider needs its own span processor and exporter. The SDK adds neither. Spans that your provider never exports are lost.
• The SDK never flushes your provider. Flush it yourself before the process stops.
• Capture is limited to metadata, so prompts and tool content are empty. This is not a bug.

Tracing is switched off

Tracing can be off for one of these reasons:

ArmorCodex

Set ARMORCODEX_OBSERVABILITY_DISABLED=true. The configuration bridge also honors a forwarded disable_observability plugin option, although the current packaged plugin metadata may not expose that option in a configuration UI. The environment variable is the reliable customer-facing opt-out. Observability is also off when no API key is configured.

ArmorClaude

Set the disable_observability plugin option or ARMORIQ_OBSERVABILITY_DISABLED=true. Observability normally requires an API key or local-mock mode.

Roll back

Roll back in small steps. Stop at the first step that fixes the problem. None of these steps changes a policy decision, and the SDK never switches to another path by itself.

1. Turn tracing off for one session. Use the per-session switch in Tracing is switched off.
2. Turn tracing off for the process. Set ARMORIQ_OBSERVABILITY=off, then restart the process.
3. Go back to the default export. Remove ARMORIQ_OTEL_EXPORTER, the OTEL_EXPORTER_OTLP_* variables, or the observability options that choose external or provider. Traces then go to ArmorIQ only.
4. Turn collection off for the organization. An administrator sets the capture mode to Off in Settings > Observability. SDKs stop creating ArmorIQ spans when they next read the policy (within 5 minutes, or sooner). To lower the data that you keep, choose Metadata instead.
5. Operator only: stop ingest. Set OBSERVABILITY_OTLP_INGEST_ENABLED=false to stop storing new telemetry while the backend still answers success. Set OTEL_CANONICAL_WRITER_ENABLED=false to stop writes. Stored data is not changed or deleted.
6. Pin an older SDK (last resort). An older SDK uses the old route. It works only while the backend still accepts that route. See Migrate from the legacy recorder.

A rollback does not delete stored traces, take back data that you already exported, or turn off policy enforcement. To delete stored traces, use Delete data. See Organization controls.

Wrong product or agent name

ArmorIQ takes the product from your API key, not from the spans. A product-specific key labels all its telemetry with that product.

The dashboard shows a name from the agent ID. If no match exists, it shows the raw ID. Set agent_id on the Python ArmorIQClient, or agentId on the TypeScript ArmorIQClient. In Python, the SDK also uses the agent ID as the OpenTelemetry service name unless you set service_name.

Tokens or cost are missing

• Model spans carry tokens. The framework adapters add them from the usage that the framework reports. A direct openai or anthropic call adds them only while an ArmorIQ session is active. In Python, streaming direct calls are not traced.
• In Python, you can add usage by hand with session.record_generation(model=..., input_tokens=..., output_tokens=...).
• ArmorIQ prices usage when it receives the span. An SDK sends a cost only when the provider reported one. When no cost is known, the dashboard shows Not reported, not $0.00.
• A stream without usage. For a TypeScript OpenAI streaming request, the SDK asks for usage in the stream by default. If your gateway rejects that option, set ARMORIQ_OTEL_STREAM_USAGE=0. Usage then cannot be read from the stream.

Slow delivery and failed exports

The SDK sends spans in batches. It does not change your agent result if a send fails.

• Python makes up to 6 attempts for a send, with a growing wait, but stops when the export timeout (10 seconds by default) is used up. It retries 429, 502, 503, 504, and connection errors. Any other answer is final. A TLS error is not retried.
• TypeScript retries up to 5 times for 429, 502, 503, 504, and common network errors. It respects Retry-After. The whole send is limited by the export timeout (10 seconds by default). Any other 4xx or 5xx answer is final.
• After the last try, the SDK drops the batch and logs a warning. It does not store it on disk and does not send it by another route.
• Closing a session waits for a short, fixed time: up to 3 seconds in Python and 1.5 seconds in TypeScript. A slow destination can delay the close by that time at most.

To raise the timeout, see the settings reference in Choose who exports your traces.

Dashboard refresh delay

The dashboard asks for new data every 30 seconds. It does not use a live stream. If you can read a trace through the API but not on the page, wait for the next poll or reload the page.

Duplicate spans

A duplicate span count usually means that a send was retried after ArmorIQ had already stored the batch. ArmorIQ keeps each span one time, matched by organization, product, trace ID, and span ID. Duplicates do not change totals. You do not need to clean up.

You can also see two spans for one model call when two tools trace the same call. In LangChain, the attribute armoriq.observability.native_span.state = coexisting on the root span tells you that another tool is also tracing the run. In TypeScript, a raw openai or anthropic call under a framework adapter can add a second model span. See Framework support.

Missing or partial tool input and output

Content is off in metadata mode, and it is limited, redacted, and bounded in enhanced and debug. If the span shows a reason such as capture_mode_metadata or organization_content_disabled, the system works as set. See What data is captured.

Safe debugging

Log only:

• the SDK name and version;
• the backend host, without query parameters;
• trace, session, and span IDs;
• the HTTP status;
• accepted, rejected, and duplicate counts;
• the exception class and a bounded message.

Do not log:

• API keys, JWTs, authorization headers, cookies, or passwords;
• full prompts, completions, tool inputs, or tool outputs;
• credential files;
• full OTLP payloads in shared logs.

When you must look at a payload, make it again with synthetic values. Use the data model and the API reference as the expected contract.

Next steps

• Set up tracing
• Organization controls
• Framework support