Observability

Backend API Reference

Send OTLP traces, read the SDK policy lease, manage organization controls, and query organization-scoped sessions, traces, policies, summaries, and tokens.

This page is for developers and operators who call the ArmorIQ backend directly. Use the OTLP route to send traces, and the organization routes to read telemetry and manage controls. SDKs and Collectors normally send traces for you.

Authentication and organization scope

APIAuthenticationOrganization selection
POST /v1/tracesAPI key in x-api-key (or Authorization: Bearer)Resolved from the API key
GET /observability/policy/leaseAPI key in x-api-keyResolved from the API key
POST /observability/spans (legacy)API key in x-api-keyResolved from the API key
All /organization/:orgId/observability/... routesBearer JWTorgId path parameter plus a membership check

The organization and product always come from the API key. A request body or a span attribute cannot choose another organization, product, or user. A key with no organization gets 400. A key with no product gets 403 with relogin_required.

Each organization route needs a permission. The table below names it. See Organization controls for what each permission allows.

Endpoints

MethodPathPermissionPurpose
POST/v1/tracesAPI keyIngest OTLP traces
GET/observability/policy/leaseAPI keyGet the SDK policy lease
POST/observability/spansAPI keyLegacy JSON ingest, for older SDKs only
GET/organization/:orgId/observability/policyMetadata readGet the organization policy
GET/organization/:orgId/observability/policy/effectiveMetadata readGet the effective policy
PATCH/organization/:orgId/observability/policySettings manageReplace the organization policy
GET/organization/:orgId/observability/policy/scopesMetadata readList scope overrides
PATCH/organization/:orgId/observability/policy/scopes/:scopeType/:scopeIdSettings manageSave a scope override
DELETE/organization/:orgId/observability/policy/scopes/:scopeType/:scopeIdSettings manageRemove a scope override
GET/organization/:orgId/observability/usageMetadata readGet quota use
GET/organization/:orgId/observability/pipeline-healthMetadata readGet ingest counters
GET/organization/:orgId/observability/alertsAlert readList quota warnings
GET, POST/organization/:orgId/observability/alert-rulesAlert read, Alert manageList and create alert rules
PATCH, DELETE/organization/:orgId/observability/alert-rules/:ruleIdAlert manageChange or disable a rule
GET/organization/:orgId/observability/alert-rules/:ruleId/firingsAlert readList firings of a rule
GET/organization/:orgId/observability/auditSettings manage (view)List change history
GET/organization/:orgId/observability/execution-storyContent readGet one normalized execution story
GET/organization/:orgId/observability/sessionsMetadata readList sessions
GET/organization/:orgId/observability/sessions/:idContent readGet one session with traces
GET/organization/:orgId/observability/tracesMetadata readList traces
GET/organization/:orgId/observability/traces/:idContent readGet one trace with spans
GET/organization/:orgId/observability/policies/:policyIdMetadata readAggregate one policy
GET/organization/:orgId/observability/analytics/summaryMetadata readGet overview metrics
GET/organization/:orgId/observability/tokensMetadata readGroup token and cost usage
POST/organization/:orgId/observability/exportExport create and content readStart an encrypted export
GET/organization/:orgId/observability/export/:jobIdExport create and content readGet export status
GET/organization/:orgId/observability/export/:jobId/downloadExport create and content readDownload the export
DELETE/organization/:orgId/observability/dataDelete executeDelete traces

Ingest traces (OTLP)

POST /v1/traces
x-api-key: YOUR_ARMORIQ_API_KEY
Content-Type: application/x-protobuf
Content-Encoding: gzip

The body is a standard OTLP ExportTraceServiceRequest in protobuf. You can send it uncompressed or with gzip. SDKs and Collectors do this for you. See Deploy an OpenTelemetry Collector for a working configuration.

Limits

RuleValue
Content typeapplication/x-protobuf only. Others, including OTLP/JSON, get 415.
Content encodingNone, or gzip. Others get 415.
Body as sentAt most 1,000,000 bytes. More gets 413.
Body after decompressionAt most 4,000,000 bytes. More gets 413.
Resource blocks in a requestAt most 64
Scope blocks in a resourceAt most 64
Spans in a scopeAt most 500
Spans in a requestAt most 2,000
Traces in a requestAt most 500
AttributesAt most 64 on a resource and on a span, 32 on a scope, an event, and a link
Events and links in a spanAt most 32 each
One stringAt most 4,096 bytes. A longer content value is cut.
Rate120 requests each minute for each API key

The body size, content type, and gzip checks run before the API key check. If a request has more than 64 resource blocks, more than 64 scope blocks in a resource, or more than 500 spans in one scope, ArmorIQ rejects the whole request with 400. For the other bounds, ArmorIQ rejects only the spans that they affect and counts them as rejected.

Responses

StatusMeaning
200The request is accepted. The body is an OTLP response. If ArmorIQ rejected some spans for a reason that a retry cannot fix, the body has partial_success with rejected_spans and a short error_message.
200 (empty body)Capture is Off for the organization or product, or ingest is switched off. Nothing was stored.
400The payload is malformed, empty, or over a structure bound
401The API key is missing, not valid, or expired
403relogin_required: the key has no product, or it is a tool-product key outside a personal workspace
413The body is too large
415Wrong content type or encoding
429A rate limit, or the monthly quota. For quota, the response has Retry-After: 60 and a partial_success body.
503Storage or the policy store is not available for a short time. Retry.

Duplicate behavior

ArmorIQ stores each span one time. It matches on organization, product, trace ID, and span ID. If a batch arrives again, ArmorIQ counts the repeats as duplicates and does not apply them twice. A request that was retried after a 429, a 503, or a network error is safe. A quota reservation for a retry is also recognized and not charged twice.

Get the SDK policy lease

GET /observability/policy/lease
x-api-key: YOUR_ARMORIQ_API_KEY

The SDK calls this route to learn what it may capture. The response has revision, captureMode, debugExpiresAt, contentCaptureAllowed, externalContentCaptureAllowed, externalContentAllowed, contentReasonCode, expiresAt, and sometimes generation. A lease lasts at most 5 minutes, or until the debug window ends if that is sooner. If ArmorIQ cannot read the policy, it returns a short metadata lease with all content flags off and contentReasonCode set to policy_unavailable.

The lease is only a hint for the SDK. ArmorIQ does not trust the SDK for policy. It reads the organization policy again, with a row lock, in the same database transaction as each write, and it redacts the data again.

Organization policy and scopes

PATCH /organization/:orgId/observability/policy replaces the whole policy. Send every field:

FieldMeaning
expectedRevisionThe revision that you loaded, at least 1. A different current revision gives 409.
captureModeoff, metadata, enhanced, or debug
debugExpiresAtRequired for debug: a future time, at most 24 hours ahead. It must be null for any other mode. You cannot make an active window longer.
contentCaptureAllowedAllow content in enhanced and debug
externalContentCaptureAllowedAllow content to go to an external destination. It needs the external export permission.
monthlySpanQuota, monthlyStorageBytesQuotaMonthly limits. Safe whole numbers, sent as numbers and returned as strings.
samplingRateA whole number from 0 to 100, or null
retentionDays, legalHoldRetention settings. A legal hold change needs the legal hold permission.

A change to the debug window needs the debug permission, as well as the settings permission.

PATCH .../policy/scopes/:scopeType/:scopeId saves an override. scopeType is product, environment, or agent. Send captureMode and, after the first save, expectedRevision. A scope can only lower the mode. Only the product scope is enforced at ingest. ArmorIQ applies the product scope that matches the product of the API key.

GET .../policy/effective returns what applies now:

  • organization: captureMode, effectiveCaptureMode, and revision;
  • scopes: for each, scopeType, scopeId, captureMode, effectiveCaptureMode, revision, and enforced;
  • switches: otlpIngest, enhancedContent, debugCapture, and leaseDelivery, the operator switches.

Usage, pipeline health, and alerts

  • GET .../usage accepts an optional product. It returns the period start and, for the organization (and for the product if you ask), the committed and reserved counts and the limit for spans and storage, as strings, plus warnings. A warning is span_quota_80_percent or storage_quota_80_percent.
  • GET .../pipeline-health needs from and to and accepts product. The range is at most 31 days. It returns requestCount, failedRequestCount, acceptedSpans, rejectedSpans, duplicateSpans, mappingFailureSpans, avgExportLatencyMs, maxExportLatencyMs, effectiveFrom, and effectiveTo. effectiveFrom and effectiveTo can be wider than your range for older data.
  • POST .../alert-rules takes conditionType, threshold, windowMinutes (1 to 1,440), and optionally severity (info, warning, critical) and enabled. The condition types are policy_denial_rate, tool_error_rate, model_error_rate, p95_latency, token_cost_budget, token_count_budget, retry_rate, timeout_rate, exporter_ingest_rejection_rate, and missing_telemetry. DELETE disables the rule. It does not remove it.
  • A firing has id, ruleId, openedAt, lastSeenAt, resolvedAt, value, lastValue, and sample IDs of the source traces.
  • GET .../audit accepts limit (at most 200) and includeLeases. Each entry has auditEventId, actorType, actorId, product, action, beforeSummary, afterSummary, and createdAt.

Execution story, export, and delete

  • GET .../execution-story finds one normalized execution story. Give one of traceId, otelTraceId (32 lower-case hexadecimal characters), sessionId, conversationId, workflowRunId, agentId, product, or risTicketId.
  • POST .../export starts an encrypted export. Send limit (1 to 100 traces). It answers 202 with a jobId. At most 2 exports can be open for an organization. Poll GET .../export/:jobId, then call the download route.
  • DELETE .../data takes traceIds, 1 to 100 UUID version 4 values. It answers with requested, deleted, status, and invalidatedExports. It answers 409 while a legal hold is active.

Legacy ingest

POST /observability/spans takes the older JSON record that older SDKs send. ArmorIQ keeps the route open for a limited time so that clients that you have not upgraded still deliver data. It uses the same policy, redaction, quota, and storage rules as OTLP. The current SDKs (armoriq-sdk 0.8.17, @armoriq/sdk 0.8.23) never call it, and no SDK falls back to it. Do not start new work on it. See Migrate from the legacy recorder.

A request has product and up to 500 batches. Each batch has one trace and its spans. The response is 202 with accepted, duplicate, rejected, and traceIds. If capture is Off, the response also has captureDisabled: true. The fields are in the data model.

List sessions

GET /organization/ORG_ID/observability/sessions?days=7&limit=50&status=active
Authorization: Bearer YOUR_JWT

Query parameters:

ParameterTypeDefaultConstraints
daysinteger71–90
limitinteger501–200
cursorstringnoneOpaque cursor from the previous page
statusstringnoneactive, completed, expired, or error
userIdUUIDnoneExact match
agentIdstringnoneExact match
{
  "items": [
    {
      "id": "22222222-2222-4222-8222-222222222222",
      "orgId": "44444444-4444-4444-8444-444444444444",
      "userId": null,
      "agentId": "support-agent",
      "product": "armoriq-sdk",
      "startedAt": "2026-07-17T09:00:00.000Z",
      "lastActivityAt": "2026-07-17T09:00:00.240Z",
      "traceCount": 1,
      "totalInputTokens": 420,
      "totalOutputTokens": 96,
      "totalCacheReadTokens": 0,
      "totalCacheWriteTokens": 0,
      "totalCostUsd": 0.00201,
      "status": "active"
    }
  ],
  "nextCursor": "OPAQUE_CURSOR",
  "hasMore": true
}

nextCursor is omitted on the final page.

Get a session

GET /organization/ORG_ID/observability/sessions/SESSION_ID
Authorization: Bearer YOUR_JWT

The response contains every session field plus a chronological traces array. Each trace includes the base trace record and list metrics:

{
  "id": "22222222-2222-4222-8222-222222222222",
  "orgId": "44444444-4444-4444-8444-444444444444",
  "userId": null,
  "agentId": "support-agent",
  "product": "armoriq-sdk",
  "startedAt": "2026-07-17T09:00:00.000Z",
  "lastActivityAt": "2026-07-17T09:00:00.240Z",
  "traceCount": 1,
  "totalInputTokens": 420,
  "totalOutputTokens": 96,
  "totalCacheReadTokens": 0,
  "totalCacheWriteTokens": 0,
  "totalCostUsd": 0.00201,
  "status": "active",
  "traces": [
    {
      "id": "11111111-1111-4111-8111-111111111111",
      "sessionId": "22222222-2222-4222-8222-222222222222",
      "name": "iap.plan",
      "startTime": "2026-07-17T09:00:00.000Z",
      "endTime": "2026-07-17T09:00:00.240Z",
      "durationMs": 240,
      "status": "ok",
      "userId": null,
      "agentId": "support-agent",
      "attributes": { "input": "Find the account owner" },
      "tags": ["armoriq-sdk", "allowed"],
      "totalInputTokens": 420,
      "totalOutputTokens": 96,
      "totalCostUsd": 0.00201,
      "spanCount": 1,
      "errorCount": 0,
      "inputPreview": "Find the account owner",
      "outputPreview": null
    }
  ]
}

The session-detail trace array is capped at 50,000 rows. If a session exceeds the cap, the response keeps the most recent traces and the backend logs a warning.

If the session ID does not exist in the organization, the current endpoint returns HTTP 200 with JSON null, not 404.

List traces

GET /organization/ORG_ID/observability/traces?days=7&limit=100&status=error&minDurationMs=500
Authorization: Bearer YOUR_JWT
ParameterTypeDefaultConstraints
daysinteger71–90
limitinteger501–200
cursorstringnoneOpaque cursor from the previous page
sessionIdUUIDnoneExact match
namestringnoneExact trace-name match
statusstringnoneok, error, or denied
userIdUUIDnoneExact match
agentIdstringnoneExact match
minDurationMsintegernoneMinimum duration, inclusive
hasErrorbooleannonetrue selects non-ok; false selects ok

hasError overrides status when both are supplied.

The response uses the same cursor envelope as sessions. Trace items contain the base trace fields plus totalInputTokens, totalOutputTokens, totalCostUsd, spanCount, errorCount, inputPreview, and outputPreview.

Get a trace

GET /organization/ORG_ID/observability/traces/TRACE_ID
Authorization: Bearer YOUR_JWT
{
  "id": "11111111-1111-4111-8111-111111111111",
  "sessionId": "22222222-2222-4222-8222-222222222222",
  "name": "iap.plan",
  "startTime": "2026-07-17T09:00:00.000Z",
  "endTime": "2026-07-17T09:00:00.240Z",
  "durationMs": 240,
  "status": "ok",
  "userId": null,
  "agentId": "support-agent",
  "attributes": { "input": "Find the account owner" },
  "tags": ["armoriq-sdk", "allowed"],
  "product": "armoriq-sdk",
  "source": "sdk",
  "spans": [
    {
      "id": "33333333-3333-4333-8333-333333333333",
      "parentSpanId": null,
      "sessionId": "22222222-2222-4222-8222-222222222222",
      "kind": "generation",
      "name": "llm.generate",
      "startTime": "2026-07-17T09:00:00.050Z",
      "endTime": "2026-07-17T09:00:00.200Z",
      "durationMs": 150,
      "status": "ok",
      "attributes": {
        "kind": "generation",
        "model": "gpt-4o",
        "inputTokens": 420,
        "outputTokens": 96,
        "cacheReadTokens": 0,
        "cacheWriteTokens": 0,
        "costUsd": 0.00201,
        "prompt": null,
        "completion": null,
        "finishReason": "stop"
      }
    }
  ]
}

If the trace ID does not exist in the organization, the current endpoint returns HTTP 200 with JSON null, not 404.

Get policy aggregates

GET /organization/ORG_ID/observability/policies/contacts-read?days=7
Authorization: Bearer YOUR_JWT
{
  "policyId": "contacts-read",
  "policyName": "Contacts read policy",
  "calls": 120,
  "allowed": 115,
  "denied": 5,
  "avgDurationMs": 18,
  "p50DurationMs": 14,
  "p95DurationMs": 42,
  "topMatchedRules": [
    { "ruleId": "allow-support", "count": 115 },
    { "ruleId": "deny-untrusted", "count": 5 }
  ]
}

days defaults to 7 and accepts 1–90. calls includes allow, deny, hold, and ask. The allowed subtotal counts only allow, while denied counts only literal deny; hold and ask appear in neither subtotal. Summary topFailingPolicies likewise counts literal deny decisions only.

Get the analytics summary

GET /organization/ORG_ID/observability/analytics/summary?days=7
Authorization: Bearer YOUR_JWT
{
  "period": 7,
  "totalTraces": 980,
  "totalSessions": 82,
  "totalErrors": 12,
  "totalTokens": {
    "input": 250000,
    "output": 64000,
    "cacheRead": 18000,
    "cacheWrite": 4000
  },
  "totalCostUsd": 4.812345,
  "p50DurationMs": 120,
  "p95DurationMs": 920,
  "topFailingPolicies": [
    { "policyId": "contacts-read", "count": 5 }
  ],
  "tracesPerMinute": 1.7
}

tracesPerMinute uses a trailing 60-minute window, clamped to the requested period. The other metrics use the requested days period.

Get token aggregates

GET /organization/ORG_ID/observability/tokens?days=7&groupBy=model
Authorization: Bearer YOUR_JWT

groupBy accepts model, tool, or session and defaults to model.

{
  "groupBy": "model",
  "items": [
    {
      "key": "gpt-4o",
      "inputTokens": 250000,
      "outputTokens": 64000,
      "cacheReadTokens": 18000,
      "cacheWriteTokens": 4000,
      "costUsd": 4.812345
    }
  ]
}

Model and session groups aggregate generation spans. Tool groups read generic span records and use attributes.toolName; tools without generation attributes can therefore have zero token totals.

Pagination

Session and trace lists use cursor pagination ordered by timestamp and ID. Treat nextCursor as opaque and send it unchanged in the next request. A malformed cursor returns 400 Invalid cursor.

Retention and aggregation limits

  • Default retention is 30 days.
  • OBSERVABILITY_RETENTION_DAYS changes retention on the backend.
  • A UTC retention job runs daily at 03:00 and deletes old traces; their spans are cascade-deleted. Old sessions are deleted by lastActivityAt.
  • In-memory analytics queries scan at most 50,000 rows. A result that reaches this cap is approximate and logs a backend warning.

Next: use the dashboard or troubleshoot API and shipping failures.

On this page