Observability

What is OpenTelemetry?

OpenTelemetry ↗ (OTel) is vendor-neutral. Instrument your service once against the OTel SDK, and any compatible backend can consume the data. That includes Grafana, Datadog, Jaeger, Honeycomb, Tempo, and New Relic. Aileron emits spans the same way any other OTel-instrumented Go service does. If you’ve used OTel before, the shape is familiar.

The terms that show up on the rest of this page:

• Span: A timed unit of work with a name, attributes (key/value tags), and a parent. A trace is the tree of spans for one logical request.
• traceparent: The W3C TraceContext ↗ HTTP header that carries trace and parent-span IDs across service boundaries. It keeps multi-service requests connected end-to-end, regardless of which language or framework each service is written in.
• OTLP: The OpenTelemetry Protocol, the wire format collectors expect.
• OTel endpoint: The URL of an OpenTelemetry Collector ↗ (or a managed backend’s ingest URL) that receives OTLP-encoded spans. The collector fans spans out to whichever backend you’ve configured. Multiple backends, no per-language SDK churn.
• Exporter: The component that ships spans out of the process. Aileron supports noop (the default; drops spans, zero overhead), stdout (writes JSON-per-line to stderr for local development), file (writes JSON-per-line to a daily-rotated file under ~/.aileron/traces/), and otlp (ships to a collector via OTLP/HTTP).
• Span status: Ok (default), Error, or Unset. Aileron sets Error on any span whose underlying operation failed, with the failure message as the status description.
• Resource: Process-level metadata attached to every span. Aileron sets service.name=aileron (configurable via AILERON_OTEL_SERVICE_NAME).

The audit log (always on)

Every load-bearing decision in the runtime emits a structured audit record. The audit log is the contract that Proof of Control builds on. The records live as daily-rotated JSONL files at ~/.aileron/audit/audit-YYYY-MM-DD.jsonl and are queryable through the CLI:

aileron audit list             # newest events first
aileron audit get <audit-id>   # full event by id

Today, six families of events land in the log:

• Install consent: Every connector and action install records artifact FQN, version, hash, signature status, and the user’s decision (ADR-0007).
• Action execution: Every invocation records which connector it called, which capability it exercised, and which binding identity satisfied it (ADR-0003, ADR-0011). Credential bytes are never recorded.
• Failure: Every failure surfaces with a stable class, boundary, retry, and audit_id (ADR-0010). The same audit_id is stamped onto the agent-visible tool-result envelope, so the LLM’s “what went wrong?” reaction can be traced back to a specific event.
• Approval lifecycle: Three event types: approval.requested, approval.approved, approval.denied. Each carries the same aileron.approval.id so a request and its decision are trivially correlated.
• Flight Plan launch provenance: A Flight Plan launch records its provenance to the audit trail. The flightplan.launch summary and the per-action flightplan.launch.action records name the actions the plan dispatched. Each materialized artifact surfaces as an output.materialized record carrying the output’s content hash, mime, byte count, and originating-step provenance; a tool-materialized output also records the executed argv as aileron.step.command rather than a per-step image, because the plan runs in one composed image already named on the launch record. Each tool step that declares a per-step trust contract surfaces a flightplan.launch.reach record naming the step’s declared network reach and whether it was enforced. When the reach is the verified lock’s sealed stepTrust and the step ran under a step-scoped proxy credential restricted to exactly those hosts, the record is marked enforced: true; a scoped call to an undeclared host is refused with a 403 at the daemon proxy before the TLS handshake and recorded as sandbox.proxy.trust_denied (ADR-0019, ADR-0027). enforced: true is a proxy-credential guarantee, not a network-layer egress guarantee: it attests the step ran under a step-scoped proxy credential restricted to its sealed reach. A non-cooperative subprocess that ignores HTTPS_PROXY and dials a raw socket directly is out of scope for this marker.
• Sandbox HTTPS data plane: Generated connector shims and transparent sandbox proxy requests emit proxy audit events. connector.proxy.proxied and connector.proxy.rejected identify the resolved connector operation, upstream scheme/host/path, decision, proxy source, and response status or rejection reason. sandbox.proxy.passthrough records cooperative HTTPS requests whose upstream did not uniquely match an installed connector spec; the proxy forwards them unmodified and audits the upstream host, path, method, and response status. sandbox.proxy.upgrade records WebSocket (and other HTTP Upgrade) handshakes forwarded through the passthrough boundary; the proxy preserves the upgrade headers, relays the upstream 101 Switching Protocols, and opens a bidirectional byte tunnel. sandbox.proxy.rejected records protocol-level failures in the transparent proxy path (non-CONNECT request, missing session CA, connector specs invalid or unavailable, upstream unreachable during passthrough). sandbox.proxy.binding_injected records a bundled-CLI request that matched a user-level host binding and was re-issued upstream with the bound credential injected; when the binding carries a per-step trust contract the record also names the declared effect and the plan/step/tool identity. sandbox.proxy.trust_denied records a bundled-CLI request that matched a host binding carrying a declared trust contract but violated it (an upstream host outside the binding’s allowlist, or a mutating method against a read-effect contract); the request is denied with a 403 before any credential is resolved or injected, and is never passed through. sandbox.proxy.disabled records launch sessions that start with the v4 HTTPS proxy not in force (operator opt-out, preflight failure, or unsupported sandbox mode). These events never record credential bytes, request bodies, raw headers, query strings, full upstream URLs, or the binding’s allowed-host values.

The schema is durable. Every payload field uses the OpenTelemetry-namespaced key shape (aileron.connector.fqn, aileron.binding.name, aileron.failure.class, etc.). Consumers (log shippers, trace tools, custom queries) read the same vocabulary regardless of which surface they came in through.

Flight Plan declared reach (flightplan.launch.reach):

One record is emitted per tool step whose per-step trust contract declares a reach. A step that declares no contract emits no reach record.

OpenTelemetry traces (opt-in)

When tracing is enabled, Aileron starts a server-root span on every request and child spans for the work inside. The child spans cover action execution, connector calls, capability checks, and approval waits. Spans propagate via W3C TraceContext ↗. An inbound traceparent header from the calling agent makes Aileron’s spans children of the agent’s trace, so your end-to-end view stays coherent. With tracing off (the default), there’s zero SDK overhead. The call sites resolve to no-op tracers. The W3C propagator is installed regardless, so an inbound traceparent is parsed and forwarded even when this process emits nothing.

Three ways to consume traces

stdout: Local debugging. Spans land on stderr as JSON-per-line. Pipe to jq:

AILERON_OTEL_ENABLED=true \
AILERON_OTEL_EXPORTER=stdout \
aileron launch claude

file: Durable retention across sessions, mirroring the audit log’s on-disk layout:

AILERON_OTEL_ENABLED=true \
AILERON_OTEL_EXPORTER=file \
aileron launch claude
# spans → ~/.aileron/traces/spans-YYYY-MM-DD.jsonl

A new file is created per local-clock day. A session that crosses midnight rolls naturally to the next day’s file. AILERON_TRACES_DIR overrides the state directory. The default (~/.aileron) keeps audit and traces side by side.

otlp: Production. Ships spans to an OpenTelemetry Collector via OTLP/HTTP. See the next section.

Hooking up to a collector

The OTLP exporter honors the standard OTel environment variables ↗ that every OTel-instrumented service in your stack already understands. There’s no Aileron-prefixed alternative. Forking the names would force you to maintain two parallel sets.

Stand up a collector locally for development:

docker run --rm -p 4318:4318 \
  otel/opentelemetry-collector-contrib:latest

Then point Aileron at it:

AILERON_OTEL_ENABLED=true \
AILERON_OTEL_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
OTEL_EXPORTER_OTLP_INSECURE=true \
aileron launch claude

For a managed backend, point at its ingest endpoint and pass auth via OTEL_EXPORTER_OTLP_HEADERS:

# Honeycomb
AILERON_OTEL_ENABLED=true \
AILERON_OTEL_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io \
OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=YOUR_API_KEY \
aileron launch claude

# Grafana Cloud
AILERON_OTEL_ENABLED=true \
AILERON_OTEL_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-central-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64(instanceID:token)>" \
aileron launch claude

Recognised env vars (handled by the OTel SDK directly):

• OTEL_EXPORTER_OTLP_ENDPOINT: Collector URL. Defaults to http://localhost:4318.
• OTEL_EXPORTER_OTLP_HEADERS: Comma-separated k=v pairs added to every export request. Use this for API keys.
• OTEL_EXPORTER_OTLP_INSECURE: Set to true to allow plain HTTP (development only).
• OTEL_EXPORTER_OTLP_TIMEOUT: Request timeout. Default 10s.

The full set is in the OTel exporter spec ↗.

What gets emitted

Span attribute schema

Every span carries the OTel-namespaced shape locked in for the audit payload. When you query traces by attribute, you query the same names you’d query the audit log by. This table is the source of truth for what’s available.

Action execution (aileron.action.execute):

Capability check (aileron.capability.check):

Connector call (aileron.connector.call):

Sandbox HTTPS data plane (connector.proxy.proxied, connector.proxy.rejected, sandbox.proxy.passthrough, sandbox.proxy.upgrade, sandbox.proxy.rejected, sandbox.proxy.binding_injected, sandbox.proxy.trust_denied audit events):

The proxy mediates credential injection at the TLS boundary. It is not a blanket egress allowlist: cooperative HTTPS requests whose decrypted target does not uniquely match an installed connector spec and matches no host binding are forwarded to the upstream unmodified and recorded as sandbox.proxy.passthrough. A host binding that carries a declared per-step trust contract is the exception: it DOES enforce a per-binding host and effect gate at the injection point. When a bundled-CLI request matches such a binding, the proxy checks the upstream host against the binding’s allowed-host list (empty means unconstrained, scoped only to the matched host pattern) and the HTTP method against the declared effect (a read effect admits only safe methods; a write-class effect admits all methods). A request that fails either check is denied with a 403 and recorded as sandbox.proxy.trust_denied before any credential is resolved or injected; it is never passed through. The proxy sees only method and host on the wire, so it cannot distinguish write from delete/spend/external-send; finer effect narrowing is enforced upstream at the runtime action-call approval seam. WebSocket (and other HTTP Upgrade) handshakes are forwarded through the same passthrough boundary with their Upgrade/Connection/Sec-WebSocket-* headers preserved; the proxy relays the upstream’s 101 Switching Protocols response and opens a bidirectional byte tunnel, recorded as sandbox.proxy.upgrade. No credential is injected on the passthrough or upgrade paths. sandbox.proxy.rejected is reserved for protocol-level failures. See ADR-0019 for the full credential-injection model and the threat-model scope.

Migration note: operators who previously queried sandbox.proxy.rejected with reason operation_not_matched or ambiguous_operation_match to surface “agent attempted unknown upstream” should switch to querying sandbox.proxy.passthrough. Those two reject reasons are no longer emitted in production. The cooperative-passthrough behavior records the same operational signal as a new event family.

sandbox.proxy.disabled (launch-session start without the v4 HTTPS proxy):

The event is emitted once per launch session by the aileron launch CLI through the daemon’s POST /v1/sandbox-proxy/disabled endpoint. It never records credential bytes, environment variables, the BYO image’s contents, or shell history. See ADR-0019 for the v4 HTTPS data plane decision and the --sandbox-proxy flag semantics.

Redaction rules

Every sandbox HTTPS data-plane audit event is built from a fixed field set. The recorders copy only the fields named in the tables above into the payload. The following inputs are never recorded in any event family, and the no-leak invariant is enforced in code, not by convention alone.

This redaction contract is pinned in code by the cross-family no-leak sweep in internal/app/sandbox_proxy_audit_shape_test.go ↗ (TestSandboxProxyAuditShape_NoCredentialLeakAcrossFamilies). The sweep drives every emitter with credential-shaped inputs (an Authorization: Bearer header, a secret-shaped header, and a query string on the upstream URL) and asserts no serialized payload contains them. That test is the authoritative definition of the no-leak rule; this prose summarizes it.

Approval wait (aileron.approval.wait):

Failure (any error span). From the closed taxonomy in ADR-0010:

When a span fails, the OTel span status is also set to Error with the failure message. Your tracing UI’s red flags work without parsing attributes.

Configuration

All Aileron-side knobs are environment variables read at daemon startup. Defaults reproduce the historic behavior: tracing off, audit on. The OTEL_EXPORTER_OTLP_* family is consumed directly by the OTel SDK and only matters when AILERON_OTEL_EXPORTER=otlp.

A misconfigured exporter degrades gracefully to no-op rather than failing daemon startup. This applies to unknown exporter names and to known exporters whose construction fails. The Aileron HTTP server keeps serving when its telemetry sidecar is misconfigured. The failure is logged at warn level so you find it without it taking the daemon down.