Observability

What is OpenTelemetry?

OpenTelemetry ↗ (OTel) is vendor-neutral. Instrument your service once against the OTel SDK, and any compatible backend can consume the data. That includes Grafana, Datadog, Jaeger, Honeycomb, Tempo, and New Relic. Aileron emits spans the same way any other OTel-instrumented Go service does. If you’ve used OTel before, the shape is familiar.

The terms that show up on the rest of this page:

• Span: A timed unit of work with a name, attributes (key/value tags), and a parent. A trace is the tree of spans for one logical request.
• traceparent: The W3C TraceContext ↗ HTTP header that carries trace and parent-span IDs across service boundaries. It keeps multi-service requests connected end-to-end, regardless of which language or framework each service is written in.
• OTLP: The OpenTelemetry Protocol, the wire format collectors expect.
• OTel endpoint: The URL of an OpenTelemetry Collector ↗ (or a managed backend’s ingest URL) that receives OTLP-encoded spans. The collector fans spans out to whichever backend you’ve configured. Multiple backends, no per-language SDK churn.
• Exporter: The component that ships spans out of the process. Aileron supports noop (the default; drops spans, zero overhead), stdout (writes JSON-per-line to stderr for local development), file (writes JSON-per-line to a daily-rotated file under ~/.aileron/traces/), and otlp (ships to a collector via OTLP/HTTP).
• Span status: Ok (default), Error, or Unset. Aileron sets Error on any span whose underlying operation failed, with the failure message as the status description.
• Resource: Process-level metadata attached to every span. Aileron sets service.name=aileron (configurable via AILERON_OTEL_SERVICE_NAME).

The audit log (always on)

Every load-bearing decision in the runtime emits a structured audit record. The audit log is the contract that Proof of Control builds on. The records live as daily-rotated JSONL files at ~/.aileron/audit/audit-YYYY-MM-DD.jsonl and are queryable through the CLI:

aileron audit list             # newest events first
aileron audit get <audit-id>   # full event by id

Today, five families of events land in the log:

• Install consent: Every connector and action install records artifact FQN, version, hash, signature status, and the user’s decision (ADR-0007).
• Action execution: Every invocation records which connector it called, which capability it exercised, and which binding identity satisfied it (ADR-0003, ADR-0011). Credential bytes are never recorded.
• Failure: Every failure surfaces with a stable class, boundary, retry, and audit_id (ADR-0010). The same audit_id is stamped onto the agent-visible tool-result envelope, so the LLM’s “what went wrong?” reaction can be traced back to a specific event.
• Approval lifecycle: Three event types: approval.requested, approval.approved, approval.denied. Each carries the same aileron.approval.id so a request and its decision are trivially correlated.
• Sandbox HTTPS data plane: Generated connector shims and transparent sandbox proxy requests emit proxy audit events. connector.proxy.proxied and connector.proxy.rejected identify the resolved connector operation, upstream scheme/host/path, decision, proxy source, and response status or rejection reason. sandbox.proxy.rejected records transparent proxy attempts that fail before a connector operation is uniquely resolved. These events never record credential bytes, request bodies, raw headers, query strings, or full upstream URLs.

The schema is durable. Every payload field uses the OpenTelemetry-namespaced key shape (aileron.connector.fqn, aileron.binding.name, aileron.failure.class, etc.). Consumers (log shippers, trace tools, custom queries) read the same vocabulary regardless of which surface they came in through.

OpenTelemetry traces (opt-in)

When tracing is enabled, Aileron starts a server-root span on every request and child spans for the work inside. The child spans cover action execution, connector calls, capability checks, and approval waits. Spans propagate via W3C TraceContext ↗. An inbound traceparent header from the calling agent makes Aileron’s spans children of the agent’s trace, so your end-to-end view stays coherent. With tracing off (the default), there’s zero SDK overhead. The call sites resolve to no-op tracers. The W3C propagator is installed regardless, so an inbound traceparent is parsed and forwarded even when this process emits nothing.

Three ways to consume traces

stdout: Local debugging. Spans land on stderr as JSON-per-line. Pipe to jq:

AILERON_OTEL_ENABLED=true \
AILERON_OTEL_EXPORTER=stdout \
aileron launch claude

file: Durable retention across sessions, mirroring the audit log’s on-disk layout:

AILERON_OTEL_ENABLED=true \
AILERON_OTEL_EXPORTER=file \
aileron launch claude
# spans → ~/.aileron/traces/spans-YYYY-MM-DD.jsonl

A new file is created per local-clock day. A session that crosses midnight rolls naturally to the next day’s file. AILERON_TRACES_DIR overrides the state directory. The default (~/.aileron) keeps audit and traces side by side.

otlp: Production. Ships spans to an OpenTelemetry Collector via OTLP/HTTP. See the next section.

Hooking up to a collector

The OTLP exporter honors the standard OTel environment variables ↗ that every OTel-instrumented service in your stack already understands. There’s no Aileron-prefixed alternative. Forking the names would force you to maintain two parallel sets.

Stand up a collector locally for development:

docker run --rm -p 4318:4318 \
  otel/opentelemetry-collector-contrib:latest

Then point Aileron at it:

AILERON_OTEL_ENABLED=true \
AILERON_OTEL_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
OTEL_EXPORTER_OTLP_INSECURE=true \
aileron launch claude

For a managed backend, point at its ingest endpoint and pass auth via OTEL_EXPORTER_OTLP_HEADERS:

# Honeycomb
AILERON_OTEL_ENABLED=true \
AILERON_OTEL_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io \
OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=YOUR_API_KEY \
aileron launch claude

# Grafana Cloud
AILERON_OTEL_ENABLED=true \
AILERON_OTEL_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-central-0.grafana.net/otlp \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64(instanceID:token)>" \
aileron launch claude

Recognised env vars (handled by the OTel SDK directly):

• OTEL_EXPORTER_OTLP_ENDPOINT: Collector URL. Defaults to http://localhost:4318.
• OTEL_EXPORTER_OTLP_HEADERS: Comma-separated k=v pairs added to every export request. Use this for API keys.
• OTEL_EXPORTER_OTLP_INSECURE: Set to true to allow plain HTTP (development only).
• OTEL_EXPORTER_OTLP_TIMEOUT: Request timeout. Default 10s.

The full set is in the OTel exporter spec ↗.

What gets emitted

Span attribute schema

Every span carries the OTel-namespaced shape locked in for the audit payload. When you query traces by attribute, you query the same names you’d query the audit log by. This table is the source of truth for what’s available.

Action execution (aileron.action.execute):

Capability check (aileron.capability.check):

Connector call (aileron.connector.call):

Sandbox HTTPS data plane (connector.proxy.proxied, connector.proxy.rejected, sandbox.proxy.rejected audit events):

Approval wait (aileron.approval.wait):

Failure (any error span). From the closed taxonomy in ADR-0010:

When a span fails, the OTel span status is also set to Error with the failure message. Your tracing UI’s red flags work without parsing attributes.

Configuration

All Aileron-side knobs are environment variables read at daemon startup. Defaults reproduce the historic behavior: tracing off, audit on. The OTEL_EXPORTER_OTLP_* family is consumed directly by the OTel SDK and only matters when AILERON_OTEL_EXPORTER=otlp.

A misconfigured exporter degrades gracefully to no-op rather than failing daemon startup. This applies to unknown exporter names and to known exporters whose construction fails. The Aileron HTTP server keeps serving when its telemetry sidecar is misconfigured. The failure is logged at warn level so you find it without it taking the daemon down.