Skip to main content

Traces

A trace is the end-to-end record of one agent turn. It starts when a user message arrives at the Runtime, covers every LLM generation, tool call, datasource query, and sub-agent invocation, and ends when the final assistant reply is returned. Each nested step is a span; spans form the parent–child tree you see in the trace viewer.

Traces List​

The Traces page lists every ingested trace with its status and metrics.

Filters sit at the top of the page:

  • Agent ID — Free-text filter by agent identifier.
  • Model — Free-text filter by model (e.g. gpt-4o, claude-3.5-sonnet).
  • Status — All · OK · ERROR.
  • From / To — Absolute date-time range. Leave blank to use the implicit window.

Each row shows:

  • Time — Trace start timestamp in local time.
  • Trace ID — Truncated UUID. Click the row to open the viewer.
  • Agent — Agent name.
  • Model — Primary model used by the trace (the last generation's model when multiple).
  • Status — OK chip (orange) or ERROR chip (red).
  • Duration — Wall-clock latency from first to last span.
  • Tokens — Total token count across all generations in the trace.
  • Cost — Total USD cost computed from the Gateway's per-model rates.

Export in the top-right downloads the filtered trace set as JSON for offline analysis.

Trace Viewer​

Clicking a trace opens the viewer. The header summarizes the full run; the left panel is the span tree; the right panel is the detail for the currently selected span.

Trace Header​

  • Agent name — With an ERROR chip when any span failed.
  • Trace ID — Full UUID.
  • Model / Duration / Tokens / Cost / Start / End — Aggregated over the whole trace.

Agent Analytics in the top-right opens the agent's rolled-up view on the Agents page.

Span Tree​

Each node in the tree shows:

  • Span name — The operation label (e.g. lint_check, code_review_parse, repo_search, llm_suggestion_gen).
  • Agent — The agent attribute the span was recorded under.
  • Tokens and cost — Per-span contribution.
  • Duration — Rendered both as a number and as a Gantt bar aligned to the trace timeline.
  • Status dot — Green for OK spans, red for ERROR spans.

The Gantt bar makes the critical path visible at a glance — the span with the widest bar on the tightest timeline is the one that drove total latency.

Span Detail Tabs​

Click any span to populate the right panel with its detail. The header of the panel shows the span's kind chip (Tool, Generation, Retrieval, etc.), status, model, latency, tokens, and cost. Below are four tabs.

Input / Output​

The default tab. For tool spans, Arguments / Response render the structured call. For generation spans, Input shows the prompt (or the messages array) and Output shows the assistant reply — including the exact error text when the span failed.

Tools​

For a generation span, the Tools tab lists every tool that was presented to the model on that call, each with its name and description. It is the fastest way to confirm which capabilities the agent had available when it produced a response, and to spot a missing or unexpected tool.

Metadata​

The Metadata tab exposes the underlying OpenTelemetry fields:

  • Identifiers — Span ID, Trace ID, Parent Span.
  • Span Info — Name, Service, Kind (internal, client, server), Operation (chat, tool, etc.), Model.
  • Attributes — All custom attributes set by the instrumentation (e.g. agent-id, user-id, prompt template name).

Timeline​

The Timeline tab places the span on the trace's wall-clock timeline:

  • Span Position in Trace — Horizontal bar showing where the span sits across the total duration.
  • Start Time / End Time — Absolute timestamps.
  • Duration — Wall-clock time for the span.
  • Offset from Trace Start — How far into the trace the span began.
  • % of Total Trace — Share of the trace's total duration this span consumed.

The offset and percentage are the fields that identify the critical path — the span with the longest duration that started last usually deserves the first round of optimization.

Reading an Error Trace​

When a trace is marked ERROR, the span tree shows a red dot on every span that failed. The failure propagates upward: in the screenshot above, llm_suggestion_gen failed with Context length exceeded and surfaced the error to its parent lint_check. The Input/Output tab for the failed span shows the exact error text the LLM returned — the fastest way to identify whether the cause is prompt size, a bad argument, or a tool response that is too long to fit in context.

Best Practices​

  • Filter the Traces list by Status = ERROR first when triaging. Every error trace is a direct signal.
  • Use the Gantt bars to find the critical path before looking at individual spans. Agents usually have one bottleneck span per turn.
  • Metadata's Attributes is the place to add custom correlation keys (tenant id, session id, request id). Add them once in the instrumentation and they become filterable across the whole fleet.