Consequence from the field guide: full tracing from the first commit. The remediation loop is observe a failure → capture its trace → convert it into an eval case → change an input-boundary → verify the eval catches it. That loop — failures flowing back into the eval set — is the actual engine of agent reliability.

How it's done

Non-reproducibility is the defining constraint. With temperature above zero, the same input can take a different path, so a failure that happens 3% of the time may never recur identically — you cannot set a breakpoint on it, and you cannot re-run to reproduce it. Therefore the trace is not a debugging aid; it is the only artifact of the failure.

Mechanically: instrument the agent with span-based distributed tracing that captures, per step, the full inputs and outputs, the tool call and its arguments, the retrieval results, the token counts, cost, and latency. Model spans, tool spans, retrieval spans, workflow spans. Step-level, not a pass/fail health check. Then close the loop: a production failure's trace becomes a new eval case (Deep-dive 4), you change something at the input boundary (context, tool, prompt), and you verify the eval now catches it.

Tools available

• Langfuse — the open-source leader (MIT, ~28k+ stars), OpenTelemetry-native, self-hostable; tracing, prompt versioning, basic LLM-as-judge, cost analytics.
• Arize Phoenix — OpenInference/OTel-based, notebook-friendly, strong for RAG and multi-step agent-trajectory analysis; 50+ research-backed metrics, trace clustering, anomaly detection.
• MLflow — end-to-end GenAI lifecycle; auto-instruments 60+ frameworks via OpenTelemetry (OpenAI Agents SDK, LangGraph, LlamaIndex, DSPy, CrewAI, Bedrock, and more).
• LangSmith — path of least resistance for LangChain/LangGraph teams.
• Braintrust — managed eval + tracing; fast to stand up.
• Helicone — proxy gateway; zero-code-change cost tracking.
• AgentOps, LangWatch — agent-run visualization (waterfalls, topology maps), online evals on production traffic.
• Standard: OpenTelemetry GenAI semantic conventions — the emerging cross-vendor standard defining agent/workflow/tool/model spans. Caveat: as of ~v1.41 most gen_ai.* attributes still carry "Development" stability, so names can change without a major version bump.

Best practices

Full tracing from commit one — it is not a later-stage add-on, because the failures you didn't trace are simply gone. Capture step-level spans with inputs, outputs, tokens, cost, and latency each. Go OTel-native so you can swap backends without re-instrumenting. Operationalize the trace→eval loop as a standing process, not a heroic one-off. Instrument cost and token usage explicitly (silent cost blowups are a top production surprise). And redact PII yourself — none of these tools auto-redact by default, so raw traces are a compliance liability.

Failure points teams ignore — and what each costs

• Logging instead of tracing. Flat logs, no spans, no causal structure. Consequence: you cannot reconstruct a non-deterministic multi-step path from logs, so failures are undebuggable.
• Not capturing full context. Traces that omit the actual prompt/tokens/tool arguments. Consequence: the failure is unreproducible and unexaminable — you have proof it happened and no way to understand it.
• Over-aggressive sampling. Sampling traces to save storage. Consequence: the rare failure — exactly the one you most need — is the one you didn't capture.
• Treating a caught failure as a one-off. Fixing it in place without adding an eval case. Consequence: it recurs, because nothing guards the regression — you paid the debugging cost and kept none of the value.
• No cost/token spans. Consequence: cost anomalies (a retry storm, a context blowup) are invisible until the invoice arrives.
• PII in traces. Consequence: your observability store becomes a data-protection incident.
• Leaning on unstable OTel GenAI attributes. Consequence: attribute names shift under you and dashboards/alerts silently break.

How to evaluate and mitigate

Evaluate with the acid test: pick any production failure — can you reconstruct exactly what happened from its trace alone? If not, your instrumentation is insufficient. Track trace coverage, time-to-root-cause, and the metric that actually predicts reliability: the percentage of production failures converted into eval cases. That conversion rate, not your dashboard's green, is the health of your reliability program.

Mitigate: full step-level tracing; OTel-native instrumentation; the trace→eval loop as a standing ritual; explicit cost/token spans with anomaly alerts; PII redaction; retention tuned so rare failures survive.

Practical vs. still research

Practical / mature: span-based tracing, cost/token instrumentation, and the trace→eval remediation loop. This is distributed tracing adapted to a new workload — familiar territory, and production-ready today.

Emerging: the OpenTelemetry GenAI semantic conventions are becoming the standard but are not yet stable; automated trace clustering and anomaly detection (Phoenix and others) are useful but young.

Still research — and important to internalize: automated root-cause of a stochastic decision. Even with the exact trace, you cannot step through why the model chose wrong; the causal explanation lives in weights, not in code you can read. So remediation stays at the boundary — better context, a narrower tool, a revised prompt, a new eval case — and "explain why the model did that" remains, at the mechanistic level, an open problem. Design your process around fixing boundaries, not around expecting a why.