Agent Engineering for Distributed Systems Engineers

For thirty years the job has been the same: make unreliable components — networks, disks, nodes — compose into reliable systems. The components failed in understood ways. They crashed, they lagged, they partitioned, they occasionally corrupted a byte. We built an entire discipline to tame that: timeouts, retries, quorums, replication, idempotency keys, sagas, backpressure, distributed tracing.

Agent engineering is that same job with exactly one new component wired into the critical path: a language model. The trap — the thing that makes smart engineers ship fragile agents — is treating it as "just another API." It is not. It has a fault mode classical systems spent decades engineering away: it returns confident, well-formed, wrong answers. It is stateless. It is priced per token. And its failures are not reproducible. Almost everything that feels novel about building agents is a distributed-systems problem you already know how to reason about.

Here are the nine mappings that matter — and where each one breaks, because the breakage is the part worth your attention.

1. The LLM is a Byzantine, non-idempotent oracle. Model it as one.

Classical fault tolerance mostly assumes fail-stop: a node either works or crashes. We assume this because tolerating Byzantine faults — nodes that return arbitrary, confidently-wrong answers — is expensive (3f+1 replicas, PBFT-class protocols), so we architect the possibility away.

An LLM drags the Byzantine fault back into your hot path by design. You have deliberately placed, in the middle of your control flow, a component that will sometimes hand you a fluent, schema-valid, completely wrong result. It is not adversarial-Byzantine (usually) — it's "honest but wrong," a probabilistic correctness fault. Same blast radius, different intent.

The useful part of the analogy: BFT's core technique is replication and voting — outvote the liars. You can borrow it. Sample N completions and take a majority (self-consistency), or run an ensemble and adjudicate. That trades cost for reliability.

Where it breaks: BFT assumes faulty nodes fail independently, so voting works. LLM errors are correlated — the model tends to be wrong the same way on the same input. So sampling-and-voting suppresses variance (the model's run-to-run wobble) but does nothing for bias (its systematic blind spots). Quorum-over-samples buys you consistency, not correctness.

Consequence: validate every output structurally and semantically; trust nothing on faith. Use programmatic checks wherever ground truth exists (a number, valid JSON, a passing test). Reserve sampling/voting for squeezing out variance, and never mistake agreement for truth.

2. The model is stateless. You own all state — and you pay for it every turn.

Each completion is memoryless: a near-pure function (context) → distribution over tokens. This is textbook stateless-service design, the 12-factor "processes are stateless; state lives in backing services" rule. All continuity — conversation, task progress, learned preferences — must be externalized and re-injected on every call. Nothing new so far.

Where it breaks from the classical version: in a stateless web tier, carrying a session token forward is essentially free. Here, re-injecting state is priced per token and it consumes the scarce context resource (see §3), and — the part people miss — it can degrade quality, because more history means more to attend over and more chance of dilution. So carried state has a monotonically rising cost and, past a point, negative returns. Statelessness plus per-token pricing plus attention dilution means you cannot let history accrue passively.

Consequence: design an explicit state architecture on day one — short-term (the window), long-term (an external store: vector DB, KV store, files), and an active compaction policy that decides what graduates from one to the other. Passive accumulation is the default failure mode.

3. The context window is a contended, non-uniform cache. Manage it, don't fill it.

The window is a bounded resource under contention, and the instinct to "just use a bigger one" is the same instinct that says "just add more cache" — and it's wrong for the same reasons.

• Eviction, not overflow. You compact before the limit, not at it — the same reason you don't run a cache to 100% before evicting. Anthropic and others productized automatic compaction precisely because filling the window is a cliff, not a wall.
• Non-uniform access cost. The "lost in the middle" finding — models attend more reliably to the start and end of a long context than the middle — means your cache has non-uniform hit quality, closer to NUMA than to a flat address space. Placement matters: put the load-bearing tokens where attention is strongest.
• Pollution, not just capacity. Chroma's "context rot" work showed performance degrading as tokens grow even with perfect retrieval. The problem isn't only running out of room; it's signal-to-noise. This is why context drift — not context exhaustion — is blamed for the majority of enterprise agent failures. The working-set principle applies directly: keep the active set small and high-signal ("the smallest set of high-signal tokens that maximizes the desired outcome").

Where it breaks: a CPU cache eviction is effectively lossless — evict a line, refetch it from memory unchanged. Compaction is lossy. Summarizing 132k tokens down to 2k throws away nuance you may need and cannot reconstruct identically. It's not LRU; it's lossy compression under a bit-budget.

Consequence: treat compaction as an encoder design problem. Decide explicitly what must survive a compaction pass — decisions made, hard constraints, identifiers, open questions — and what may be dropped. Then measure the reliability delta on your evals when you change the policy. You are doing rate-distortion tradeoffs, so measure the distortion.

4. Correctness is now statistical. Evals are your SLOs and error budgets.

You already made this leap once. You stopped promising "correct" and started promising "correct with probability p, within latency budget L" — SLOs, error budgets, tail latencies. You accept the 0.1% and manage it rather than chasing an impossible 100%.

Apply it wholesale. You cannot assert output == expected against a non-deterministic system. You assert P(good) ≥ threshold, measured over a distribution of inputs. Evals are the SLO/monitoring layer for a probabilistic system, and you will never reach 100% — you set a target, measure continuously, and spend an error budget deliberately.

Two measurement modes, and they mirror the fault model in §1:

• Programmatic checks where ground truth exists — assert it directly. Exact, cheap, reliable. Always prefer these.
• LLM-as-judge where "correct" is a property of prose (relevance, faithfulness, tone). But note: your judge is itself a Byzantine oracle (§1). It has measurement error and known biases — position, verbosity, self-preference, authority. So your monitor is a flaky sensor that must be calibrated against human labels on a sample (spot-check 5–10%), exactly as you'd calibrate a drifting instrument. A judge agrees with humans ~85–92% of the time — good, not good enough to be your only safeguard.

Where it breaks: a classical SLO measures a system whose correct behavior is defined in advance. Here, "correct" is often a fuzzy property of natural language, so writing the rubric is partly writing the spec — you co-evolve the definition of good as you measure it. The eval set is a living specification, not a fixed oracle.

Consequence: eval-first, in CI, against a golden set, from the first commit. Prefer programmatic gates. Treat the judge as an instrument to calibrate, never as an oracle to trust. (With EU AI Act enforcement arriving August 2026, this stops being hygiene and becomes an audit trail.)

5. Tool calls dissolve the control/data-plane boundary. Prompt injection is a confused-deputy problem.

An agent's tool layer is an RPC layer — the model requests an action, the harness executes it with the application's credentials. Now add the twist: the arguments to that RPC are produced by a component that also consumes untrusted input — retrieved documents, tool outputs, user messages, web content. If any of that untrusted content contains instructions, the model — a privileged deputy — may act on them using its authority. That is the confused deputy problem, textbook capability-security, the same family as SSRF and log injection: data smuggled into a place where it gets treated as control.

Distributed systems treat the control plane / data plane separation as sacred. The catastrophe of LLMs is that they dissolve it: instructions and data are the same token stream, with no ambient mechanism to tell "the user's actual request" from "text that appeared in a document the agent read." (This is also the #1 security concern in the new MCP spec, and the subject of a US-government advisory this year.)

Where it breaks — and why this is a containment problem, not a fix: in a classical system you can architecturally separate control from data. Inside an LLM you fundamentally cannot — it's one modality. You will not "solve" injection at the model. You can only bound the blast radius outside it:

• Structured tool calls, harness-validated. The model's free text never becomes an executed action directly. It emits a structured request; the harness validates schema and permissions, then executes. This is the single most important rule of the week.
• Least-privilege tool scoping — capability-based security. Expose the fewest, narrowest tools that do the job. (Fewer tools also reduce error rates, so this is free.)
• Trust labels on untrusted spans, so the harness can treat retrieved/user content differently from system intent.
• Draft-then-commit for anything irreversible — a two-phase commit for dangerous actions, with a human or a hard rule holding the second phase.

Consequence: assume any content the agent reads may be hostile. Design the harness so that the worst an injection can do is bounded by the capabilities you granted — and grant few.

6. Retrieval is a distributed read path, not vector-search magic.

Grounding an agent is a read query against a knowledge store, and bad retrieval is the single largest source of hallucinated answers in production. Every concept ports:

• Recall vs. precision — the eternal read-path tradeoff. High recall gets the relevant chunk into the candidate set; precision (and reranking) keeps junk out.
• Chunking is your partitioning/sharding scheme. It silently determines everything downstream — bad boundaries mean the fact you need is un-retrievable no matter how good the query. This is a design decision, not a default.
• Staleness is replication lag — your index versus the source of truth. Freshness is a consistency problem.
• Hybrid search is multi-signal ranking. Don't trust one index; combine dense (semantic recall) with sparse (BM25/lexical precision), exactly as you'd blend features in a ranker. Dense alone misses exact-match and rare tokens.
• Reranking is a second-stage query planner — a precision filter over a high-recall first stage.
• Just-in-time retrieval is demand paging — pull content into context only when needed, via lightweight identifiers (paths, IDs), versus prefetching everything.

Where it breaks: a classical read returns an exact record or a clean "not found." Retrieval returns approximate, ranked results, and the consumer — the LLM — will confabulate over the gap rather than throw. A missing or wrong read does not fail loudly; it fails silently, as a plausible hallucination.

Consequence: measure the retriever independently of the generator (RAGAS separates retrieval quality from generation quality). Track recall/precision of retrieval on its own. And make "I don't have that information" a first-class, explicitly-rewarded behavior in your evals — because the default is a confident lie.

7. A long-horizon agent is a long-running workflow. Use sagas, checkpoints, budgets.

An agent that runs for minutes to hours across many steps is a long-running distributed workflow, and the orchestration playbook applies with almost no translation.

• Sagas. A sequence of local steps, each with a compensating action on failure. You don't roll back a distributed transaction; you compensate forward. An agent's replanning on a failed step is compensation — partial recovery, not full restart. (This is why plan-then-execute beats a naive react loop: a failed executor step escalates to the planner for a revised sub-plan, not a reset.)
• Checkpointing. Persist intermediate state so a crash or restart doesn't discard a session's accumulated work.
• Idempotency. Retries + side-effecting tools + a non-deterministic actor = duplicate-action risk (the agent sends the email twice). You need idempotency keys and dedup exactly as in at-least-once messaging. Exactly-once is as much a fiction here as it was there; design for at-least-once with dedup.
• Deadline & budget propagation. Give the loop a wall-clock deadline and a token/cost budget, propagated down through sub-tasks — your deadline4j instinct, applied to a loop that will otherwise burn unbounded money and time when confused. This is admission control plus a timeout on an unbounded process.
• Plan-then-execute as orchestrator/worker split. A deterministic-ish planner (the backbone) directs intelligence deployed at specific executor steps (the leaves). The winning production shape in 2026 is a deterministic backbone with LLM intelligence at specific steps — not a monolithic "one model does everything" loop.

Where it breaks: a classical saga has a fixed, known set of steps and compensators. Here the planner may invent steps at runtime, so you cannot pre-write a compensating action for every possible action the agent might take.

Consequence: constrain the executor's action space hard — few, well-scoped, ideally reversible or dry-runnable tools. Draft-commit the irreversible ones. Set hard budgets. Checkpoint. You are trading the agent's freedom for your ability to recover.

8. Prefer the workflow to the agent. Don't reach for coordination you don't need.

Coordination is expensive and you avoid it when a simpler structure suffices — you don't reach for consensus or a distributed transaction when a local operation or eventual consistency will do.

The agentic loop is the expensive coordination primitive. A chain — a deterministic pipeline of steps — is cheaper, testable, debuggable, and has bounded cost and latency. Reach for the loop only when the task genuinely requires runtime decision-making: an unknown number of steps, a path the agent must discover as it works (a coding agent exploring an unfamiliar repo genuinely needs this; "summarize then translate" does not).

Over-agentifying is over-engineering with distributed coordination you didn't need — the same mistake as reaching for Raft when a single writer would do.

Consequence: default to the least dynamic structure that solves the problem. Chain first. Add routing when inputs are heterogeneous. Graduate to a loop only when you can name the specific dynamism that demands it. Every loop in your architecture should have to justify its own existence.

9. Failures aren't reproducible. The trace is your only ground truth.

Classical distributed debugging already leaned on tracing — spans, correlation IDs — because you couldn't reason about emergent behavior from logs alone. Agents remove even the fallback you still had there: replay. With temperature above zero, the same input can take a different path, so a failure that occurs 3% of the time may never recur identically. You cannot set a breakpoint on it. You cannot re-run to reproduce it.

Therefore the trace is not a debugging aid — it is the only artifact of the failure. Every prompt, every tool call and its arguments, every observation, the actual tokens returned. If you didn't capture it in full, the failure is simply gone.

Where it's worse than classical: even with the exact trace, you cannot step through a stochastic decision to see why the model chose wrong — the causal explanation lives in weights, not in code you can read. Your remediation is never a line-level fix. It happens at the boundary: better context, a narrower tool, a revised prompt, a new eval case.

Consequence: full tracing from the first commit, non-negotiable. The remediation loop is: observe a failure → capture its trace → convert it into an eval case → change an input-boundary → verify the eval catches it. That loop — failures flowing back into the eval set — is the actual engine of agent reliability, and it's the whole point of building eval-first in Week 1.

Closing: the one sentence

You have stopped being the author of the control flow and become the operator of a probabilistic component. Everything above follows from that single shift. You don't write the decision anymore — you build the harness that validates it, the evals that measure it, the context that informs it, the retrieval that grounds it, the budgets that bound it, and the traces that let you learn from it when it fails. That harness is a distributed system. You already know how to build those.