Subscribe for more posts like this →

Deep-dive 5 — Injection & Tool Security (Confused Deputy Containment)

Share

Consequence from the field guide: assume any content the agent reads may be hostile. Design the harness so the worst an injection can do is bounded by the capabilities you granted — and grant few. This is a containment problem, not a fix.


How it's done

You cannot separate the control plane from the data plane inside the model — instructions and data are the same token stream. So you contain the blast radius outside it:

  • Structured, harness-validated tool calls. The model's free text never becomes an executed action. It emits a structured request; the harness validates schema and permissions, then executes. This is the load-bearing defense.
  • Least-privilege tool scoping. Capability-based security: expose the fewest, narrowest tools that do the job. Fewer tools also lowers error rates — the defense is free.
  • Trust labels. Mark untrusted spans (retrieved docs, tool outputs, user input) so the harness can treat them differently from system intent.
  • Draft-then-commit. For irreversible or high-blast-radius actions, the agent drafts; a human or a hard rule commits. A two-phase commit for dangerous side effects.
  • Dual-stage guardrails. Screen inputs before the model and outputs before the user (injection, PII, jailbreak, policy).
  • Continuous red-teaming. Automated adversarial probing in CI, not a one-time audit.

Tools available

  • Runtime guards: NVIDIA NeMo Guardrails (Colang DSL, five-stage pipeline), Guardrails AI, LLM Guard, Lakera Guard (acquired by Check Point, Sept 2025), Rebuff, Meta's Llama Guard 3, OpenAI/Azure Prompt Shields.
  • Gateways: Bifrost (open-source AI gateway, enforces guardrails at the gateway layer across providers; aggregates Patronus, GraySwan, and others for defense-in-depth).
  • Specialized output-side / adversarial: Patronus, GraySwan; PromptArmor (attention-analysis approach, low overhead, emerging).
  • Testing / red-team: Garak (NVIDIA scanner, 100+ attack probes), PyRIT (Microsoft), Promptfoo, DeepTeam. Run these in CI.
  • Standards: OWASP LLM Top 10 (LLM01 Prompt Injection, LLM02 Sensitive Info Disclosure, LLM06 Excessive Agency, LLM07 System Prompt Leakage), NIST AI RMF, and the 2026 US-government MCP security advisory.
  • Research guards: PIShield (intrinsic-feature detection), PIGuard (calibrated boundary), PISmith (RL red-teaming).

Best practices

Assume-breach: design so a compromised agent can't do much, rather than betting on perfect prevention. Least privilege on tools is the most durable and portable defense — unlike vendor-tied detectors, it survives a model swap (defense portability is genuinely poor in 2026; many detectors are tied to one ecosystem). Human-in-the-loop or draft-commit for anything irreversible. Layer defenses (dual-stage guardrails + harness validation + least privilege) — no single control is sufficient. Red-team continuously so new payloads are caught as regressions.

Failure points teams ignore — and what each costs

  • Model output becomes action directly. No harness validation between "the model said" and "the system did." Consequence: prompt injection escalates to arbitrary execution — the confused deputy at full authority. This is the catastrophic one.
  • Over-privileged tools. Broad, powerful tools "for flexibility." Consequence: OWASP LLM06 excessive agency — a single successful injection now has a large blast radius.
  • Detection filters treated as sufficient. Consequence: adaptive attacks are specifically built to evade detectors; you get latency overhead and a false sense of security while the real hole stays open.
  • Indirect injection ignored. Teams filter user input but forget that retrieved documents and tool outputs are also untrusted. Consequence: the sneakiest, most common real-world vector — a poisoned web page or document steers the agent, and nobody was watching that channel.
  • System-prompt / PII leakage. Consequence: LLM02/LLM07 — the agent discloses its instructions or sensitive data, a direct compliance and IP loss.
  • Point-in-time-only testing. Consequence: the app was safe at audit and unsafe a model-version later; injection is a moving target.

How to evaluate and mitigate

Evaluate: attack-success-rate under a red-team probe suite (Garak/PyRIT), including indirect injection via retrieved content; over-defense / false-positive rate (a guardrail that blocks legitimate traffic is its own incident); and a blast-radius analysis — enumerate, given the tools you granted, the worst a fully-compromised agent could do. Map coverage against the OWASP LLM Top 10.

Mitigate: harness validation + least privilege + draft-commit are the portable, durable core. Add dual-stage guardrails as defense-in-depth, never as the sole line. Run red-teaming continuously. Architect assume-breach: bound damage rather than relying on prevention.

Practical vs. still research

Practical / the real defense: harness discipline, least-privilege tools, draft-commit, continuous red-teaming, dual-stage guardrails as one layer. These work today and are mostly architecture, not magic.

Practical but defeatable: detection-based guardrails — useful in depth, but adaptive attacks evade them, so never rely on them alone.

Still research: provable injection prevention (keeping the model on-task even with injected content), detection robust to adaptive attack, and intrinsic-feature approaches (PIShield-style). The field's own summary: perfect security remains elusive because of the fundamental nature of generative AI. Plan for containment, report it honestly to stakeholders, and don't let a vendor sell you "solved."

Read more