For Engineers
The fastest route: read the Five-Minute Intro, build the minimal delegation-engineering layer, and follow the Core Path when you want the reasons. The Quick Start checklist walks through a first assessment.
Where the layer lives in an agent stack
Section titled “Where the layer lives in an agent stack”Put every limit in the harness, not in the model. A system prompt is not a security boundary: an instruction in the prompt is one more input the model may fail to follow or be talked out of, including by text it reads in a tool result.
| Layer element | Where it lives in a current agent stack |
|---|---|
| Harm surface | An inventory of what the agent can touch: each MCP server and the tools it exposes, credentials and their scopes, writable paths, network destinations, and anything that sends messages or moves money. Also list what the agent reads that someone else can write: inboxes, issue trackers, web results, shared documents. |
| Exposure envelope | Enforced outside the model. Run code in a sandbox (container or VM, scoped filesystem). Use an egress allow-list, tool allow/deny lists per MCP server, spend and rate caps, and scoped, short-lived credentials. |
| Risk preflight | Pre-tool-use hooks that can block a call, require approval or rewrite arguments. Use permission modes that decide which tools run unattended. The estimator sits at a more verifiable tier than the agent: rules or a narrow classifier, not a sibling of the agent’s model. |
| Correlation budget | Guard models and monitors from a different model family than the agent, plus at least one rule-based check per consequential action. |
| Hijack defenses | Keep untrusted content (web pages, documents, emails, tool outputs, and the tool descriptions of third-party MCP servers) away from control flow. No component should hold private data, read untrusted content and have an outbound channel at once (the lethal trifecta). CaMeL extracts the control and data flows from the trusted query, so data retrieved later cannot change which tools are called (Debenedetti et al. 2025, arXiv:2503.18813). Beurer-Kellner et al. 2025 catalogue design patterns for agents that resist prompt injection (arXiv:2506.08837). Test with adaptive attacks, for example in AgentDojo (arXiv:2406.13352); adaptive attacks bypassed all 12 published defenses tested by Nasr et al. 2025, most of which had reported near-zero attack success (arXiv:2510.09023). |
| Ledger | Post-tool-use hooks and tracing that record each consequential action with its preflight estimate, per delegate. Reconcile monthly against incidents and near misses. |
Harnesses already provide much of the first four rows. What they do not provide is the accounting: a preflight that estimates expected damage rather than asking a yes/no question, and a ledger whose estimates are checked against outcomes. Runtime Accounting describes both.
Patterns and examples
Section titled “Patterns and examples”The pattern catalogue has about twenty AI-specific patterns. The ones most engineers reach for first:
- Structural: Escalation Ladder (approval modes), the exposure-envelope gateway, Task Decomposition Pipeline, Bulkhead Isolation.
- Monitoring: Trusted Monitor, Random Audit, Flight Recorder.
- Verification: Capability Probing, Honeypot Injection, Consistency Triangulation.
- Information: Need-to-Know Compartments, which separate untrusted context from capabilities.
- Recovery: Checkpoint-Rollback and Graceful Degradation.
There are two worked examples: a coding agent and a research assistant. Composing Patterns covers how patterns interact.
Three mistakes specific to this framework
Section titled “Three mistakes specific to this framework”Generic engineering advice (threat-model before copying patterns, don’t over-engineer low-stakes tools) applies here as everywhere. These three are the ones the framework is specifically about.
1. Multiplying miss rates as if checks were independent
Section titled “1. Multiplying miss rates as if checks were independent”“Three checks, each catching 90%, so we miss 0.1%.” That holds only if the checks fail independently. Checks built on one model family, trained on overlapping data or reading the same context tend to miss the same items. With common-cause weight ρ, three such checks miss (1−ρ)·0.001 + ρ·0.1 of harmful actions: about 11× the naive figure at ρ = 0.1, 21× at ρ = 0.2 and 50× at ρ = 0.5 (the entanglement tax). On benchmark questions strong LLMs give ρ of about 0.3–0.6 across providers and 0.5–0.7 within one (evidence); for harm detection it is unmeasured, so treat these values as priors.
These figures describe benign, common-cause failure. Against an adaptive attacker, such as an injected prompt or a scheming component searching for an input that defeats every layer, assume ρ near 1, so that only your single best check counts, unless a control evaluation shows otherwise.
To avoid this mistake, give each consequential action at least one check from a different tier or provider. Measure ρ where you can, using the phi coefficient of per-item misses on a shared labeled challenge set (recipe).
2. Letting the wrong party set the risk budget
Section titled “2. Letting the wrong party set the risk budget”If the team that ships the agent sets its risk budget but users or third parties bear the harm, the team will tend to accept more risk than the people exposed would. In other words, it will overestimate how much risk is acceptable. This is moral hazard, the problem The Insurer’s Dilemma works through. Have the risk-bearers, or someone accountable to them, set the budget. Get the estimates reviewed by someone who is not paid to ship.
3. Estimates that are never reconciled
Section titled “3. Estimates that are never reconciled”A risk budget built from guesses and never compared with outcomes is fake precision with bookkeeping aesthetics. The probabilities also go stale: a model upgrade, a new tool or a widened use case changes the harm surface. Keep the ledger, and compare predicted against observed incidents and near misses on a schedule and after every capability or tool change. When they diverge, revise the estimates, not the incident log.
A reading mistake
Section titled “A reading mistake”Skipping Entanglements. Two components that share a model are enough to pay the tax; you do not need a large system.