Skip to content
Delegation Risk

Delegation Risk

This site treats the risk of handing work to people, software and, above all, AI agents as a quantity: something to estimate before acting, budget across a system and design to keep small.

Epistemic status: early-stage, exploratory work, written quickly with LLM assistance. The core ideas seem promising, but the framework is largely untested on real AI systems and most numbers on the site are illustrative. See About This Site.

Start with Delegation Risk in 5 Minutes, the whole argument on one page. Then follow the Core Path, fifteen stops through the strongest material, in order. If you build agent systems, For Engineers shows where the ideas sit in a current agent stack.

The idea in brief

Section titled “The idea in brief”

Risk is a number you can budget. For each way a delegation can go wrong, multiply its probability by the damage it would do, and add up the results:

Delegation risk = Σ P(harm mode) × Damage

A research assistant that might leak proprietary data (P = 0.001 a month, damage $50,000) contributes $50 a month. Harm arrives through three channels: accident, defection and hijack.

Stacked checks fail together. Three checks that each miss 10% of harmful actions promise a 0.1% miss rate if they fail independently. If they share a common cause with weight ρ (the same model, data or context), the beta‑factor correction gives a miss rate of (1−ρ)·0.001 + ρ·0.1. That is about 11 times the promised rate at ρ = 0.1, 21 times at 0.2 and 50 times at 0.5. This multiplier is the entanglement tax.

Counted in independent checks, three are worth about 2.0 at ρ = 0.1, 1.7 at 0.2 and 1.3 at 0.5. No study has measured ρ for checks on harmful actions. Error correlations on benchmarks suggest about 0.3–0.6 for LLMs from different providers and 0.5–0.7 for models from one provider. Against an attacker who searches for inputs that defeat every layer at once, assume ρ is near 1: three checks are then worth one.

1 2 3 4 5 0 0.2 0.4 0.6 0.8 1 Correlation ρ between the checks’ misses 2 checks 5 checks 3 checks 2.0 1.7 1.3 Same model Same provider Different providers Worth this many independent checks 1 2 3 4 5 0 0.2 0.4 0.6 0.8 1 Correlation ρ between the checks’ misses 2 checks 5 checks 3 checks 2.0 1.7 1.3 Same model Same provider Different providers Worth this many independent checks
What a stack of checks is worth, counted as independent checks, when each check misses 10% of harmful actions. Each step down the scale is a tenfold rise in the chance that every check misses. Five checks already sit on the ceiling that no number of checks can pass, 1 − log10 ρ. Brackets: the ranges of ρ the site suggests assuming for LLM checks (Measuring Entanglement).

Do the accounting at runtime. Before a consequential action, the agent’s harness runs a risk preflight that estimates expected damage. It enforces an exposure envelope of hard limits that hold whatever the agent intends, and keeps an exposure ledger reconciled against real incidents.

Contents

Section titled “Contents”
Delegation Risk
The formula, harm channels, propagation, runtime accounting and insurance.
Entanglements
Why stacked safety checks fail together, and how to measure and reduce it.
Design Patterns
Twenty patterns for bounding delegation risk, each mapped to its prior art.
Case Studies
Agent incidents, and institutions such as nuclear launch authority and juries.
Research
Evidence on scheming, precedents from other fields, and open problems.
Power Dynamics
Speculative: how a delegate’s agency and power map onto the risk formula.
Reference
Concepts, Glossary, Quick Reference, FAQ, Related Approaches, Bibliography.

Sibling sites: Robust Reasoning Processes, Evaluation Engineering and the CAIRN Handbook.

Maintained by the Quantified Uncertainty Research Institute. Source on GitHub.