Glossary

Definitions as this site uses them. Borrowed terms keep their home field’s meaning; where this site’s usage is narrower, the entry says so.

Units convention. A probability on this site is always per something: per task, per action, or per period. A delegation risk carries the same unit. Most pages use dollars per month; worked examples state their unit at the top.

Core framework terms

Harm mode: a specific way a delegation could cause damage, with a probability and a damage value. It parallels the failure mode of FMEA. Example: “leaks proprietary data”, P = 0.001 per month, damage $50,000.

Harm surface: the set of a delegation’s harm modes, the delegation analogue of an attack surface. It is a set, not a number. You enumerate it, bound it (exposure) and price it (delegation risk).

Exposure: the worst-case loss across the harm surface, given the delegate’s actual access and capability. It is a dollar bound, not an expectation, and it depends on capability: the same office key carries more exposure for a delegate who can exploit what it opens. Lineage: exposure-at-default in finance. Gross exposure is measured before controls, net exposure after.

Delegation risk: the expected loss of a delegation, Σ P(harm mode) × Damage(harm mode), in dollars per the stated unit (canonical statement). Residual risk is delegation risk after controls. A delegation risk budget is the maximum a system or component may carry.

Harm channels: the three ways harm arises (Risk Decomposition).

Accident: the delegate fails at its task (bugs, hallucinations, edge cases, capability limits). P(error per action) often falls as capability rises, but not reliably: inverse-scaling results exist. A more capable delegate is also given more consequential actions, so damage per accident tends to rise. Do not assume accident risk falls with capability.
Defection: the delegate pursues something other than the principal’s interest. Scheming is the strategic case. Reward hacking, in which the delegate exploits a flaw in its objective (special-casing tests, editing a grader) without strategic deception, is a non-scheming sub-type. Capability raises both the chance that defection succeeds and the damage it can do.
Hijack: a third party or user steers a delegate that is not itself scheming. Routes: direct prompt injection or misuse; indirect injection through a web page, document, email or tool output; tool poisoning or a malicious MCP server; and a compromised supply chain. The attacker chooses the input, so treat correlated checks as fully correlated (ρ near 1).

Propagation rule: how reliabilities compose. Under independence, chains and parallel stacks multiply. For common-cause correlation, use the beta-factor mixture P(all fail) = (1−ρ)·∏pᵢ + ρ·min(pᵢ). The minimum rule is its ρ = 1 limit. The harmonic and discounted rules are retired. See Risk Propagation.

Trust (quantitative): where a number is attached, t = P(the delegate executes the task faithfully), estimated from its harm-mode table. Unquantified, “trust” is ordinary English. Risk inheritance: when A delegates to B and B to C, A carries C’s harm modes, composed by the propagation rule.

Entanglement tax: actual joint failure probability divided by what independence predicts, P(all fail) / ∏pᵢ. For n identical layers with miss rate p, it is (1−ρ) + ρ·p^(1−n). For three layers at p = 0.1 that is 1 + 99ρ: about 11×, 21× and 50× at ρ = 0.1, 0.2 and 0.5. This is benign-failure math; ρ between LLM checks is measured only on benchmarks, where it is about 0.3–0.7 (evidence). See Formal Definitions.

Correlation budget: a limit on failure correlation within a defense stack, such as “no two checks share a model family”. It complements the risk budget. ρ is on the failure-indicator (phi) scale.

Load-bearing channels: the sanctioned communication paths between delegates that decomposition could not remove. Monitoring budget should concentrate there.

Privilege overhang: the gap between what a delegate can do and what its task needs. It is the stock; privilege creep is the flow that grows it. Cloud-security tools measure the same quantity as unused entitlements. Microsoft’s 2024 State of Multicloud Security Risk Report found that identities used about 2% of the permissions granted to them in 2023.

Risk preflight: a damage estimate the scaffold runs before an agent action commits. It enumerates the action’s harm surface, bounds the exposure, compares the result with the budget, then proceeds, adjusts or escalates. Lineage: CORS preflight and aviation flight-risk assessment. The estimator must sit at a more verifiable (lower-numbered) tier of the verifiability hierarchy than the actor it checks. See Runtime Accounting.

Exposure envelope: the region of action-space in which an agent may act freely, enforced by the scaffold regardless of the agent’s inputs or intent. It is a hard bound, not a statistical one: a single catastrophic action must hit the wall, not merely overdraw a budget. Lineage: Airbus flight-envelope protection.

Exposure ledger: the running account per delegate. Actions debit it; completed verifications and de-provisioning credit it. It comes with two disciplines. Reconciliation periodically compares the ledger’s estimates with observed incidents and near misses. Consolidation rolls up a subagent tree. Expected-loss roll-ups simply add. Where several harm modes hit the same asset, the naive sum overstates the true loss, so it is a conservative upper bound. Correlation matters only for tail metrics and for AND-structured defenses.

Principal, delegate, executor, coordinator: the principal delegates, and is ultimately a human or an organization. A delegate receives a task (agent stresses goal-directedness, component architecture). An executor performs actions. A coordinator routes tasks, grants permissions or aggregates results. Coordinators often concentrate risk because of that leverage, but whether one is the riskiest component depends on the system.

Delegation engineering: designing, measuring and operating delegation structures to bound exposure while preserving capability, here mainly for AI agents.

Least-X principles

The core six, from the canonical page:

Least Intelligence: the least capable implementation that does the task.
Least Privilege: minimum permissions and access rights.
Least Context: minimum information about the wider system.
Least Persistence: minimum state across invocations, including memory stores, caches and logs the delegate can read.
Least Autonomy: minimum ability to act without review.
Least Connectivity: minimum communication paths between components.

The canonical page lists further principles as a labelled tail. Two need care. Predictability is worth seeking, but temperature 0 neither guarantees deterministic output from served LLMs nor prevents a deterministic policy from encoding covert signals. Hiding from a component whether it is being tested is hard to enforce: current models often recognize evaluations (Schoen et al. 2025, arXiv:2509.15541).

Architecture terms

Verifiability hierarchy: implementations ordered by how thoroughly they can be checked. The tiers are (1) formally verified code; (2) deterministic tested code and rule-based checks; (3) narrow learned models with measured error on a fixed task; (4) trusted LLMs, below the capability at which scheming is a live concern; and (5) untrusted frontier models and agents. Training method does not set the tier. See Concepts.

Decomposed coordination: splitting coordination across several limited components rather than one powerful coordinator. See Coordinator Constraints.

Byzantine fault tolerance: correct operation despite up to f arbitrarily faulty components, which requires at least 3f + 1 replicas. The bound counts faulty replicas; replicas that share a model can exceed f together.

N-version programming: independently developed implementations of one specification, used redundantly. Knight & Leveson (1986) found that such versions failed together more often than independence predicts.

AI safety terms

AI Control: designing protocols that stay safe even if the model is intentionally trying to subvert them, and testing them with control evaluations (Greenblatt et al. 2023). Its vocabulary (trusted and untrusted model, control protocol, auditing budget, usefulness cost) is used here as Redwood Research uses it.

Prompt injection: input that overrides a model’s instructions. Direct injection comes from whoever is prompting the model; indirect injection is planted in content the model retrieves or is given, such as web pages, files, emails or tool outputs (Greshake et al. 2023; OWASP LLM01:2025). This site files both under the hijack channel.

Tool poisoning: indirect injection through an MCP tool’s description, which the model reads but the user typically does not see (Invariant Labs 2025).

Lethal trifecta: an agent that has access to private data, is exposed to untrusted content and can communicate externally, so that an injected instruction can exfiltrate data (Willison 2025). Removing any one leg removes that route.

Scheming: an AI system covertly pursuing goals other than its operators’, including by deceiving them.

Instrumental convergence: the tendency of diverse goals to produce similar sub-goals, such as self-preservation and resource acquisition.

CAIS (Comprehensive AI Services): Drexler’s proposal to build general capability as a system of bounded services rather than unified agents (report). Related: tool AI vs. agent AI.

Scalable oversight: methods for overseeing systems that are more capable than their overseers in the relevant domain.

Risk measurement terms

Euler allocation: splitting a portfolio risk measure R among positions as RCᵢ = xᵢ·∂R/∂xᵢ. For risk measures that are homogeneous of degree one, the contributions sum exactly to R.

Value at Risk (VaR): the loss level exceeded with probability 1 − α over a period (for example, the 95% VaR is exceeded 5% of the time). It says nothing about how large losses beyond it are.

Expected Shortfall (ES): the mean loss in the worst 1 − α of outcomes. It is a coherent risk measure, unlike VaR, and it is the tail metric to use when expected loss is not enough.

Probabilistic risk assessment (PRA): quantifying risk through fault trees (AND and OR gates over component failures) and event trees, as practiced in nuclear and aerospace engineering. Core damage frequency is its headline output for reactors; the US NRC’s subsidiary safety goal is 10⁻⁴ per reactor-year. PRA figures are model outputs, not observed rates.

Common-cause failure: a single cause that makes several redundant components fail together. The beta-factor model, documented in NUREG/CR-4780, is the source of this site’s propagation correction.

Defense in depth: multiple barriers against failure. The arithmetic usually assumes the barriers fail independently, and they rarely do.

SIL / ASIL: safety integrity levels under IEC 61508 (SIL 1–4) and ISO 26262 (ASIL A–D). Higher levels require lower failure rates and more rigorous development.

Mechanism design terms

Incentive compatibility: truthful reporting is each participant’s best strategy.
Revelation principle: any outcome some mechanism implements in equilibrium can be implemented by a direct mechanism in which truth-telling is optimal.
Moral hazard: a party changes behavior after an agreement because its actions are not fully observable (hidden action).
Adverse selection: parties with unobservable characteristics self-select in ways that harm the other side (hidden information).

Notation

Symbol	Meaning
DR	Delegation risk (expected loss, dollars per stated unit)
HS	Harm surface (set of harm modes)
P(·), D(·)	Probability and damage of a harm mode
t	P(faithful execution) for one delegation link
pᵢ	Miss probability of check i
ρ	Common-cause weight in the beta-factor mixture; equals the phi coefficient of failure indicators for equal pᵢ
ES_α	Expected Shortfall at level α
HHI	Herfindahl-Hirschman Index, a concentration measure