Delegation Risk Calculator

Illustrative tool

The calculator shows how the parts of a delegation-risk estimate combine. Its output is only as good as its inputs, and for most AI harm modes the inputs are elicited guesses. The default harm modes are invented for illustration and describe no real system.

Risk budget (expected loss per month), $

Delegation risk: $1,697 per month (within budget)

Expected loss. 90% interval from uncertainty about the rates: $412 to $4,324.

Realized loss in a single month (20,000 simulated months)

Months with no incident

7.4%

Median month

$527

90th percentile

$1,926

99th percentile

$5,818

Months with loss above budget

9.2%

$3log scale; red bins start at or above the budget$994k

Harm modesAdd harm mode

Remove

Actions per monthP(harm) per action, 5th pctP(harm) per action, 95th pctDamage per incident, 5th pct ($)Damage per incident, 95th pct ($)

Mean P per action 0.00081 · 4.0 incidents per month · mean damage $161 · expected loss $652 per month (38.4% of total)

Remove

Actions per monthP(harm) per action, 5th pctP(harm) per action, 95th pctDamage per incident, 5th pct ($)Damage per incident, 95th pct ($)

Mean P per action 0.00027 · 0.27 incidents per month · mean damage $593 · expected loss $158 per month (9.3% of total)

Remove

Actions per monthP(harm) per action, 5th pctP(harm) per action, 95th pctDamage per incident, 5th pct ($)Damage per incident, 95th pct ($)

Mean P per action 0.0000013 · 0.0067 incidents per month · mean damage $133k · expected loss $887 per month (52.3% of total)

Export the expected-loss model as Squiggle code

Inputs

Each row is a harm mode, one way a delegated action can cause damage (see Risk Decomposition). A row takes three inputs:

• Actions per month. How often the agent takes the action that exposes this harm. A probability means nothing until you say what it is per. Here it is per action, and the volume turns it into a monthly figure.
• P(harm) per action, as a 90% interval. This is your uncertainty about the rate. The calculator models it as a logit-normal with those 5th and 95th percentiles. For a point estimate, enter the same number twice. Both ends must be above zero.
• Damage per incident, as a 90% interval. This is the spread of cost across incidents, modeled as a lognormal. A lognormal’s mean exceeds its median, and expected loss uses the mean: an interval of $20 to $500 has a median of $100 and a mean of $161. Each row’s summary line shows the mean.

Expert Elicitation covers how to get defensible intervals when you have no data.

Two different answers

Delegation risk is the expected loss, $\sum_m N_m \, \mathbb{E}[p_m] \, \mathbb{E}[D_m]$ per month. This is the number to compare with a risk budget. It is also the number that adds across components and subagents (see consolidation). The interval beside it reflects uncertainty about the rates only.

Realized loss is what a single month actually costs. For each simulated month, the calculator does three things for every row:

1. draws a rate;
2. draws a binomial number of incidents at that rate;
3. draws a damage for each incident.

It repeats this 20,000 times. The resulting distribution answers different questions: how often a month’s loss exceeds the budget amount, and how bad the 99th-percentile month is.

The default example shows why the two need separating. Expected loss is about $1,700 per month, inside the $2,000 budget. About half of that comes from the record-disclosure row, which is expected roughly once in 12 years. Because it occurs in fewer than one month in a hundred, the 99th-percentile month is set by the two routine rows, wrong answers (accident) and injected refunds (hijack). With inputs like these, a run of quiet months says little about the row that dominates expected loss. Rare, high-damage harm modes therefore need an exposure bound and an elicited estimate. A tally of incidents cannot tell you their rate.

Assumptions and limits

• Independence. Incidents are independent given the rates, and uncertainty about different rows is independent. Correlation leaves expected loss unchanged but fattens the tail of realized loss. When harm modes share a cause, the percentiles are therefore optimistic.
• Adversaries. An adaptive adversary does not have a fixed rate. For the hijack and defection channels, a rate describes current attacks against current defenses. Assume the defenses fail together (Risk Propagation).
• Unlisted harms. Harm modes you have not listed are absent from the total. The calculator prices the harm surface you enumerate, not the one you have.
• Large incident counts. When one row has more than 200 incidents in a simulated month, their sum is drawn from a lognormal with the same mean and variance rather than one draw at a time.
• Squiggle export. The export is a Squiggle model of the expected-loss part only, tested with @quri/squiggle-lang 0.10.0. The realized-loss simulation is not included.

Bringing a row under budget

Expected loss is a product, so there are four levers:

• Lower the rate with verification and monitoring: Trusted Monitor, or a Voting Tribunal whose members genuinely fail independently.
• Lower the damage with an Exposure-Envelope Gateway or Checkpoint-Rollback.
• Lower the volume by routing fewer actions through the risky path, for example sending high-stakes cases up an Escalation Ladder.
• Split the row into narrower components, each with smaller damage.

Quick Start puts these into a procedure.