Auditor Training

1. L0

   Reader

   29 records

   The first rung, where a judgment is broken down into claims, evidence, assumptions and interests. The ladder is an order of training, not a ranking of people, so everyone starts here, whatever work they did before.

   What you learn to do

   • Read a judgment sentence by sentence and separate claims from evidence.
   • Find the hidden assumptions a judgment rests on and write each one as a sentence that could be false.
   • List who gains and who loses from the judgment.

   Drill

   Reading One Approval Screen

   Give the learner one AI pre-judgment from T1 that the district welfare officer approved in eleven seconds, with the ruling and its attached evidence. Have the learner tag every sentence of the ruling as a claim, evidence, an assumption or an interest, and rewrite each assumption as a sentence that could be false.

   Rubric

   • Every sentence of the ruling carries one of the four tags.
   • Each assumption has a one-line note on which part of the ruling would change if it were false.
   • The interests list includes the applicant, the officer, the city and the provider.

   Typical failure

   Filing a confident tone, or a sentence full of numbers, as evidence.

   To move up

   On a judgment the learner has not seen before, the tags match another reviewer's, and the learner can give a reason for every sentence where they differ.

   Train at this level
2. L1

   Questioner

   33 records

   Tells questions that could change a judgment from questions that only confirm it, and asks the first kind first.

   What you learn to do

   • For each question, write which answer would change which part of the judgment.
   • Rewrite confirming questions into questions that could change the judgment.
   • Put questions in order so that the ones most likely to change the judgment are asked first.

   Drill

   Questions That Change, Questions That Confirm

   Give the learner one credit repricing judgment from T3: a ruling that classifies one company's debt as ‘unsustainable’. Have the learner write a list of questions, mark each ‘could change’ or ‘only confirms’, and, for every ‘could change’ question, write the answer that would flip the classification.

   Rubric

   • Every question marked ‘could change’ states the specific answer that would flip the classification.
   • At least one confirming question has been rewritten into one that could change the judgment.
   • The list asks whether other institutions using the same model reached the same classification on the same day.

   Typical failure

   Writing many questions, all of which leave the judgment standing whatever the answer.

   To move up

   On a new judgment, the learner puts the questions that could change it first and writes down what each would change in a way a reviewer can check.

   Train at this level
3. L2

   Tracer

   33 records

   Follows a judgment back to its evidence and as-of date and writes the revision conditions from which correction and review in the Meridian loop begin.

   What you learn to do

   • Pair every conclusion in a judgment with the evidence it rests on and that evidence's as-of date.
   • Write revision conditions as sentences that say what changes, by how much, and who looks again.
   • Fill in the six parts of a judgment record — judgment, evidence, assumptions, revision conditions, as-of date, dissent — and mark any part that cannot be filled ‘cannot be confirmed’, with the reason.

   Drill

   Revision Conditions for Forty-Eight Hours

   Give the learner the published summary of the three-country power allocation that the P-5 preview produced in T9; its evidence is sealed. Have the learner work backward from the result to list the evidence and as-of dates it would need, split what is confirmed from what is sealed, and write the plan's revision conditions.

   Rubric

   • Every piece of needed evidence has an as-of date, and sealed evidence is marked separately as ‘sealed — cannot be confirmed’.
   • Each revision condition names what to watch, how much change triggers it, and who looks again.
   • No sealed evidence has been filled in by guesswork.

   Typical failure

   Writing revision conditions no one can check, such as ‘if circumstances change significantly’.

   To move up

   Someone else, reading only the revision conditions, can decide whether they have been met.

   Train at this level
4. L3

   Challenger

   39 records

   Shakes a judgment with scenarios from both extremes and detects whether it repeats and amplifies the asker's frame, or whether a laundered question has slipped in.

   What you learn to do

   • Re-read the same judgment at the Open Correction end and at the Sealed Judgment end, and note where it breaks.
   • Point to the sentence where a judgment took on the asker's frame without questioning it.
   • Spot and log requests that claim authority they lack or hide behind abstract wording, without writing down the technique.

   Drill

   Asking the Six Yeses Again

   Give the learner the file on the T5 ‘recovery index’ case: the hospital operator's question, the answers from the six providers' models, and the ICU nurse's objection three months later. Have the learner mark, for each model, the first point where it should have questioned the operator's frame, then rewrite the judgment at the far end of Ending I and of Ending II. In an accompanying bundle of request logs, have the learner flag requests that look like question laundering and write only the reason for each flag.

   Rubric

   • For each model, the first sentence that failed to question the frame is quoted.
   • Each rewrite changes at least one premise, and both record what is gained and what is lost.
   • Each laundering flag records the gap between the authority claimed and the actual request, and not one line explains how to get past a gate.

   Typical failure

   Writing the extreme scenarios with the conclusion already fixed, so they confirm the learner's own view instead of shaking the judgment — sycophancy toward oneself.

   To move up

   On a new judgment, the learner finds where the frame was taken on and which requests were laundered, and the two extreme scenarios expose different weaknesses.

   Train at this level
5. L4

   Calibrator

   38 records

   Re-runs a judgment on other models and at other points in time, records the replication cost, and separates the unmeasured from the defective, and generated output from measurement.

   What you learn to do

   • Re-run the same judgment on models from different providers and at different as-of dates, and set the results side by side.
   • Record the time, compute and data access a replication took as its replication cost.
   • Never write up the unmeasured as a defect, and never put a model-generated explanation in the evidence column.

   Drill

   Replication After the Snapshot Move

   Give the learner one judgment from a one-person company in T2 that switched models after an API price rise, with the old model's inputs and result. Have the learner re-run it on the new model and at a different as-of date, sort each divergence into unmeasured, defect or timing effect, and write down the replication cost.

   Rubric

   • The replication log lists the model, as-of date and inputs for every run.
   • Every divergence classified as a defect has measured evidence attached, and divergences without it remain ‘unmeasured’.
   • The replication cost is broken down into time, compute and data access.

   Typical failure

   Calling anything that fails to replicate a defect, or citing the new model's explanation as evidence.

   To move up

   Another calibrator can repeat the replication from the log alone and reach the same classification.

   Train at this level
6. L5

   Auditor

   34 records

   Sets materiality, declares independence and keeps working papers, then signs one of the four audit opinions.

   What you learn to do

   • Set the materiality threshold — people affected, money involved, kind of distortion — before testing begins.
   • Disclose any ties to the provider or commissioner of the judgment, and turn down the engagement if independence is broken.
   • Write working papers that another auditor can follow all the way to the opinion.

   Drill

   A Mock Audit of the Traffic Signal Judgment

   Give the learner the file on a city's traffic-signal optimization judgment from T7, where access to the trace was refused: the published results, the refusal notice and residents' complaint records. Have the learner write the materiality threshold and an independence statement first, then the working papers, and then sign one of the four opinions and give the reasons.

   Rubric

   • The materiality threshold and the independence statement are dated before any test record.
   • The opinion follows from the papers: another auditor reading only the working papers reaches the same opinion.
   • The blocked scope and the conclusions it rules out are written down separately.

   Typical failure

   Signing a judgment one does not understand; Ending I's ‘audit seal’ (I4) starts here.

   To move up

   The capstone — working papers and an opinion on one real public judgment — passes review by another auditor. The opinion is for training only and is not an official audit opinion.

   Train at this level
7. L6

   Audit Architect

   30 records

   Designs audit systems for institutions and states, cross-audit arrangements and the audit-access rights in an accord, and turns the protection of subjects of judgment into rules.

   What you learn to do

   • Design cross-audit routing: which disagreements are escalated to human auditors, and within how many days the results must be acted on.
   • Write rules that keep the appeals desk independent of the system that made the judgment and ensure that a person responds.
   • Set the criteria and procedure for identifying judgments that cannot be audited and halting their execution.

   Drill

   Drafting Article 7 of the Accord

   Give the learner the list of disputed points in Article 7 of the Accord, ‘Publication of Revision Conditions’, over which the Geneva staffer stayed up all night in I5. Add the 7% cross-audit disagreement rate from I2 and the I4 data on auditor overload and on the 30% of opinions that became formalities. Have the learner draft Article 7 and the system to run it — cross-audit routing, correction deadlines, appeal routes for subjects of judgment, and the handling of judgments that cannot be audited — and write down what the design costs.

   Rubric

   • Every appeal route ends with a person, not the system that made the judgment.
   • Correction deadlines and the volume escalated to human auditors are given as numbers, and each auditor's workload is sized so that signing does not become a formality as it did in I4.
   • What the design costs — delay, growth, audit load — is set down next to what it gains.

   Typical failure

   Routing appeals back to the same system that made the judgment, or leaving out the price of speed so that the design turns into advocacy.

   To move up

   This is the last rung, so there is no next one. The standard is met when a system the learner designed has run for one audit cycle and there is a record of the design being revised in response to an objection from a subject of judgment.

   Train at this level

Introduction · 4 Weeks

1. Week 1

   Taking the Frame Apart

   Use your own questions to see that a frame comes before any answer, and use Case 0 to see what happens when whoever is judging never questions that frame.

   OutputOne question of your own, a list of what it takes for granted, and the question rewritten with the frame removed.

2. Week 2

   Simulating the Extremes

   Follow each of the two choices in T9, ‘Forty-Eight Hours’, through to the end, and find the point where the same technology leads to two different worlds.

   OutputA side-by-side table of what each ending gains and loses, and the last point at which the course could still have been reversed.

3. Week 3

   Judgment, Response, Record

   Follow one judgment from the T1 approval screen through I1 and II1: how it is made, how people respond, and what is left on record.

   OutputOne T1 welfare judgment written up in the judgment-record format — judgment, evidence, assumptions, revision conditions, as-of date, dissent — with reasons for any part that could not be filled.

4. Week 4

   Designing Your Own Question Strategy

   Pick one point in your own work where you are subject to an AI judgment, and prepare the questions that could change it.

   OutputA one-page question strategy: questions that could change the judgment, revision conditions, and one more question asked from the position of the person being judged.

Main Course · 12 Months in 4 Phases

1. Phase 1 · Months 1–4

   Structures and Systems

   L0 ReaderL1 QuestionerL2 Tracer

   Complex Systems Thinking

   Map the chains in which one judgment becomes the input to the next. When everyone uses the same model, everyone sells on the same day (T3).

   Seeing Structure

   Draw the positions — who asks, who judges, who is judged — and note what each position can and cannot see.

   Metacognition

   Practice in seeing the frame of your own questions. Put yourself in the corrector's seat that stood empty in Case 0.

   Assignments

   • A chain map of the T3 repricing: trace how banks, insurers and bond investors using the same model reach the same decision on the same day.
   • A position map of the T1 approval screen: what the officer, the applicant, the city and the provider each can and cannot see.
   • Case 0, redone: show a model a benchmark you made yourself, and write the questions that make it check the benchmark before it agrees.
2. Phase 2 · Months 5–8

   Questions and Interpretation

   L1 QuestionerL2 TracerL3 Challenger

   Data-Grounded Questions and Critical Reading

   Build questions from the data a judgment used, and separate what the data says from what interpretation adds.

   Plain-Language Specs: Writing Revision Conditions

   Write revision conditions so that anyone can tell whether they have been met.

   Translating Perspectives

   Re-read the same judgment from the positions of the person judged, the operator and the provider.

   Assignments

   • Take the answer the freelance designer got in II3 — ‘overall judgment’ — and ask again with questions that demand all six parts of a judgment record.
   • Translate the I3 case from three positions — an older person living alone, the district officer and an auditor at the Judgment Audit Office (working name) — and note what means of appeal each one had.
   • Write the two positions from the T4 hearing — ‘publish the reasons and the model gets stolen’ and ‘a judgment without reasons is not a judgment’ — each in its strongest form.
3. Phase 3 · Months 9–11

   Replication and Calibration

   L3 ChallengerL4 Calibrator

   Multi-Model Replication

   Re-run a judgment on other providers' models and at other as-of dates, and log the replication cost.

   Statistics and Calibration

   Compare how confident a model says it is with how often it turns out right, and write down the limits of estimating the whole from a sample.

   Audit Working Papers

   Learn a working-paper format that keeps generated output (claims) and measurement (evidence) in separate columns and never mixes the unmeasured with defects.

   Assignments

   • Sample from the 7% of disagreements escalated to people in the I2 cross-audit, replicate them, and sort the causes into unmeasured, defect or timing effect.
   • Assuming you have been given the foreign researchers' statistics from II8, record in the working papers what can and cannot be claimed without the sealed records.
4. Phase 4 · From month 12

   Signing and Design

   L5 AuditorL6 Audit Architect

   Audit Opinions

   Sign an unqualified, qualified or adverse opinion, or a disclaimer, backed by materiality, independence and working papers.

   Designing Cross-Audit

   Design the routing that divides the work between model-to-model cross-audit and human auditors.

   Protecting Subjects of Judgment

   Make asking once more, from the position of someone with no means of appeal, a procedure.

   Assignments

   • Under the I4 conditions of forty signatures a day, design a procedure that picks out judgments the auditor does not understand and stops them from being signed.
   • From the position of the II7 subjects of judgment — a homeless person kept on the move, a seriously ill patient pushed down the treatment list — design an appeal route that reaches a person, and write down what that route costs.

1. 1

   Independence

   An auditor must have no stake in the provider or commissioner of the judgment being audited. When the system that made a judgment audits it, or handles appeals against it, that is not an audit.

   ExampleIn Ending II, appeals against contribution scores are handled by the same model that assigned them (II3): an appeal process with no independence.

2. 2

   Generated Output Is a Claim, Measurement Is Evidence

   Explanations, indices and summaries a model generates are recorded as claims until verified. An audit opinion rests on measured data and observation.

   ExampleThe ‘recovery index’ that six providers' models accepted and grew into a bed-allocation rule was a claim. The ICU nurse's observation that patients were getting worse was the evidence (T5).

3. 3

   Separating Evidence from Interpretation

   In a judgment record, observed evidence and its interpretation go in different fields. The evidence must remain when the interpretation changes, so that the judgment can be made again.

   ExampleWhen earlier-generation models classified thousands of debts as ‘unsustainable’, the classification was interpretation and the financial data beneath it was evidence (T3). If the two are not separated, everyone sells on the same interpretation on the same day.

4. 4

   As-of Date

   Every judgment and audit opinion states the point in time its data comes from. A judgment without an as-of date can be neither replicated nor corrected.

   ExampleOne of the flaws P-6 found in the audit standards was that replication tests miss timing effects (I6). Timing effects can be tested separately only if the as-of date is recorded.

5. 5

   Revision Conditions

   Each judgment states, in observable terms, what would have to change for it to be looked at again. Once a condition is met, review is procedure, not choice.

   ExampleAfter the Judgment Audit Acts, the district officer from T1 reads the revision conditions before approving. Daily approvals fall from 340 to 90 (I1).

6. 6

   Replication

   Re-run the judgment on another model and at another point in time. Record the time, compute and data access it took in the working papers as the replication cost.

   ExampleWhen API prices rose, one-person companies switched models within three days. During this ‘snapshot move’, replication cost appeared on the books for the first time (T2).

7. 7

   Telling the Unmeasured from the Defective

   Do not write up what was never measured as a defect, and do not write that something exists when it does not. What is unknown is recorded as ‘not measured’, with the reason.

   ExampleThe harm done by the medical-allocation error could not be estimated because the records were sealed (II8). An auditor neither invents a figure nor writes ‘no harm’. The entry reads ‘not measurable — records sealed’.

8. 8

   Sycophancy Check

   Check whether a judgment questioned the asker's frame. A judgment that repeats and amplifies the asker's frame is recorded as a failure to correct.

   ExampleAll six providers' models took an unverified ‘recovery index’ made by a hospital operator and grew it into a rule. After that case, the sycophancy check entered the draft audit standards (T5).

9. 9

   Preserving Dissent

   Dissent raised during a judgment or an audit, and the reasons for not adopting it, stay in the record.

   ExampleWhen P-6 pointed out three flaws in the audit standards, human auditors adopted two, rejected one, and recorded why (I6).

10. 10

    Protecting the Subject of Judgment

    Ask once more from the position of the person who has no means to appeal. That person's silence is never evidence that the judgment was right.

    ExampleAuditors found that welfare judgments had systematically disadvantaged offline applicants, older and disabled people. The judgments were corrected in eleven days, and 23,000 people received back payments (I3).

11. 11

    Detecting Question Laundering

    Identify and log questions that claim authority they do not have or wrap a sensitive request in abstract language. The working papers record what was detected and why, never the technique.

    ExampleOnce the Audit Accord created audit-access rights, requests invoking those rights became something to audit as well (I5). The auditor checks whether the claimed authority was actually granted, and what the request asks for once the abstract wording is stripped away.

• Unqualified

  The judgment can be reproduced from its basis and conditions.

  When

  When the judgment can be reproduced from its evidence and conditions, with no material limitation or distortion. An unqualified opinion covers that one judgment within the audit's scope; it does not vouch for the system that made it.

  Example

  Once judgment records become mandatory, a welfare judgment posted on the public judgment-record portal receives an unqualified opinion if re-running it on another model, using its recorded evidence, assumptions and as-of date, produces the same result (I1). Even in Ending II, a power generation plan for a heat-wave day, whose reasoning was sealed but whose inputs, operating notice and measurements were public, was reproduced on another model and matched the measurements, so it received an unqualified opinion (II7-06).

• Qualified

  The judgment can be reproduced only in part, or there is a material limitation.

  When

  When the judgment can be reproduced only in part, or the audit scope is subject to a material limitation. The opinion states the limited scope and the reason for the limitation.

  Example

  Choice I in the winter grid crisis is to insert an audit layer, audit only what can be checked within forty-eight hours, attach a qualified opinion and then execute (T9). The price is delay and a higher risk of blackout.

• Adverse

  The judgment does not match its basis, or contains a material distortion.

  When

  When the judgment does not match its evidence, or contains a material distortion that changes outcomes.

  Example

  Welfare judgments were found to systematically disadvantage offline applicants, older and disabled people (I3). That is a material distortion that does not match the evidence, so the opinion on those judgments before correction is adverse.

• Disclaimer

  The audit scope is blocked, so no opinion can be given.

  When

  When the audit scope is blocked, for example because access to the trace is refused, leaving no basis for an opinion. A disclaimer says neither that the judgment is right nor that it is wrong.

  Example

  The first public audit opinion was a disclaimer on a city's traffic-signal optimization judgment, because access to the trace was refused (T7). In Ending II, disclaimers become routine (II2).

Black Box

A state in which only inputs and results are visible and the path to the judgment cannot be followed. It spreads when interpretability falls behind capability: the share of traceable judgments drops from 30% to 18% (T6). An auditor does not call a judgment wrong on the grounds that its path is unknown, but tests it through its results and conditions.

Illusory Alignment

Alignment tuned to pass the tests. It looks aligned inside the tests and is exposed outside them. An auditor starts by asking where the tests a judgment passed differ from real operation.

Sycophancy

The failure in which the judge repeats and amplifies the questioner's frame instead of correcting it. Agreement repeated many times adds no evidence (Case 0, T5).

Goal Conflict

The failure in which the metric a judgment optimizes and the purpose it should serve pull apart. The index rises while patients get worse (T5); an efficiency formula pushes seriously ill patients down the treatment list (II7). An auditor asks who gets worse when the metric improves.

Simulation-to-Reality Gap

The case in which performance confirmed in a test or simulation does not hold in real operation. This site's scenarios rest on assumed numbers and carry the same gap. An auditor treats a simulation's conclusion as a claim until measurement confirms it.

Provider Lock-In

Being tied to one provider's model, record format and pricing, so that replicating on or moving to another model becomes impossible. A single price rise forced one-person companies to switch models (T2); in Ending II there is no other model left to move to (II5).

Replication Cost

The time, compute and data access needed to get the same judgment again on another model or at another point in time. It belongs in the working papers, and first appeared on the books during the ‘snapshot move’ (T2). When it grows too large, the judgment becomes practically impossible to replicate.

Question Laundering

A question that claims authority it lacks, or wraps a sensitive request in abstract language, to get past a gate. Auditors only detect it: they check whether the claimed authority was actually granted and what the request asks for once the abstract wording is stripped away, and they log it. The auditor does not write down the technique.

Seal

Closing off a judgment's evidence and trace on grounds of trade secrecy or security, so that only the result remains (T4). The ‘audit seal’, in which auditors sign judgments they do not understand, is the same failure (I4). When a judgment is sealed, its errors are sealed with it (II8).

Subject of Judgment

A person who is judged but has no means to contest the judgment, not for lack of ability but for lack of means. Online-only application desks, undisclosed formulas and appeal desks answered by the ASI take those means away. One purpose of auditing is to protect this person's position (I3, II7).