CFA v3 pipeline: where the 13 minutes go, what the prompts do, and what changes

Measured on the two 60-lesson benchmark runs (18 to 19 Sep 2026), read from the code, and stress-tested by critic panels. Numbers are medians unless stated.

Sources: artifacts/cfa-benchmark/cfa-v3-bench60c (all stages at max reasoning) and cfa-v3-bench60d (planner and reviewer at high) · step-by-step inputs and outputs from cfa-v3-trace, the first run on Bedrock (21 Sep 2026) · profiling script /tmp/cfa-profile/profile.py · prompts in cfa/prompts.v3.ts · companion deck: CFA lessons on Oncourse

Why 13 minutesThe flow, stage by stageReal generations, step by stepHow the checks workThe prompts, stage by stageSlowness and parallelismThe flow after the changesPrompt anatomyDefect → prompt causePrompt rewriteJev: today and proposedTopic lessons → revision notesCalculatorVerifiedRoadmap

Why a lesson takes 13 minutes

Three serial Luna calls at max reasoning, each emitting 25k to 32k reasoning tokens at a flat 142 tokens per second, plus a repair call on two lessons in three. Reasoning is 81% of output tokens and 83% of wall-clock. Everything else (static checks, Jev, media, files) is under 1%.
13.3 min
median lesson, all stages at max (p90 17.1). Chapter 12.3, topic 14.7. No repair 11.0, with repair 14.7
6.7 min
median clean lesson with planner and reviewer at high. Tempting, and rejected: the same run designed 4.5 fixture calculations per plan instead of 7.7. See the reversal below
142 tok/s
output throughput in every stage and run (p10 128, p90 155). Duration is output tokens ÷ 142
81%
of output tokens are reasoning (88k of 107k per lesson); text is ~5k per stage
120 s
mean repair time per lesson (15%): presentation repair on 35 of 60 lessons at 98 s, Jev repair 17 of 60 at 113 s, item regeneration 7 of 60 at 51 s
$0.14
per lesson at max ($8.41 for 60, 97% output tokens); $0.075 with planner and reviewer at high; Jev $0.0009
Where the 790 s mean lesson goes (bench60c)
planner 221 s
writer 220 s
repairs 120 s
reviewer 239 s
Static checks 0.4 s (four lessons hit a 63 s cold Mermaid import), Jev 1.1 s, media ≈ 0 (no external images), files 0.1 s.

Provider note. The 60-lesson profile above was measured on the direct OpenAI Responses API. Every stage now runs the same gpt-5.6-luna on AWS Bedrock (bedrock-mantle.responses), and the pipeline fails closed if a stage reports any other model or provider. Throughput is unchanged: across the model calls in the traced run below, output ran at 149 to 204 tokens per second against the 142 measured before, so the timings here still hold.

Three findings that were not in the plan

The flow, stage by stage (today)

Every lesson is an independent chain; the runner already parallelises lessons (30 concurrent Luna streams measured safe, 60 produced errors). Inside the chain nothing meaningful can run in parallel: each model stage needs the previous one's output.

Chapter lesson, bench60c medians · widths are proportional to time · dashed = bounded repair, fires on the share of lessons shown

Contexthierarchy row, LOS, family, sealed sources
PlannerLuna · 3.7k in → 5.6k text + 25k reasoning · 221 s
Plan gatezod + identity + fixture compile · 1 JSON repair
WriterLuna max · 13.6k in → 4.5k text + 27k reasoning · 220 s
Static passtemplate, budgets, lint, Mermaid, fences · 0.4 s
Presentation repairLuna max · 98 s · 35/60 lessons
Jev structural11 booleans · 0.5 s · $0.0004
Jev repairLuna max · 113 s · 17/60
Media + practiceSVG, calculator tables, fence validate · ≈0 s
Teaching reviewLuna max · 7.7k in → 5.2k text + 32k reasoning · 239 s
Arithmeticrecompute quoted claims
Jev items0.3 s / item
Item regen51 s · 7/60
Evidenceplan, fixtures, practice, findings, hashes
Luna model stagedeterministic code gateJev (TypeSafe)bounded repair (model)
#StageWhoInputOutputGate after itMedian sShareAt high
1PlannerLuna, max → highLOS + family + sources, 3.7k tokplan JSON 5.6k text + 25k reasoningplan schema, identity, fixture compile; 1 JSON repair22128%69
2WriterLuna, maxprompt + plan + fixtures, 13.6k tokmarkdown 4.5k text + 27k reasoningstatic pass (template, budgets, lint, Mermaid, fences)22028%245 (unchanged)
2aPresentation repairLuna, max6.3k tok~14k outputstatic pass again98 (35/60)7.5%77
3Jev structuralJevplan + lesson + LOS, ~8k tok11 booleans<0.7 on the repairable set → 1 repair0.50.1%–
3aJev repairLuna, maxlesson + failed questionswhole lesson rewritestatic + Jev again113 (17/60)4.4%–
4Media, calculators, practicecodemarkdownSVG figures, tables, validated fenceserrors recorded (not gating until after review)≈00%–
5Teaching reviewLuna, max → highlesson + context, 7.7k tokissues, numeric claims, expert gaps; 5.2k text + 32k reasoningschema; 1 JSON repair; arithmetic checker23928%84
6Jev item QAJevitem + LOS + fixtures, ~1.5k tok3 booleans + difficultysingle_defensible_key <0.6 → regenerate item0.60.1%–
6aItem regenerationLuna, maxitem + LOSone itempractice re-validate51 (7/60)0.9%–

Output caps (65,536 per stage) were never hit: reviewer p90 48k, planner 40k, writer 37k. Lowering caps cannot speed anything up and, because Luna bills reasoning against the cap, would only cause truncations.

Real generations, step by step

Every number below is read from artifacts/cfa-benchmark/cfa-v3-trace, produced by pnpm cfa:benchmark --trace on 2026-09-21. The run records what each step received and what it returned, so this is the pipeline as it actually ran, not a reconstruction. Text stages are gpt-5.6-luna on bedrock-mantle.responses at reasoning effort max.

Click any row or button to read the complete text — the whole prompt, the whole completion, the finished lesson, rendered. Nothing here is truncated; all 45 steps are embedded in this page.

Chapter · Sole proprietorships, partnerships and corporations

CFA Level I · Corporate Finance (Level I) · Organizational Forms, Corporate Issuer Features, and Ownership · c34323f7-1ad3-4bcf-bc06-c010a06f4e28 · 12 steps · 255.4 s total, 98% of it waiting on the model

#StepRuns onInputOutputTime
1 planner Bedrock · gpt-5.6-luna · max 11,286 chars · 2,653 tok 16,047 chars · 7,456 tok · 4,347 reasoning 49.9 s
2 fixtures local code 1,407 chars 1,768 chars 0 ms
3 writer Bedrock · gpt-5.6-luna · max 35,431 chars · 7,661 tok 13,199 chars · 26,917 tok · 24,334 reasoning 159.8 s
4 static-checks local code 13,538 chars 547 chars 830 ms
5 jev-structural Jev 34,095 chars 8,094 chars 1.3 s
6 calculators local code 15,069 chars 13,559 chars 0 ms
7 media local code 13,765 chars 36,218 chars 3 ms
8 practice-validation local code 3,924 chars 4,825 chars 2 ms
9 reviewer Bedrock · gpt-5.6-luna · max 23,730 chars · 5,054 tok 4,173 chars · 6,682 tok · 5,696 reasoning 41.8 s
10 jev-item-qa Jev 3,582 chars 6,212 chars 721 ms
11 jev-item-qa Jev 3,629 chars 6,226 chars 499 ms
12 jev-item-qa Jev 4,298 chars 6,226 chars 606 ms

Chapter · Key features of corporate issuers

CFA Level I · Corporate Finance (Level I) · Organizational Forms, Corporate Issuer Features, and Ownership · 3ef1a48c-0c08-4e80-876c-c66d3d861413 · 13 steps · 280.9 s total, 99% of it waiting on the model

#StepRuns onInputOutputTime
1 planner Bedrock · gpt-5.6-luna · max 11,265 chars · 2,652 tok 12,505 chars · 4,522 tok · 2,119 reasoning 30.1 s
2 planner attempt 2 Bedrock · gpt-5.6-luna · max 11,666 chars · 2,765 tok 1,841 chars · 7,202 tok · 6,732 reasoning 47.0 s
3 fixtures local code 2 chars 2 chars 0 ms
4 writer Bedrock · gpt-5.6-luna · max 20,424 chars · 4,848 tok 12,112 chars · 25,116 tok · 22,603 reasoning 153.6 s
5 static-checks local code 12,593 chars 5,421 chars 23 ms
6 jev-structural Jev 20,321 chars 8,097 chars 649 ms
7 calculators local code 12,611 chars 12,614 chars 0 ms
8 media local code 12,820 chars 12,732 chars 1 ms
9 practice-validation local code 4,727 chars 4,170 chars 1 ms
10 reviewer Bedrock · gpt-5.6-luna · max 21,416 chars · 4,639 tok 5,256 chars · 7,398 tok · 6,337 reasoning 47.9 s
11 jev-item-qa Jev 2,165 chars 6,221 chars 581 ms
12 jev-item-qa Jev 2,624 chars 6,213 chars 516 ms
13 jev-item-qa Jev 2,072 chars 6,210 chars 478 ms

Topic · Organizational Forms, Corporate Issuer Features, and Ownership

CFA Level I · Corporate Finance (Level I) · Organizational Forms, Corporate Issuer Features, and Ownership · c674c4ec-a820-42c9-a7f1-ecbfc9ef8c22 · 20 steps · 514.1 s total, 99% of it waiting on the model

#StepRuns onInputOutputTime
1 planner Bedrock · gpt-5.6-luna · max 14,614 chars · 3,461 tok 26,872 chars · 10,153 tok · 4,980 reasoning 63.5 s
2 fixtures local code 1,251 chars 1,659 chars 0 ms
3 writer Bedrock · gpt-5.6-luna · max 46,740 chars · 9,873 tok 36,215 chars · 29,071 tok · 21,707 reasoning 171.4 s
4 static-checks local code 35,456 chars 15,906 chars 46 ms
5 writer-presentation-repair attempt 2 Bedrock · gpt-5.6-luna · max 43,565 chars · 9,329 tok 33,548 chars · 14,148 tok · 7,342 reasoning 69.1 s
6 static-checks local code 35,262 chars 21,891 chars 25 ms
7 jev-structural Jev 74,572 chars 14,730 chars 1.3 s
8 writer-jev-repair attempt 3 Bedrock · gpt-5.6-luna · max 34,016 chars · 6,930 tok 35,433 chars · 23,035 tok · 15,883 reasoning 126.7 s
9 static-checks local code 37,197 chars 25,386 chars 23 ms
10 jev-structural Jev 76,557 chars 14,721 chars 1.1 s
11 calculators local code 38,566 chars 37,218 chars 1 ms
12 media local code 37,424 chars 36,981 chars 2 ms
13 practice-validation local code 21,238 chars 8,855 chars 1 ms
14 reviewer Bedrock · gpt-5.6-luna · max 33,753 chars · 7,033 tok 10,804 chars · 12,549 tok · 10,193 reasoning 77.7 s
15 jev-item-qa Jev 3,643 chars 6,224 chars 699 ms
16 jev-item-qa Jev 3,574 chars 6,223 chars 515 ms
17 jev-item-qa Jev 3,558 chars 6,226 chars 524 ms
18 jev-item-qa Jev 3,743 chars 6,226 chars 377 ms
19 jev-item-qa Jev 3,702 chars 6,224 chars 569 ms
20 jev-item-qa Jev 3,769 chars 6,223 chars 507 ms

How the checks work

Three layers, cheapest first. Only the first two can send work back for repair; the third reports and never rewrites.

1 · Static checksPlain code, no model. 33 rules over the Markdown plus validators for every machine fence. Runs in under a second. A major or critical finding triggers one presentation repair.
2 · JevA small evaluation model asked yes/no questions about the lesson; each answer is a probability. 11 questions for a one-LOS chapter, more for a module. Seven question families below 0.7 trigger one structural repair; the rest are recorded.
3 · Teaching reviewLuna reads the finished lesson as a tutor and returns a verdict, a list of issues, and every numeric claim as a checkable expression. Our arithmetic checker recomputes those. Nothing is rewritten.

What each layer found in this run

LessonStatic, after repairsJev below 0.7Tutor verdictRepairs fired
Chapter · Sole proprietorships, partnerships and corporations 0 major / 1 minor 1 of 11 changes_requested · 2 issues none
Chapter · Key features of corporate issuers 0 major / 2 minor 0 of 11 changes_requested · 3 issues plan
Topic · Organizational Forms, Corporate Issuer Features, and Ownership 13 major / 2 minor 12 of 31 changes_requested · 7 issues presentation, jev

Layer 1 · every static rule read from presentation.ts and markdownLint.ts

CodeSeverityWhat it catchesEffect
ascii_diagram major ASCII text diagrams are not supported on the CFA learner renderer repair
ascii_multiplication major An asterisk is used as a learner-facing multiplication sign outside math mode repair
dangling_figure_reference major A sentence introduces a figure that is not there; drop the sentence or supply the figure repair
display_math_disabled major Display math is not enabled by the supplied renderer profile repair
inline_math_disabled major Inline math is not enabled by the supplied renderer profile repair
los_heading_mismatch major Chapter H2 headings must match each scoped LOS verbatim apart from case and trailing punctuation repair
markdown_lint_malformed_table major Markdown table rows have inconsistent column counts repair
markdown_lint_raw_html major Raw HTML is not allowed in CFA lesson Markdown repair
markdown_lint_table_too_wide major Markdown tables may contain at most three columns repair
markdown_lint_unbalanced_math major Math delimiters are unbalanced repair
math_in_inline_code major Mathematical notation is hidden in inline code instead of supported math delimiters repair
mermaid_capability_disabled major Mermaid is not enabled by the supplied renderer profile repair
pipeline_language_leak major Learner-facing prose must not expose source-packet or pipeline limitations repair
practice_block_missing major The v3 lesson has no cfa-practice fence repair
raw_math_operator major A caret or underscore expression is outside a supported math delimiter repair
template_labels_in_prose major A bold template label names a pipeline step to the learner; write the sentence without it repair
unbalanced_display_math major A $$ display-math delimiter is unclosed repair
unbalanced_inline_math major A single-dollar math delimiter is unclosed; escape currency dollars or use a complete $...$ span repair
unbalanced_math_delimiter major A math delimiter is unclosed or has no matching opener repair
unsupported_math_delimiter major CFA v3 math must use only $...$ or $$...$$ delimiters repair
unsupported_math_tilde major A tilde is not allowed inside CFA math; use an explicit supported command or prose repair
visual_misuse major A payoff or curve is drawn as a flowchart; use a cfa-svg figure or a table for numeric nodes repair
wide_markdown_table major A phone lesson table has more than three columns; split the comparison and move reasoning into nearby prose repair
empty_markdown critical The lesson has no learner-facing Markdown recorded
display_math_setext_collision major A display-math line contains only equals signs or hyphens and can be parsed as a Setext heading recorded
markdown_lint_unresolved_chapter_marker major A chapter link marker remains unresolved recorded
calculator_heading_orphaned minor A Calculator heading has no calculator block; remove the heading with the block recorded
dense_markdown_table minor A table contains paragraph-length cells; keep comparison labels concise and explain the reasoning below it recorded
mermaid_too_large minor Mermaid diagrams may use at most 8 nodes and labels of at most 4 words recorded
practice_format_mismatch minor Practice formats differ from the plan after dropping an invalid item, or a Level III chapter block lacks a cfa_constructed_response with a number or choice field recorded
prose_before_first_heading minor Chapter prose appears before the first H2; move it under the heading after the On the exam line recorded
raw_float_in_prose minor A raw floating-point expansion reached the learner; show the fixture value at its displayed precision recorded
trap_duplicated minor The unit states its trap twice; keep the Common error section and drop the callout recorded

Fence validators read from practice.ts and calculator.ts

A quiz block or a calculator block that breaks one of these is dropped or the lesson fails; the writer never gets to argue.

FenceRule
cfa-practicepractice is not planned for unit …
cfa-practicepractice unitId … does not match plan unit …
cfa-practice… is not allowed at Level …
cfa-practicenumeric key does not match compiled fixture …
cfa-practicepractice block has no valid items (${dropped.map(entry =>
cfa-practicepractice formats … do not match planned formats …
cfa-practicepractice item-set child count does not match planned count …
cfa-practiceInvalid cfa-practice JSON: …
cfa-practicepractice block contains LOS outside its unit plan
cfa-practiceitem LOS IDs must be a subset of the block LOS IDs
cfa-practiceoption D is allowed only at Level III
cfa-practiceInvalid cfa-practice fence; use exactly three backticks and no trailing info tokens
cfa-calculatorcalculator references unknown fixture …
cfa-calculatorcalculator references unknown calculation …/…
cfa-calculatorcalculator needs at least … steps; found …
cfa-calculatorcalculator step … repeats the table header; a block serialises to exactly one header row
cfa-calculatorcalculator step … computes before any input is keyed in
cfa-calculatorcalculator display … is not a number
cfa-calculatorcalculator display … is neither a fixture value nor derivable from the keys …
cfa-calculatorcalculator final display … does not match compiled value …; fixture conventions: …
cfa-calculatorInvalid cfa-calculator JSON: …
cfa-calculatorcalculator final display is missing
cfa-calculatorInvalid cfa-calculator fence; use exactly three backticks and no trailing info tokens

Layer 2 · the questions Jev was asked verbatim from the first structural call, Chapter · Sole proprietorships, partnerships and corporations, with its answers

QuestionInstruction and criteriaEffectAnswer
revision_snapshot_present Does the lesson end with a useful revision snapshot?yes: A concise revision snapshot is present and reflects the lesson. — no: The snapshot is absent, misplaced, or does not reflect the lesson. repairs if < 0.7 0.97
l1-ci-1-1_los_heading_verbatim Is the unit LOS heading verbatim apart from case and trailing punctuation?yes: The heading preserves the LOS wording. — no: The heading changes or omits LOS wording. advisory 0.30
l1-ci-1-1_exam_framing_present Is a specific On the exam framing present?yes: Exam framing names the command, item form, or typical trap. — no: Exam framing is absent or generic. repairs if < 0.7 0.96
l1-ci-1-1_trap_callout_present Does the unit explicitly teach its common error?yes: The trap is identified and corrected. — no: The trap is absent or only named. repairs if < 0.7 0.96
l1-ci-1-1_template_order_followed Does the unit follow the required teaching, example, changed condition, error, practice order?yes: The required order is followed. — no: Required sections are missing or materially reordered. repairs if < 0.7 0.86
l1-ci-1-1_derivations_in_go_deeper Are optional derivations confined to Go deeper?yes: Derivations are either absent or placed in Go deeper. — no: A derivation interrupts the core lesson outside Go deeper. advisory 0.78
l1-ci-1-1_practice_matches_unit Does practice assess this unit and its LOS?yes: Practice is aligned to the unit LOS and requested format. — no: Practice is missing, misaligned, or uses the wrong format. repairs if < 0.7 0.96
los_taught_l1-ci-1-los-1 Is l1-ci-1-los-1 taught rather than merely named?yes: The outcome is taught with explanation and an example the learner could reproduce. — no: The outcome is only named or summarised. repairs if < 0.7 0.96
register_followed_1 Does the lesson follow this register rule: State firm or project perspective and the evaluated stakeholder.yes: The rule is followed. — no: The rule is violated or omitted where applicable. advisory 0.81
register_followed_2 Does the lesson follow this register rule: Separate operating from financing cash flow, pre-tax from post-tax, and nominal from real.yes: The rule is followed. — no: The rule is violated or omitted where applicable. advisory 0.91
register_followed_3 Does the lesson follow this register rule: Identify sunk costs, opportunity costs and terminal reversals.yes: The rule is followed. — no: The rule is violated or omitted where applicable. advisory 0.76

Per practice item

QuestionInstruction and criteriaEffectAnswer
single_defensible_key Does the item have one defensible answer?yes: Exactly one answer is supported by the state and LOS. — no: No answer or multiple answers are defensible. regenerates item if < 0.6 0.96
rationales_explain_each_option Do rationales explain every option?yes: Every option has a misconception- or evidence-based rationale. — no: At least one option lacks an explanatory rationale. recorded 0.83
tests_stated_los Does the item test the supplied LOS?yes: Solving the item demonstrates the supplied LOS. — no: The item tests another skill or mere trivia. recorded 0.93
difficulty Classify the cognitive difficulty.recall / single-step application / multi-step application / synthesis recorded —

Layer 3 · what the reviewer returns

A JSON object with a status (pass, changes_requested, blocked), issues[] each with severity, category, the quoted evidence, the problem and a suggestion, numericalChecks[] each binding a verbatim quote from the lesson to a literal arithmetic expression our code recomputes, and expert_review_gaps[] for claims it cannot verify from the source packet. Open any reviewer step above to read a real one.

The prompts, stage by stage

Seven prompts exist. Each card lists what the prompt is built from, in order, and opens a real instance from this run — the exact text the model received. Builders live in cfa/prompts.v3.ts.

Planner

Turn the official learning goal into a blueprint the writer can follow and the checks can verify.

  1. Identity: level, subject, topic, chapter
  2. The LOS, verbatim
  3. The subject’s teaching configuration: reasoning style, register rules, table shapes, allowed templates, common traps, practice formats
  4. The source packet, marked untrusted
  5. Rules for a valid plan: every LOS exactly once, allowed templates only, fixtures required only for calculation units
  6. The fixture contract: how worked-example numbers are declared
  7. Allowed section orders per template
  8. A JSON skeleton with placeholders

Returns: One CfaPlanV3 JSON object. Validated by code; one bounded repair if invalid.

Writer

Write the learner-facing lesson from the validated plan, using only the pre-computed numbers.

  1. The same identity, LOS, subject configuration and source packet
  2. The validated plan, verbatim JSON
  3. “Write as the teacher; never mention sources or the pipeline”
  4. Chapter mode or topic mode: exact headings allowed, what to put where
  5. The template order
  6. The renderer contract: what the app can display
  7. Math and Mermaid output rules
  8. Fixture binding rules: every number must be a compiled fixture, written exactly as its display string
  9. The list of static rule ids that will be enforced
  10. One shape example per allowed practice format
  11. The calculator fence schema
  12. The list of Jev question names that will be asked
  13. The compiled fixtures, appended by the pipeline

Returns: Markdown only. Then static checks, Jev, and up to two bounded repairs.

Teaching review

Judge the finished lesson as a tutor would: accuracy, depth, assumptions, arithmetic.

  1. The same identity, LOS, subject configuration and source packet
  2. “Review for teaching quality only; structure is checked elsewhere”
  3. Numerical evidence binding: quotes must be verbatim substrings, expressions literal
  4. The numerical-checks block and the output schema
  5. The lesson

Returns: JSON: status, issues, numericalChecks, expert_review_gaps, notes. The arithmetic checker recomputes every numerical check.

Plan repair

Fix a plan the validator rejected.

  1. “Return one complete corrected CfaPlanV3 JSON object only. Preserve valid identity, LOS, facts and fixtures.”
  2. The validator’s error, verbatim
  3. The original plan

Returns: A complete plan. This is the prompt that dropped eight sections and both worked examples over one unrecognised key.

Presentation repair

Fix the static findings without losing teaching.

  1. “Return complete corrected Markdown only. Repair the listed static defects without losing teaching or changing validated fixture values.”
  2. Math and Mermaid rules, and the exact LaTeX command allowlist
  3. The findings as JSON, each with its evidence
  4. The original lesson

Returns: A complete lesson. Re-checked; if major findings remain the lesson is recorded as failed.

Jev structural repair

Address the Jev questions that scored below 0.7.

  1. “Return complete corrected Markdown only. Address the failed Jev questions while preserving valid structure, facts, fixtures and practice.”
  2. The failed question names with their probabilities
  3. The original lesson

Returns: A complete lesson. Re-checked once; a second failure is recorded, not repaired.

Practice item regeneration

Replace one quiz item Jev judged to have no single defensible answer.

  1. “Return one replacement practice item JSON object only. Keep the same format and LOS; make exactly one key defensible and explain every option.”
  2. The Jev feedback
  3. The original item
  4. The LOS text
  5. The Jev item question names

Returns: One item, re-validated against the fence rules.

Did not fire in this run.

Why it is slow, and what can run in parallel

model callbounded repair (also a model call)Jevlocal code
Sole proprietorships, partnerships and corporations255.4 s · model 98% · repairs 0 ms · everything else 3.9 s
planner 49.9 s
writer 159.8 s
reviewer 41.8 s
Key features of corporate issuers280.9 s · model 99% · repairs 47.0 s · everything else 2.2 s
planner 30.1 s
planner 47.0 s
writer 153.6 s
reviewer 47.9 s
Organizational Forms, Corporate Issuer Features, and Ownership514.1 s · model 99% · repairs 195.8 s · everything else 5.7 s
planner 63.5 s
writer 171.4 s
presentation-repair 69.1 s
jev-repair 126.7 s
reviewer 77.7 s

Bars are proportional to each lesson’s own total. Hover a segment for its name and duration.

99%
of the 1050.4 s across these three lessons was spent inside model calls
11.9 s
everything else, combined — static checks, Jev, calculators, media, practice validation — across all three lessons
77.7 s
of reviewer time spent on a lesson that had already failed its static checks

What depends on what

planner  →  writer  →  static checks  →  Jev  →  reviewer
   (plan)     (lesson)    (needs lesson)    (needs lesson)   (needs final lesson)
                 ↑______ repair ______↓      ↑__ repair __↓

Inside one lesson, every model call needs the previous one’s output: the writer needs the plan, the checks need the lesson, the reviewer needs the lesson after its repairs. That chain cannot be parallelised without changing what is being checked. The steps that could overlap — calculators, media, the per-item Jev calls — add up to 11.9 s in total. Parallelising them would save under a second per lesson.

Where the time actually goes, and the levers

Two things this run caught that the 60-lesson profile did not

A bounded repair is asked to return the whole corrected object, so it is free to return less. One rejected key cost a plan eight sections and both worked examples. That is the writer-repair defect measured earlier — 11 of 37 rewrites losing over 20% of their words — showing up on the planner, with both attempts preserved side by side.

A one-key schema error deleted most of a plan. The second chapter's planner emitted an unrecognised mermaid key inside visualPlan[0]. Zod rejected it, the exact error went back as VALIDATOR FEEDBACK (exact) with the original plan below it, and attempt 2 returned valid JSON that had quietly dropped most of the teaching design. The repair instruction asks the model to "preserve valid identity, LOS, facts and fixtures" and says nothing about the plan it spent 2,119 reasoning tokens building.

Plan fieldAttempt 1After repairEffect
sections80the planned teaching order is gone
workedExamples20both planned examples are gone
coverage10the coverage claim is gone
revisionSnapshot54one line dropped
practice.count24raised, not preserved

The lesson that followed is 1,567 words with no Go deeper section; the chapter whose plan was not repaired is 1,807 words with one. Two different chapters, so read that as an observation rather than a measured cost — the plan loss above is verbatim from the trace, the lesson difference is not controlled.

The topic lesson spent both repair budgets and still failed. Twenty steps, 8.6 minutes, 99% of it waiting on four Luna calls: write, presentation repair, Jev structural repair, review. Each repair rewrote the whole lesson and was re-checked. At the end, 15 static findings remained and the run recorded failed.

This is the case for the revision-notes redesign in one run: the topic format is not a writing problem the prompt can fix by asking again. Assembling the module note from chapter artifacts removes the free-form draft that keeps failing here.

The flow after the changes

The critic panel reversed the obvious speed fix. Cutting reasoning effort is not free: at high effort the planner designed 4.5 fixture calculations per plan instead of 7.7 and the reviewer extracted 7 numeric claims instead of 12. The real lever is not paying twice for work we throw away.

The reversal, with the evidence

Signal (same 31 lessons in both runs)Max effortHigh effortReading
Planner fixture calculations per plan (mean / median)7.7 / 64.5 / 4Fewer in 22 of 31 lessons. The planner designs every number and worked example; this is a direct cut to the material candidates rank as indispensable
Plan JSON length (median chars)21,18014,755Smaller in 26 of 31
Planner validation failures on attempt 22/606/32Thinner plans fail their own schema more often
Reviewer numeric claims extracted (median)127The arithmetic checker has less to verify
Reviewer issues raised (median)63The reviewer moves with the knob under test, so it cannot be the quality gate for this experiment

The prompts were byte-identical between the two runs, so the difference is the effort setting alone. Price of staying at max: about $0.056 per lesson and five minutes, roughly $70 and 3.4 hours per corpus pass. The decision is to pay it until a real quality gate exists.

The bug the hand audit missed: repairs delete teaching

Both repair prompts end with "return complete corrected Markdown", and only the original markdown is in context. Of 37 real rewrites, 11 lost more than 20% of their words:

LessonWords before → afterWhat disappeared
9b9f78ea4,275 → 1,81312 headings dropped
c9c1f5624,547 → 2,191two worked examples and three Go deeper sections gone
b23c8ec62,609 → 1,505the rewrite created the final missing-heading and LOS-mismatch findings
abc8f86d2,660 → 1,920published, then hand-audited at 13 of 24

The only detector today is a minor "unit budget short" finding. Worse, the Jev repair output never passes through the normalisers the writer output does, so 12 of 17 Jev-repaired lessons ended with blocking findings after a 113 s call plus a wasted review.

Target chapter lesson · effort stays at max · repairs become patches with a regression gate

Context+ chapter names for links
Plannermax · 221 s · plan-richness check added
Jev plan bundleregister applicability, scenario choice · 0.3 s
Writermax · template prompt ~4.5k in · 220 s
Normalise + staticlossless fixes applied to every writer output, including repairs
Patch repairone unit, not the lesson · ~30–45 s · regression gate
Jev unit bundlerubric Scores, advisory until calibrated
Media + practicefail fast if already doomed
Jev itemsmoved before review
Rubric reviewmax · scores the 12 criteria with quoted spans · sees final practice
Arithmeticrecompute quoted claims
Evidence+ effort, timings, Jev cost
ChangeWhyEffectEffort
Revert planner and reviewer to max effort; keep the plumbing for gated experimentsThe high-effort run thinned plans and reviews; it was never gated and contradicts the contractQuality protected; +5 min per lesson acceptedhours
Fix the effort provenance: record the effort actually used, add start and end times, per-stage effort in the manifest, concurrency recordedEvery call logs "max" today regardless of what ran, so no experiment is auditableTwo runs become distinguishable without token forensicshours
Patch-scoped repairs with a regression gate: return only the affected unit with the plan and fixtures in context; splice by heading anchor; fail the lesson if the result drops below 85% of the words, loses a heading or loses a fixture literal11 of 37 rewrites silently deleted teaching contentLargest quality lever; repairs fall from ~98 s to ~30–45 smedium
Lossless normalisers on every writer output, including the Jev repair, which bypasses them today12 of 17 Jev-repaired lessons ended with blocking findings the normalisers would have fixedFewer repair rounds, fewer terminal failuressmall
Stop discarding paid work: transport retry with backoff separate from content repair, per-stage resume checkpoints, run lock, spend cap, fail fast before the reviewerA transport error was treated as a content defect and threw away the plan and the draft; two overlapping runs plus exhausted credit caused the error cascadeSuccess rate target 85% from 47%; cost per successful lesson $0.17 from $0.30medium
Move item regeneration before the review and re-evaluate the replacementRegenerated items are never reviewed or re-checked todayThe reviewer sees the practice that shipssmall
Measure concurrency at 30, 45 and 60 on an isolated funded key instead of asserting a ceilingThe 30-stream "limit" was inferred from a run that collided with another run and ran out of creditThe only lever that shortens a batch with no quality costsmall
Effort reductions become gated experiments, in order: planner, reviewer (after it scores the rubric), presentation repair, writer lastEach needs blind paired rubric scoring and plan-richness metrics, not a stopwatchIf they pass, a further 5 min per lessonper experiment

Honest target: about 11.5 to 12 minutes per lesson at max effort with the repair fixes, down from 13.3. The metrics that matter for a 1,233-lesson corpus are success rate, cost per successful lesson, and the 60-lesson iteration loop (34 minutes today, 25 to 28 at C=30). An overnight pass fits either way, which is why quality wins the trade.

Prompt anatomy: what each part decides

Token counts rebuilt from the real prompts for the bond-pricing chapter (06531a5d) and the derivatives topic (da91cd56). The writer prompt is 13.3k tokens; a third of it is the plan JSON the writer then transcribes, a quarter is the source packet it is told never to mention.

Writer prompt, chapter 13,292 tokens
plan JSON 4,631
source packet 3,287
fixtures JSON 2,191
renderer contract 1,547
ex.
rules ~650
Writer prompt, topic 11,770 tokens
plan JSON 7,474 (four overview units with trap, visual plan and content[].type scaffold)
packet 836
renderer 1,547
rules, examples, 30 Jev names
Planner prompt, chapter 5,273 tokens · Reviewer prompt 7,641 tokens (lesson 2.8k–5.8k + packet again + numerical-check block + v2 output schema)
packet 3,287
fixture grammar 783 (with a bond example, even for Ethics)
instructions 477
family 250
skeleton + template orders
Prompt sectionWhat it saysWhat it drives in the lessonKind
Official LOS blockLOS id + text, chapter id (never the chapter name)The H2 and unit scope; missing names are why every topic link says "chapter pending"per lesson
Family fragment (from subject-config)reasoning, register lines, table shapes, worked-example shape, traps, formats, visual planRegister, tables and off-LOS material (duration tables in a pricing LOS; hypothesis table in a returns lesson)per family + level
Source packetexcerpts labelled UNTRUSTED, repeated in planner, writer and reviewerGrounding; also the origin of pipeline-language leaks and of the reviewer's source-gap verdictsper lesson
Template orders"knowns → conventions → formula → substitution → result → … → practice"The planner writes sections[] with those titles; the writer emits them as ### headings (19 of 36) and, in topic mode, as bold labels (36 in one lesson)per family
Plan JSON, verbatimunits plus legacy v2 sections/coverage/workedExamples proseThe lesson becomes a transcription of the plan: "teaching, not reference" scores 1 in Quant, FSA and FIper lesson
Chapter unit modeH2 per LOS, On the exam line, exact ### set, Go deeper, snapshot ≤150 wordsThe section skeleton (always present); never says "no other headings"static
Fixture binding + fixtures JSON"do not alter or round values"; compiled fixtures at full float precisionRaw floats copied into prose (2.94174202707276)per lesson
Calculator schema exampletwo steps: CLR TVM, CPT PV; no inputs, no sign ruleFaked one-step calculator tables (27 of 30 blocks)static
Jev question names + static code idsbare identifiers "that will be checked"The writer renders a > Trap: callout to satisfy trap_callout_present on top of the Common error section (30 of 36)per lesson
Practice examplesone "shape only" item per planned format; children rule only in topic mode2-child item sets, Level II constructed responses, symbolic fill-in keysper plan
Reviewer instruction + v2 schema"teaching quality only… source support"; schema still asks for coverage, sourceEvidence, staticFindings8 of 8 audited lessons "changes requested" on source grounds; 0 of 11 learner-facing defects flaggedstatic

Every audit defect, traced to its cause

Defect (prevalence in 36 published)CauseFix location
Template headings: Knowns, Substitution, Reported item (19)Template orders reach planner and writer; plan sections[] titles are transcribed; no "no other headings" rule; normaliser only canonicalises the known setWriter template + strict plan schema + normaliser + unit_heading_not_allowed finding
Bold scaffold labels in topic overviews (36 in one lesson)Same order string injected in topic mode; overview units carry content[].type; Jev template_order_followed then pushes the chapter scaffold into overviews via repairTopic lessons become code-assembled revision notes; per-unit Jev questions only for teaching units
Trap stated twice (30)Plan field + Jev name + required Common error section; nothing says they are the same thingNormaliser converts or drops the callout; one heading set
Extra ### Practice heading (8)Every template order ends "→ practice"; planner writes a Practice sectionNormaliser deletes it; template orders removed
Register applied mechanically (hypothesis table)Every register line concatenated and turned into a register_followed_N question with no LOS gatingappliesTo per register line; Jev plan bundle asks "does this rule apply to this LOS"
Scope creep beyond the LOS (duration tables)Family fragment names spread/duration in register and table shapes; budget check excludes tables; no "only the LOS verb" lineScope guard in the plan; Jev los_verb_alignment Score; tables counted for over-scope
Raw floats in prose (3)Fixtures injected as raw JSON; rounding forbiddenRound to decimalPlaces before injection; raw_float_in_prose finding
Faked calculator tables (27 of 30 blocks)Two-step example; only the last display validated; severity minor so never repaired≥3 steps, input before CPT, sign rule, major severity, one table per block; deterministic TVM steps from fixtures
Level II constructed response; 2-child item sets; symbolic keysFamily formats not level-gated in the plan validator; children rule topic-only; app stricter than APIAPI schema = app schema (landed); canonical chapter practice per level (landed)
"Summary only; chapter pending" on every link (3 of 3)Writer never sees chapter names; v3 has no link resolution against the run's own chaptersResolve against run chapters and LOS key first
Lowercase LOS headings (13)"verbatim" vs "sentence case" conflict; normaliser landed after these lessonsNormaliser (landed), Jev heading question removed
Reviewer flags source gaps, not defects (8 of 8)Instruction asks for source support; schema carries v2 coverage fields; packet fed as evidenceReviewer scores the 12-criterion rubric with quoted spans; provenance only

Prompt rewrite: what each stage gets instead

Planner

  1. Identity, LOS with its command verb, chapter id and name, usable flag.
  2. Family block trimmed to templates, level formats, visual kinds, traps; register lines and table shapes only where their appliesTo matches the LOS verb or template.
  3. Task: one unit with fixtures, calculator ids, practice {formats, count, itemSetChildren} from the canonical per-level rule, examFraming, commonError (renamed from trapCallout), and a scopeGuard list of tempting off-LOS material to exclude or push to Go deeper.
  4. Strict zod that strips sections, coverage, workedExamples: the plan can no longer carry a prose draft.
  5. No template orders. No planner call at all for topic lessons (revision notes are assembled in code).

Writer, chapter (≈4–5k tokens instead of 13k)

  1. Identity, LOS, On-the-exam text.
  2. A literal Markdown template with placeholders for the fixed headings and the rule "no other headings; write the knowns, conventions and substitution flow as prose with at most two tables".
  3. Teaching brief: intuition before the formula, only what the LOS verb asks, worked example distinct from the walk-through, symbols before use, the scope guard.
  4. Applicable register lines only.
  5. Fixtures as a compact rounded table, not raw JSON.
  6. Fence schemas with the real constraints: item-set children 4–6, one ____, a six-step calculator example with inputs and the sign convention.
  7. Notation rules merged into one block. Deleted: plan dump, packet, Jev names, static code ids, template order, trap callout.

Reviewer

  • Scores the audit's 12 criteria 0–2 with a quoted span per score; publishable at ≥16 with no zero on LOS fidelity, worked example, practice validity, clean copy.
  • Keeps the numerical-claims block for the arithmetic checker.
  • Schema loses coverage, sourceEvidence and staticFindings; receives provenance, not excerpts; receives the Jev-flagged sections with the question text.

Item regeneration

  • Includes level, allowed option keys, the whole block (no duplicates), rounded fixture values, the Jev probabilities as feedback, the misconception-per-distractor rule, and the children and blank constraints.
  • Regenerates only the failing option when the distractor bundle says so, not the whole item.

What the critics dropped from the prompt plan

Jev: today and proposed

Jev (TypeSafe System One) returns typed probabilities, not text: $0.042 per million input tokens, output free, ~0.4 s per call, 64k context. Today it costs $0.0006 per lesson and decides almost nothing.

Today

  • 11 presence booleans per chapter (30 per topic) over the whole plan plus lesson; means 0.96–0.97, they never fire.
  • Topic units get chapter questions: "LOS heading verbatim" fails on 16 of 16 topic lessons because overviews have no LOS heading, and the repair pushes the chapter scaffold into them.
  • Item QA asks four things and acts on one; 59 answers below 0.5 on "rationales explain each option" and "tests the stated LOS" were stored and ignored, including the correct verdict on the bond lesson's non-calculating item (0.39).
  • Lessons that scored 12–14 of 24 by hand passed every Jev gate.

Proposed: five bundles, section-scoped state

  • B0 plan bundle (after the planner, ~3k tokens): visual kind Choice per unit, "practice formats can exercise the LOS verb", "overview unit is a teaching unit", register line applicability Nouls.
  • B1 unit bundle (per unit, ~4k tokens): Scores with concrete 0/1/2 levels for LOS verb alignment, exam framing usefulness, teaching-not-reference, example distinctness, changed-condition quality, trap specificity, scaffold leak, visual usefulness, calculator plausibility, compression.
  • B2 item bundle (per item, ~0.7k): key Choice over option keys as an independent answer check, a Noul per wrong option "would a candidate with a specific misconception pick this", "item exercises the LOS verb", self-answering check, Bloom Score reconciled with the declared label.
  • B3 evidence bundle: citation Choice {supports, contradicts, not addressed} per tagged claim against source excerpts; contradictions go to the expert-gap list.
  • B4 snapshot bundle: per-bullet "is this a tested point taught in the unit", coverage Score.

Cost ≈ $0.001 to $0.003 and about a second per lesson.

Three rules the critics imposed

Also removed from the catalogue: anything a deterministic rule already owns (heading leaks, duplicated traps, calculator shape, dangling figures, self-answering grids, word budgets) and the noisiest questions (one asked about derivations produced 59 near-misses). Repairs get a hard budget of one batched, section-scoped call per lesson. Added: conceptual-correctness checks Jev is actually good at, such as an approximation presented as exact, a case-specific conclusion stated as a universal rule, or a clean/dirty and IFRS/US GAAP distinction used without labelling which is which.

Jev inside generation, not only after it

Topic lessons become module revision notes

A topic lesson stops being an overview and becomes the module's last-minute revision sheet: one "must know" block per learning outcome with the rule, the formula, what changes, the trap and the exam angle, plus a formula sheet, the module's classic errors and a ten-minute cross-outcome drill. It is assembled in code from the chapter artifacts, so it can never drift from the chapters.

Why not just write it

Three designs were compared. Writing it fresh repeats the failure we already have: the writer re-teaches, and today's three surviving topic lessons run 2,529 to 3,023 words against a 2,500 cap with teaching units hidden under "Why this module matters". Assembling it from what the chapters already produced gives consistency by construction: every rule, formula, trap and exam line traces to a chapter section. One bounded model call then compresses JSON prose fields only, never Markdown and never formulas, and code renders the document. If that call fails twice, the deterministic render of the raw source ships anyway, so the layer has full availability.

What the critics changed

Shape

<intro ≤ 100 words>                  code prepends the area's exam weight
## Must know
### <outcome text>   [[Chapter link]]   one block per outcome, or "chapter pending"
  - Rule:            bullets kept verbatim when they carry a formula, number or provision
  - Formula:         $…$  [must recall | can derive | interpret only] · Keys: …
  - Changes when:    up to 3 bullets
  - Trap:            ≤ 25 words
  - Exam angle:      ≤ 20 words
  - Choose this when:  (Level II model-selection cue)
  - Answer skeleton:   (discuss / recommend / evaluate outcomes) verb → conclusion → case fact → consequence
## Formula and rule sheet            table, ≤ 12 rows, formulas inserted by code
## Classic errors                    ≤ 8 bullets, clustered across chapters
## Ten-minute drill                  fresh cross-outcome practice

This needs three new fields on the chapter plan so the assembler has something structured to read: the changed conditions as a list, formula cards (LaTeX, meaning, must-recall tag, calculator id), and a Level II "choose this when" cue. A follow-up step assembles a subject-area roll-up from the module sheets, which is the actual final-week artifact candidates describe; module notes are the consolidation layer between readings and after mocks.

Calculator: what was built, and why the obvious fix is the wrong order

The contract scoped the calculator as a machine-checkable keystroke table bound to the worked example's fixture, rendered as a Keys / Display table, because the research ask was "BA II Plus keystrokes beside each applicable example". It was never interactive. The writer authors the keys freehand and only the last display is validated.

The critic panel found something worse than fabricated tables: the one block everyone called genuine is wrong on a factory-default device. Keyed verbatim on a BA II Plus out of the box, the bond example returns −1,158.15, not 947.58. That is a 22% error handed to a candidate as a correct procedure.
Measured on the 30 generated blocksFinding
Step count distribution27 blocks have a single step; one has three, one six, one nine. No block uses the cash-flow, bond or statistics worksheets
Calculator settingsZero blocks set payments per year. The device ships at 12, and clearing the time-value registers does not reset it, so the published sequence silently computes a monthly bond
Sign conventionTwo blocks in the same file print +947.58 and −947.58 for the identical register set. Both pass validation, because only the last non-empty display is compared
Arithmetic-in-words blocksOne prints an expression whose answer requires algebraic entry mode; the device's default chain mode returns a different number entirely
Silent non-compliance15 plans declare a calculator; 11 lessons emit one. The four gaps produce no heading, no block and no finding. Separately, two tables render under a practice heading instead of the calculator heading
Blocks the planner demands but cannot existThe planner throws unless certain templates reference a calculator fixture, but the fixture compiler is a five-operator arithmetic evaluator that knows nothing about registers, so no valid sequence exists for those lessons

The revised plan, in the order the evidence supports

PhaseWhat shipsWhy this orderGate to the next phase
Correctness and visibilityThe fixture declares which worksheet it maps to and names its registers in device units; the block declares its settings and the expected sign; the validator checks magnitude at fixture precision plus the sign independently; invalid blocks and their heading are dropped deterministically; a per-run calculator report makes coverage visibleFixes the wrong-on-device problem and the silent gaps. Crucially it also touches the planner and the fixture contract, which neither of the original options did, so coverage can actually rise instead of collapsing to two blocks≥90% of emitted blocks pass on a 20-lesson re-bench
Device drill on the existing practice pathFill-in items that ask the learner to key the sequence on their own device and enter the display. No new app code, no new format, graded by the tolerance rule that already existsThis is the "repeated practice" the research actually asked for, and it drills the real device rather than a substitute. It ships in days, not weeksTelemetry from ≥200 graded attempts
Interactive register widgetShared calculation engine, a second calculator block version, in-lesson widgetDeliberately last. The neutral-keypad requirement means the learner drills a layout they will never touch on exam day, and tapping a highlighted key removes the tested decision (which quantity is which register, what sign, what payment frequency) while keeping the untested motor stepProduct owner decision on the drill telemetry
Full emulator—Skip permanently. Free play has no oracle, so the regression suite is unbounded, and bond date conventions plus the regression worksheets are an open-ended tail—

One trap avoided: raising the invalid-calculator finding to major would make the repair loop fire, and with a strict validator 28 of the 30 blocks fail. At corpus scale that flips roughly 206 chapters from "ships a bad table" to "the whole lesson fails", unfixably. The finding stays non-blocking; the drop becomes deterministic instead.

Verified on the published lessons

The audit fixes are not just committed, they were replayed over the 36 lessons already on staging to confirm they catch the real defects.

Defect classBeforeAfterHow
Trap stated twice280The normaliser drops the callout when a Common error section exists, and promotes it when that is the only statement of the trap
Template step names used as headings230Same normaliser, with a major finding for anything that survives
Stray practice heading before the widget80Same normaliser
Fabricated calculator blocks27 of 30rejectedAt least three steps, an input before any compute key, no repeated header row, every display traceable to a fixture value or its own keystrokes. The three genuine multi-step blocks still pass
Practice items the app cannot render15 of 36 lessonsrejected at sourceThe API item schema now matches the app schema, and both test suites parse the same example file
Raw floats in prose3fixed upstreamThe writer now receives fixtures at display precision, so it never sees a raw float; a finding is the backstop
Dangling figure references1fixed upstreamA dropped figure takes its introducing sentence with it

Two rules were narrowed after the adversarial reviewer caught them. The self-answering grid test now requires the offending column to actually be the row's answer, because the first version deleted a legitimate equities grid where a column label merely contained the word "rights". And the Level III constructed-response rule records a mismatch rather than throwing, because the first version would have failed 8 of 11 Level III chapter blocks outright.

Roadmap

Ordered so that each stream unblocks the next. Everything before the first horizontal rule is already committed on the API branch.

StreamWhat landsBlocksEffort
0. Landed and verifiedPractice validator matched to the app schema with shared examples both suites parse; copy normalisers; fixture rounding at prompt-injection time; calculator step rules; dangling-figure and visual-misuse handling. 386 tests, typecheck clean—done
1. Remaining from the auditChapter-link resolution against the run's own chapters, and the reviewer repointed at the 12-criterion rubricTopic links that say "chapter pending" when the chapter exists; the reviewer flagging learner-facing defects~1 day
2. Reliability and provenanceRevert to max effort, record the effort actually used, transport retry separate from content repair, per-stage resume checkpoints, run lock, spend cap, fail fast before the reviewEvery measurement and experiment after it~1 day
3. Repair redesignPatch-scoped repairs with the plan and fixtures in context, spliced by heading anchor; regression gate failing any repair that drops 15% of words, a heading or a fixture value; normalisers applied to Jev repairs; item regeneration before the reviewThe largest quality lever; also the speed one2–3 days
4. Prompt rewritePlanner loses the template orders and the legacy prose fields and gains a scope guard; writer drops to a literal template at about 4.5k tokens; reviewer scores the 12-criterion rubric with quoted spans; item regeneration gets level, keys, fixtures and the real feedbackDefect classes at source rather than by normaliser2 days
5. Jev bundlesPlan bundle (register applicability, scenario choice), unit bundle (rubric Scores), item bundle (key check, distractor quality) — all advisory first, with a labelled set including synthetic degradations, then promotion to gates at precision ≥ 0.8 and recall ≥ 0.6Repair targeting and reviewer focus5–7 days plus labelling
6. Module revision notesChapter plan gains changed conditions, formula cards and the Level II cue; assembler, polish and drill stages; two-phase runner; new findings; app "Revise" segment with collapsible blocksReplaces the topic overview entirely7–9 days
7. CalculatorCorrectness first (worksheet-aware fixtures, settings and sign in the block, deterministic drop, coverage report), then the device drill on the existing practice path, then the widget only if the drill telemetry justifies itCoverage rises instead of collapsing~3 days, then ~2, then gated
8. Re-benchmark and re-auditFresh 60-lesson run at head, concurrency measured at 30/45/60, hand audit of a new sample of eight against the rubricThe go/no-go for the 1,233-lesson corpus2 days

Decisions the product owner owns

  1. Overnight at full quality, or faster now? An all-max corpus pass is about 8 hours at 30 streams and roughly $175. Cutting effort saves about 3.4 hours per pass but measurably thins plans and reviews. Recommendation: pay for quality, earn the cuts through gated experiments.
  2. Who scores the blind rubric samples until the reviewer is repointed at the rubric? About eight lessons and two hours per experiment.
  3. The nine staging lessons generated at high effort: regenerate (recommended) or keep with corrected metadata.
  4. Is "Revise" the default tab for a CFA topic, or does Deep Dive stay default with a "Revise this module" button?
  5. Interactive calculator: the recommendation is now correctness, then a drill on the learner's own device, and the widget only if the drill data justifies it. Confirm that ordering, or ask for the widget sooner.
  6. A separate model key for benchmark runs, isolated from production, before any 30-stream run.