CFA v3 pipeline: where the 13 minutes go, what the prompts do, and what changes
Measured on the two 60-lesson benchmark runs (18 to 19 Sep 2026), read from the code, and stress-tested by critic panels. Numbers are medians unless stated.
Sources: artifacts/cfa-benchmark/cfa-v3-bench60c (all stages at max reasoning) and cfa-v3-bench60d (planner and reviewer at high) · step-by-step inputs and outputs from cfa-v3-trace, the first run on Bedrock (21 Sep 2026) · profiling script /tmp/cfa-profile/profile.py · prompts in cfa/prompts.v3.ts · companion deck: CFA lessons on Oncourse
Why a lesson takes 13 minutes
Provider note. The 60-lesson profile above was measured on the direct OpenAI Responses API. Every stage now runs the same gpt-5.6-luna on AWS Bedrock (bedrock-mantle.responses), and the pipeline fails closed if a stage reports any other model or provider. Throughput is unchanged: across the model calls in the traced run below, output ran at 149 to 204 tokens per second against the 142 measured before, so the timings here still hold.
Three findings that were not in the plan
- The effort switch was on but the logs say max. Since commit
1531ee08the planner and reviewer run at high, but the stage log records the constant provider options (pipeline.ts:626), so every call reports max. bench60d proves the switch works: reasoning tokens 25.3k → 5.1k on the planner, 31.9k → 8.6k on the reviewer, text output unchanged. - Repairs run at max. Every writer call, including a bounded repair that only has to fix a listed defect, is hard-wired to max (
pipeline.ts:1550): 14k output tokens to emit 5k text tokens, 98 s per repair. - The reviewer runs on lessons that already failed. Media errors are known before the reviewer (
pipeline.ts:1841) but pushed after it (1913): 26 of 60 lessons spent 277 s of reviewer time on a lesson that could not succeed, about two hours of model time per run.
The flow, stage by stage (today)
Every lesson is an independent chain; the runner already parallelises lessons (30 concurrent Luna streams measured safe, 60 produced errors). Inside the chain nothing meaningful can run in parallel: each model stage needs the previous one's output.
Chapter lesson, bench60c medians · widths are proportional to time · dashed = bounded repair, fires on the share of lessons shown
| # | Stage | Who | Input | Output | Gate after it | Median s | Share | At high |
|---|---|---|---|---|---|---|---|---|
| 1 | Planner | Luna, max → high | LOS + family + sources, 3.7k tok | plan JSON 5.6k text + 25k reasoning | plan schema, identity, fixture compile; 1 JSON repair | 221 | 28% | 69 |
| 2 | Writer | Luna, max | prompt + plan + fixtures, 13.6k tok | markdown 4.5k text + 27k reasoning | static pass (template, budgets, lint, Mermaid, fences) | 220 | 28% | 245 (unchanged) |
| 2a | Presentation repair | Luna, max | 6.3k tok | ~14k output | static pass again | 98 (35/60) | 7.5% | 77 |
| 3 | Jev structural | Jev | plan + lesson + LOS, ~8k tok | 11 booleans | <0.7 on the repairable set → 1 repair | 0.5 | 0.1% | – |
| 3a | Jev repair | Luna, max | lesson + failed questions | whole lesson rewrite | static + Jev again | 113 (17/60) | 4.4% | – |
| 4 | Media, calculators, practice | code | markdown | SVG figures, tables, validated fences | errors recorded (not gating until after review) | ≈0 | 0% | – |
| 5 | Teaching review | Luna, max → high | lesson + context, 7.7k tok | issues, numeric claims, expert gaps; 5.2k text + 32k reasoning | schema; 1 JSON repair; arithmetic checker | 239 | 28% | 84 |
| 6 | Jev item QA | Jev | item + LOS + fixtures, ~1.5k tok | 3 booleans + difficulty | single_defensible_key <0.6 → regenerate item | 0.6 | 0.1% | – |
| 6a | Item regeneration | Luna, max | item + LOS | one item | practice re-validate | 51 (7/60) | 0.9% | – |
Output caps (65,536 per stage) were never hit: reviewer p90 48k, planner 40k, writer 37k. Lowering caps cannot speed anything up and, because Luna bills reasoning against the cap, would only cause truncations.
Real generations, step by step
Every number below is read from artifacts/cfa-benchmark/cfa-v3-trace, produced by pnpm cfa:benchmark --trace on 2026-09-21. The run records what each step received and what it returned, so this is the pipeline as it actually ran, not a reconstruction. Text stages are gpt-5.6-luna on bedrock-mantle.responses at reasoning effort max.
Click any row or button to read the complete text — the whole prompt, the whole completion, the finished lesson, rendered. Nothing here is truncated; all 45 steps are embedded in this page.
Chapter · Sole proprietorships, partnerships and corporations
CFA Level I · Corporate Finance (Level I) · Organizational Forms, Corporate Issuer Features, and Ownership · c34323f7-1ad3-4bcf-bc06-c010a06f4e28 · 12 steps · 255.4 s total, 98% of it waiting on the model
| # | Step | Runs on | Input | Output | Time | |
|---|---|---|---|---|---|---|
| 1 | planner | Bedrock · gpt-5.6-luna · max | 11,286 chars · 2,653 tok | 16,047 chars · 7,456 tok · 4,347 reasoning | 49.9 s | |
| 2 | fixtures | local code | 1,407 chars | 1,768 chars | 0 ms | |
| 3 | writer | Bedrock · gpt-5.6-luna · max | 35,431 chars · 7,661 tok | 13,199 chars · 26,917 tok · 24,334 reasoning | 159.8 s | |
| 4 | static-checks | local code | 13,538 chars | 547 chars | 830 ms | |
| 5 | jev-structural | Jev | 34,095 chars | 8,094 chars | 1.3 s | |
| 6 | calculators | local code | 15,069 chars | 13,559 chars | 0 ms | |
| 7 | media | local code | 13,765 chars | 36,218 chars | 3 ms | |
| 8 | practice-validation | local code | 3,924 chars | 4,825 chars | 2 ms | |
| 9 | reviewer | Bedrock · gpt-5.6-luna · max | 23,730 chars · 5,054 tok | 4,173 chars · 6,682 tok · 5,696 reasoning | 41.8 s | |
| 10 | jev-item-qa | Jev | 3,582 chars | 6,212 chars | 721 ms | |
| 11 | jev-item-qa | Jev | 3,629 chars | 6,226 chars | 499 ms | |
| 12 | jev-item-qa | Jev | 4,298 chars | 6,226 chars | 606 ms |
Chapter · Key features of corporate issuers
CFA Level I · Corporate Finance (Level I) · Organizational Forms, Corporate Issuer Features, and Ownership · 3ef1a48c-0c08-4e80-876c-c66d3d861413 · 13 steps · 280.9 s total, 99% of it waiting on the model
| # | Step | Runs on | Input | Output | Time | |
|---|---|---|---|---|---|---|
| 1 | planner | Bedrock · gpt-5.6-luna · max | 11,265 chars · 2,652 tok | 12,505 chars · 4,522 tok · 2,119 reasoning | 30.1 s | |
| 2 | planner attempt 2 | Bedrock · gpt-5.6-luna · max | 11,666 chars · 2,765 tok | 1,841 chars · 7,202 tok · 6,732 reasoning | 47.0 s | |
| 3 | fixtures | local code | 2 chars | 2 chars | 0 ms | |
| 4 | writer | Bedrock · gpt-5.6-luna · max | 20,424 chars · 4,848 tok | 12,112 chars · 25,116 tok · 22,603 reasoning | 153.6 s | |
| 5 | static-checks | local code | 12,593 chars | 5,421 chars | 23 ms | |
| 6 | jev-structural | Jev | 20,321 chars | 8,097 chars | 649 ms | |
| 7 | calculators | local code | 12,611 chars | 12,614 chars | 0 ms | |
| 8 | media | local code | 12,820 chars | 12,732 chars | 1 ms | |
| 9 | practice-validation | local code | 4,727 chars | 4,170 chars | 1 ms | |
| 10 | reviewer | Bedrock · gpt-5.6-luna · max | 21,416 chars · 4,639 tok | 5,256 chars · 7,398 tok · 6,337 reasoning | 47.9 s | |
| 11 | jev-item-qa | Jev | 2,165 chars | 6,221 chars | 581 ms | |
| 12 | jev-item-qa | Jev | 2,624 chars | 6,213 chars | 516 ms | |
| 13 | jev-item-qa | Jev | 2,072 chars | 6,210 chars | 478 ms |
Topic · Organizational Forms, Corporate Issuer Features, and Ownership
CFA Level I · Corporate Finance (Level I) · Organizational Forms, Corporate Issuer Features, and Ownership · c674c4ec-a820-42c9-a7f1-ecbfc9ef8c22 · 20 steps · 514.1 s total, 99% of it waiting on the model
| # | Step | Runs on | Input | Output | Time | |
|---|---|---|---|---|---|---|
| 1 | planner | Bedrock · gpt-5.6-luna · max | 14,614 chars · 3,461 tok | 26,872 chars · 10,153 tok · 4,980 reasoning | 63.5 s | |
| 2 | fixtures | local code | 1,251 chars | 1,659 chars | 0 ms | |
| 3 | writer | Bedrock · gpt-5.6-luna · max | 46,740 chars · 9,873 tok | 36,215 chars · 29,071 tok · 21,707 reasoning | 171.4 s | |
| 4 | static-checks | local code | 35,456 chars | 15,906 chars | 46 ms | |
| 5 | writer-presentation-repair attempt 2 | Bedrock · gpt-5.6-luna · max | 43,565 chars · 9,329 tok | 33,548 chars · 14,148 tok · 7,342 reasoning | 69.1 s | |
| 6 | static-checks | local code | 35,262 chars | 21,891 chars | 25 ms | |
| 7 | jev-structural | Jev | 74,572 chars | 14,730 chars | 1.3 s | |
| 8 | writer-jev-repair attempt 3 | Bedrock · gpt-5.6-luna · max | 34,016 chars · 6,930 tok | 35,433 chars · 23,035 tok · 15,883 reasoning | 126.7 s | |
| 9 | static-checks | local code | 37,197 chars | 25,386 chars | 23 ms | |
| 10 | jev-structural | Jev | 76,557 chars | 14,721 chars | 1.1 s | |
| 11 | calculators | local code | 38,566 chars | 37,218 chars | 1 ms | |
| 12 | media | local code | 37,424 chars | 36,981 chars | 2 ms | |
| 13 | practice-validation | local code | 21,238 chars | 8,855 chars | 1 ms | |
| 14 | reviewer | Bedrock · gpt-5.6-luna · max | 33,753 chars · 7,033 tok | 10,804 chars · 12,549 tok · 10,193 reasoning | 77.7 s | |
| 15 | jev-item-qa | Jev | 3,643 chars | 6,224 chars | 699 ms | |
| 16 | jev-item-qa | Jev | 3,574 chars | 6,223 chars | 515 ms | |
| 17 | jev-item-qa | Jev | 3,558 chars | 6,226 chars | 524 ms | |
| 18 | jev-item-qa | Jev | 3,743 chars | 6,226 chars | 377 ms | |
| 19 | jev-item-qa | Jev | 3,702 chars | 6,224 chars | 569 ms | |
| 20 | jev-item-qa | Jev | 3,769 chars | 6,223 chars | 507 ms |
How the checks work
Three layers, cheapest first. Only the first two can send work back for repair; the third reports and never rewrites.
What each layer found in this run
| Lesson | Static, after repairs | Jev below 0.7 | Tutor verdict | Repairs fired |
|---|---|---|---|---|
| Chapter · Sole proprietorships, partnerships and corporations | 0 major / 1 minor | 1 of 11 | changes_requested · 2 issues | none |
| Chapter · Key features of corporate issuers | 0 major / 2 minor | 0 of 11 | changes_requested · 3 issues | plan |
| Topic · Organizational Forms, Corporate Issuer Features, and Ownership | 13 major / 2 minor | 12 of 31 | changes_requested · 7 issues | presentation, jev |
Layer 1 · every static rule read from presentation.ts and markdownLint.ts
| Code | Severity | What it catches | Effect |
|---|---|---|---|
ascii_diagram |
major | ASCII text diagrams are not supported on the CFA learner renderer | repair |
ascii_multiplication |
major | An asterisk is used as a learner-facing multiplication sign outside math mode | repair |
dangling_figure_reference |
major | A sentence introduces a figure that is not there; drop the sentence or supply the figure | repair |
display_math_disabled |
major | Display math is not enabled by the supplied renderer profile | repair |
inline_math_disabled |
major | Inline math is not enabled by the supplied renderer profile | repair |
los_heading_mismatch |
major | Chapter H2 headings must match each scoped LOS verbatim apart from case and trailing punctuation | repair |
markdown_lint_malformed_table |
major | Markdown table rows have inconsistent column counts | repair |
markdown_lint_raw_html |
major | Raw HTML is not allowed in CFA lesson Markdown | repair |
markdown_lint_table_too_wide |
major | Markdown tables may contain at most three columns | repair |
markdown_lint_unbalanced_math |
major | Math delimiters are unbalanced | repair |
math_in_inline_code |
major | Mathematical notation is hidden in inline code instead of supported math delimiters | repair |
mermaid_capability_disabled |
major | Mermaid is not enabled by the supplied renderer profile | repair |
pipeline_language_leak |
major | Learner-facing prose must not expose source-packet or pipeline limitations | repair |
practice_block_missing |
major | The v3 lesson has no cfa-practice fence | repair |
raw_math_operator |
major | A caret or underscore expression is outside a supported math delimiter | repair |
template_labels_in_prose |
major | A bold template label names a pipeline step to the learner; write the sentence without it | repair |
unbalanced_display_math |
major | A $$ display-math delimiter is unclosed | repair |
unbalanced_inline_math |
major | A single-dollar math delimiter is unclosed; escape currency dollars or use a complete $...$ span | repair |
unbalanced_math_delimiter |
major | A math delimiter is unclosed or has no matching opener | repair |
unsupported_math_delimiter |
major | CFA v3 math must use only $...$ or $$...$$ delimiters | repair |
unsupported_math_tilde |
major | A tilde is not allowed inside CFA math; use an explicit supported command or prose | repair |
visual_misuse |
major | A payoff or curve is drawn as a flowchart; use a cfa-svg figure or a table for numeric nodes | repair |
wide_markdown_table |
major | A phone lesson table has more than three columns; split the comparison and move reasoning into nearby prose | repair |
empty_markdown |
critical | The lesson has no learner-facing Markdown | recorded |
display_math_setext_collision |
major | A display-math line contains only equals signs or hyphens and can be parsed as a Setext heading | recorded |
markdown_lint_unresolved_chapter_marker |
major | A chapter link marker remains unresolved | recorded |
calculator_heading_orphaned |
minor | A Calculator heading has no calculator block; remove the heading with the block | recorded |
dense_markdown_table |
minor | A table contains paragraph-length cells; keep comparison labels concise and explain the reasoning below it | recorded |
mermaid_too_large |
minor | Mermaid diagrams may use at most 8 nodes and labels of at most 4 words | recorded |
practice_format_mismatch |
minor | Practice formats differ from the plan after dropping an invalid item, or a Level III chapter block lacks a cfa_constructed_response with a number or choice field | recorded |
prose_before_first_heading |
minor | Chapter prose appears before the first H2; move it under the heading after the On the exam line | recorded |
raw_float_in_prose |
minor | A raw floating-point expansion reached the learner; show the fixture value at its displayed precision | recorded |
trap_duplicated |
minor | The unit states its trap twice; keep the Common error section and drop the callout | recorded |
Fence validators read from practice.ts and calculator.ts
A quiz block or a calculator block that breaks one of these is dropped or the lesson fails; the writer never gets to argue.
| Fence | Rule |
|---|---|
cfa-practice | practice is not planned for unit … |
cfa-practice | practice unitId … does not match plan unit … |
cfa-practice | … is not allowed at Level … |
cfa-practice | numeric key does not match compiled fixture … |
cfa-practice | practice block has no valid items (${dropped.map(entry => |
cfa-practice | practice formats … do not match planned formats … |
cfa-practice | practice item-set child count does not match planned count … |
cfa-practice | Invalid cfa-practice JSON: … |
cfa-practice | practice block contains LOS outside its unit plan |
cfa-practice | item LOS IDs must be a subset of the block LOS IDs |
cfa-practice | option D is allowed only at Level III |
cfa-practice | Invalid cfa-practice fence; use exactly three backticks and no trailing info tokens |
cfa-calculator | calculator references unknown fixture … |
cfa-calculator | calculator references unknown calculation …/… |
cfa-calculator | calculator needs at least … steps; found … |
cfa-calculator | calculator step … repeats the table header; a block serialises to exactly one header row |
cfa-calculator | calculator step … computes before any input is keyed in |
cfa-calculator | calculator display … is not a number |
cfa-calculator | calculator display … is neither a fixture value nor derivable from the keys … |
cfa-calculator | calculator final display … does not match compiled value …; fixture conventions: … |
cfa-calculator | Invalid cfa-calculator JSON: … |
cfa-calculator | calculator final display is missing |
cfa-calculator | Invalid cfa-calculator fence; use exactly three backticks and no trailing info tokens |
Layer 2 · the questions Jev was asked verbatim from the first structural call, Chapter · Sole proprietorships, partnerships and corporations, with its answers
| Question | Instruction and criteria | Effect | Answer |
|---|---|---|---|
revision_snapshot_present |
Does the lesson end with a useful revision snapshot?yes: A concise revision snapshot is present and reflects the lesson. — no: The snapshot is absent, misplaced, or does not reflect the lesson. | repairs if < 0.7 | 0.97 |
l1-ci-1-1_los_heading_verbatim |
Is the unit LOS heading verbatim apart from case and trailing punctuation?yes: The heading preserves the LOS wording. — no: The heading changes or omits LOS wording. | advisory | 0.30 |
l1-ci-1-1_exam_framing_present |
Is a specific On the exam framing present?yes: Exam framing names the command, item form, or typical trap. — no: Exam framing is absent or generic. | repairs if < 0.7 | 0.96 |
l1-ci-1-1_trap_callout_present |
Does the unit explicitly teach its common error?yes: The trap is identified and corrected. — no: The trap is absent or only named. | repairs if < 0.7 | 0.96 |
l1-ci-1-1_template_order_followed |
Does the unit follow the required teaching, example, changed condition, error, practice order?yes: The required order is followed. — no: Required sections are missing or materially reordered. | repairs if < 0.7 | 0.86 |
l1-ci-1-1_derivations_in_go_deeper |
Are optional derivations confined to Go deeper?yes: Derivations are either absent or placed in Go deeper. — no: A derivation interrupts the core lesson outside Go deeper. | advisory | 0.78 |
l1-ci-1-1_practice_matches_unit |
Does practice assess this unit and its LOS?yes: Practice is aligned to the unit LOS and requested format. — no: Practice is missing, misaligned, or uses the wrong format. | repairs if < 0.7 | 0.96 |
los_taught_l1-ci-1-los-1 |
Is l1-ci-1-los-1 taught rather than merely named?yes: The outcome is taught with explanation and an example the learner could reproduce. — no: The outcome is only named or summarised. | repairs if < 0.7 | 0.96 |
register_followed_1 |
Does the lesson follow this register rule: State firm or project perspective and the evaluated stakeholder.yes: The rule is followed. — no: The rule is violated or omitted where applicable. | advisory | 0.81 |
register_followed_2 |
Does the lesson follow this register rule: Separate operating from financing cash flow, pre-tax from post-tax, and nominal from real.yes: The rule is followed. — no: The rule is violated or omitted where applicable. | advisory | 0.91 |
register_followed_3 |
Does the lesson follow this register rule: Identify sunk costs, opportunity costs and terminal reversals.yes: The rule is followed. — no: The rule is violated or omitted where applicable. | advisory | 0.76 |
Per practice item
| Question | Instruction and criteria | Effect | Answer |
|---|---|---|---|
single_defensible_key |
Does the item have one defensible answer?yes: Exactly one answer is supported by the state and LOS. — no: No answer or multiple answers are defensible. | regenerates item if < 0.6 | 0.96 |
rationales_explain_each_option |
Do rationales explain every option?yes: Every option has a misconception- or evidence-based rationale. — no: At least one option lacks an explanatory rationale. | recorded | 0.83 |
tests_stated_los |
Does the item test the supplied LOS?yes: Solving the item demonstrates the supplied LOS. — no: The item tests another skill or mere trivia. | recorded | 0.93 |
difficulty |
Classify the cognitive difficulty.recall / single-step application / multi-step application / synthesis | recorded | — |
Layer 3 · what the reviewer returns
A JSON object with a status (pass, changes_requested, blocked), issues[] each with severity, category, the quoted evidence, the problem and a suggestion, numericalChecks[] each binding a verbatim quote from the lesson to a literal arithmetic expression our code recomputes, and expert_review_gaps[] for claims it cannot verify from the source packet. Open any reviewer step above to read a real one.
The prompts, stage by stage
Seven prompts exist. Each card lists what the prompt is built from, in order, and opens a real instance from this run — the exact text the model received. Builders live in cfa/prompts.v3.ts.
Planner
Turn the official learning goal into a blueprint the writer can follow and the checks can verify.
- Identity: level, subject, topic, chapter
- The LOS, verbatim
- The subject’s teaching configuration: reasoning style, register rules, table shapes, allowed templates, common traps, practice formats
- The source packet, marked untrusted
- Rules for a valid plan: every LOS exactly once, allowed templates only, fixtures required only for calculation units
- The fixture contract: how worked-example numbers are declared
- Allowed section orders per template
- A JSON skeleton with placeholders
Returns: One CfaPlanV3 JSON object. Validated by code; one bounded repair if invalid.
Writer
Write the learner-facing lesson from the validated plan, using only the pre-computed numbers.
- The same identity, LOS, subject configuration and source packet
- The validated plan, verbatim JSON
- “Write as the teacher; never mention sources or the pipeline”
- Chapter mode or topic mode: exact headings allowed, what to put where
- The template order
- The renderer contract: what the app can display
- Math and Mermaid output rules
- Fixture binding rules: every number must be a compiled fixture, written exactly as its display string
- The list of static rule ids that will be enforced
- One shape example per allowed practice format
- The calculator fence schema
- The list of Jev question names that will be asked
- The compiled fixtures, appended by the pipeline
Returns: Markdown only. Then static checks, Jev, and up to two bounded repairs.
Teaching review
Judge the finished lesson as a tutor would: accuracy, depth, assumptions, arithmetic.
- The same identity, LOS, subject configuration and source packet
- “Review for teaching quality only; structure is checked elsewhere”
- Numerical evidence binding: quotes must be verbatim substrings, expressions literal
- The numerical-checks block and the output schema
- The lesson
Returns: JSON: status, issues, numericalChecks, expert_review_gaps, notes. The arithmetic checker recomputes every numerical check.
Plan repair
Fix a plan the validator rejected.
- “Return one complete corrected CfaPlanV3 JSON object only. Preserve valid identity, LOS, facts and fixtures.”
- The validator’s error, verbatim
- The original plan
Returns: A complete plan. This is the prompt that dropped eight sections and both worked examples over one unrecognised key.
Presentation repair
Fix the static findings without losing teaching.
- “Return complete corrected Markdown only. Repair the listed static defects without losing teaching or changing validated fixture values.”
- Math and Mermaid rules, and the exact LaTeX command allowlist
- The findings as JSON, each with its evidence
- The original lesson
Returns: A complete lesson. Re-checked; if major findings remain the lesson is recorded as failed.
Jev structural repair
Address the Jev questions that scored below 0.7.
- “Return complete corrected Markdown only. Address the failed Jev questions while preserving valid structure, facts, fixtures and practice.”
- The failed question names with their probabilities
- The original lesson
Returns: A complete lesson. Re-checked once; a second failure is recorded, not repaired.
Practice item regeneration
Replace one quiz item Jev judged to have no single defensible answer.
- “Return one replacement practice item JSON object only. Keep the same format and LOS; make exactly one key defensible and explain every option.”
- The Jev feedback
- The original item
- The LOS text
- The Jev item question names
Returns: One item, re-validated against the fence rules.
Did not fire in this run.Why it is slow, and what can run in parallel
Bars are proportional to each lesson’s own total. Hover a segment for its name and duration.
What depends on what
planner → writer → static checks → Jev → reviewer
(plan) (lesson) (needs lesson) (needs lesson) (needs final lesson)
↑______ repair ______↓ ↑__ repair __↓
Inside one lesson, every model call needs the previous one’s output: the writer needs the plan, the checks need the lesson, the reviewer needs the lesson after its repairs. That chain cannot be parallelised without changing what is being checked. The steps that could overlap — calculators, media, the per-item Jev calls — add up to 11.9 s in total. Parallelising them would save under a second per lesson.
Where the time actually goes, and the levers
- Across lessons is already parallel. The runner generates up to 30 lessons at once. Wall-clock for a corpus is the slowest lesson, not the sum.
- Repairs are whole rewrites at max effort. The topic lesson spent 508.4 s in model calls, of which two repairs were 195.8 s — and it still failed. Patch-scoped repairs return only the broken piece: less to generate, less to lose.
- The reviewer runs on lessons that already failed. 77.7 s here. Gate it on the static result and that time disappears.
- Reasoning effort is the multiplier. Output runs at a flat ~150–200 tokens per second, and 80–90% of output tokens are reasoning. Repairs at
highinstead ofmaxroughly halve their time; the earlier 60-lesson runs showed the writer must stay atmax, but a repair only has to fix a listed defect. - Prompt caching helps cost, not speed. Bedrock served 2,651 of 2,653 planner input tokens from cache on this run; generation time is unchanged because it is output-bound.
Two things this run caught that the 60-lesson profile did not
A one-key schema error deleted most of a plan. The second chapter's planner emitted an unrecognised mermaid key inside visualPlan[0]. Zod rejected it, the exact error went back as VALIDATOR FEEDBACK (exact) with the original plan below it, and attempt 2 returned valid JSON that had quietly dropped most of the teaching design. The repair instruction asks the model to "preserve valid identity, LOS, facts and fixtures" and says nothing about the plan it spent 2,119 reasoning tokens building.
| Plan field | Attempt 1 | After repair | Effect |
|---|---|---|---|
sections | 8 | 0 | the planned teaching order is gone |
workedExamples | 2 | 0 | both planned examples are gone |
coverage | 1 | 0 | the coverage claim is gone |
revisionSnapshot | 5 | 4 | one line dropped |
practice.count | 2 | 4 | raised, not preserved |
The lesson that followed is 1,567 words with no Go deeper section; the chapter whose plan was not repaired is 1,807 words with one. Two different chapters, so read that as an observation rather than a measured cost — the plan loss above is verbatim from the trace, the lesson difference is not controlled.
The topic lesson spent both repair budgets and still failed. Twenty steps, 8.6 minutes, 99% of it waiting on four Luna calls: write, presentation repair, Jev structural repair, review. Each repair rewrote the whole lesson and was re-checked. At the end, 15 static findings remained and the run recorded failed.
- 13 of the 15 are the same finding:
unit_heading_not_allowed— "a topic overview uses only its five H2 sections; write the step as prose instead of a###heading". The writer keeps reaching for the chapter template inside a format that forbids it, and two full rewrites did not stop it. - Jev agreed, on every unit: of 31 structural questions, 12 came back below 0.7, and they are the same three questions repeated across all four overview units —
los_heading_verbatim(0.38–0.42),template_order_followed(0.50–0.54) andderivations_in_go_deeper(0.49–0.53).
This is the case for the revision-notes redesign in one run: the topic format is not a writing problem the prompt can fix by asking again. Assembling the module note from chapter artifacts removes the free-form draft that keeps failing here.
The flow after the changes
The reversal, with the evidence
| Signal (same 31 lessons in both runs) | Max effort | High effort | Reading |
|---|---|---|---|
| Planner fixture calculations per plan (mean / median) | 7.7 / 6 | 4.5 / 4 | Fewer in 22 of 31 lessons. The planner designs every number and worked example; this is a direct cut to the material candidates rank as indispensable |
| Plan JSON length (median chars) | 21,180 | 14,755 | Smaller in 26 of 31 |
| Planner validation failures on attempt 2 | 2/60 | 6/32 | Thinner plans fail their own schema more often |
| Reviewer numeric claims extracted (median) | 12 | 7 | The arithmetic checker has less to verify |
| Reviewer issues raised (median) | 6 | 3 | The reviewer moves with the knob under test, so it cannot be the quality gate for this experiment |
The prompts were byte-identical between the two runs, so the difference is the effort setting alone. Price of staying at max: about $0.056 per lesson and five minutes, roughly $70 and 3.4 hours per corpus pass. The decision is to pay it until a real quality gate exists.
The bug the hand audit missed: repairs delete teaching
Both repair prompts end with "return complete corrected Markdown", and only the original markdown is in context. Of 37 real rewrites, 11 lost more than 20% of their words:
| Lesson | Words before → after | What disappeared |
|---|---|---|
9b9f78ea | 4,275 → 1,813 | 12 headings dropped |
c9c1f562 | 4,547 → 2,191 | two worked examples and three Go deeper sections gone |
b23c8ec6 | 2,609 → 1,505 | the rewrite created the final missing-heading and LOS-mismatch findings |
abc8f86d | 2,660 → 1,920 | published, then hand-audited at 13 of 24 |
The only detector today is a minor "unit budget short" finding. Worse, the Jev repair output never passes through the normalisers the writer output does, so 12 of 17 Jev-repaired lessons ended with blocking findings after a 113 s call plus a wasted review.
Target chapter lesson · effort stays at max · repairs become patches with a regression gate
| Change | Why | Effect | Effort |
|---|---|---|---|
| Revert planner and reviewer to max effort; keep the plumbing for gated experiments | The high-effort run thinned plans and reviews; it was never gated and contradicts the contract | Quality protected; +5 min per lesson accepted | hours |
| Fix the effort provenance: record the effort actually used, add start and end times, per-stage effort in the manifest, concurrency recorded | Every call logs "max" today regardless of what ran, so no experiment is auditable | Two runs become distinguishable without token forensics | hours |
| Patch-scoped repairs with a regression gate: return only the affected unit with the plan and fixtures in context; splice by heading anchor; fail the lesson if the result drops below 85% of the words, loses a heading or loses a fixture literal | 11 of 37 rewrites silently deleted teaching content | Largest quality lever; repairs fall from ~98 s to ~30–45 s | medium |
| Lossless normalisers on every writer output, including the Jev repair, which bypasses them today | 12 of 17 Jev-repaired lessons ended with blocking findings the normalisers would have fixed | Fewer repair rounds, fewer terminal failures | small |
| Stop discarding paid work: transport retry with backoff separate from content repair, per-stage resume checkpoints, run lock, spend cap, fail fast before the reviewer | A transport error was treated as a content defect and threw away the plan and the draft; two overlapping runs plus exhausted credit caused the error cascade | Success rate target 85% from 47%; cost per successful lesson $0.17 from $0.30 | medium |
| Move item regeneration before the review and re-evaluate the replacement | Regenerated items are never reviewed or re-checked today | The reviewer sees the practice that ships | small |
| Measure concurrency at 30, 45 and 60 on an isolated funded key instead of asserting a ceiling | The 30-stream "limit" was inferred from a run that collided with another run and ran out of credit | The only lever that shortens a batch with no quality cost | small |
| Effort reductions become gated experiments, in order: planner, reviewer (after it scores the rubric), presentation repair, writer last | Each needs blind paired rubric scoring and plan-richness metrics, not a stopwatch | If they pass, a further 5 min per lesson | per experiment |
Honest target: about 11.5 to 12 minutes per lesson at max effort with the repair fixes, down from 13.3. The metrics that matter for a 1,233-lesson corpus are success rate, cost per successful lesson, and the 60-lesson iteration loop (34 minutes today, 25 to 28 at C=30). An overnight pass fits either way, which is why quality wins the trade.
Prompt anatomy: what each part decides
Token counts rebuilt from the real prompts for the bond-pricing chapter (06531a5d) and the derivatives topic (da91cd56). The writer prompt is 13.3k tokens; a third of it is the plan JSON the writer then transcribes, a quarter is the source packet it is told never to mention.
| Prompt section | What it says | What it drives in the lesson | Kind |
|---|---|---|---|
| Official LOS block | LOS id + text, chapter id (never the chapter name) | The H2 and unit scope; missing names are why every topic link says "chapter pending" | per lesson |
| Family fragment (from subject-config) | reasoning, register lines, table shapes, worked-example shape, traps, formats, visual plan | Register, tables and off-LOS material (duration tables in a pricing LOS; hypothesis table in a returns lesson) | per family + level |
| Source packet | excerpts labelled UNTRUSTED, repeated in planner, writer and reviewer | Grounding; also the origin of pipeline-language leaks and of the reviewer's source-gap verdicts | per lesson |
| Template orders | "knowns → conventions → formula → substitution → result → … → practice" | The planner writes sections[] with those titles; the writer emits them as ### headings (19 of 36) and, in topic mode, as bold labels (36 in one lesson) | per family |
| Plan JSON, verbatim | units plus legacy v2 sections/coverage/workedExamples prose | The lesson becomes a transcription of the plan: "teaching, not reference" scores 1 in Quant, FSA and FI | per lesson |
| Chapter unit mode | H2 per LOS, On the exam line, exact ### set, Go deeper, snapshot ≤150 words | The section skeleton (always present); never says "no other headings" | static |
| Fixture binding + fixtures JSON | "do not alter or round values"; compiled fixtures at full float precision | Raw floats copied into prose (2.94174202707276) | per lesson |
| Calculator schema example | two steps: CLR TVM, CPT PV; no inputs, no sign rule | Faked one-step calculator tables (27 of 30 blocks) | static |
| Jev question names + static code ids | bare identifiers "that will be checked" | The writer renders a > Trap: callout to satisfy trap_callout_present on top of the Common error section (30 of 36) | per lesson |
| Practice examples | one "shape only" item per planned format; children rule only in topic mode | 2-child item sets, Level II constructed responses, symbolic fill-in keys | per plan |
| Reviewer instruction + v2 schema | "teaching quality only… source support"; schema still asks for coverage, sourceEvidence, staticFindings | 8 of 8 audited lessons "changes requested" on source grounds; 0 of 11 learner-facing defects flagged | static |
Every audit defect, traced to its cause
| Defect (prevalence in 36 published) | Cause | Fix location |
|---|---|---|
| Template headings: Knowns, Substitution, Reported item (19) | Template orders reach planner and writer; plan sections[] titles are transcribed; no "no other headings" rule; normaliser only canonicalises the known set | Writer template + strict plan schema + normaliser + unit_heading_not_allowed finding |
| Bold scaffold labels in topic overviews (36 in one lesson) | Same order string injected in topic mode; overview units carry content[].type; Jev template_order_followed then pushes the chapter scaffold into overviews via repair | Topic lessons become code-assembled revision notes; per-unit Jev questions only for teaching units |
| Trap stated twice (30) | Plan field + Jev name + required Common error section; nothing says they are the same thing | Normaliser converts or drops the callout; one heading set |
Extra ### Practice heading (8) | Every template order ends "→ practice"; planner writes a Practice section | Normaliser deletes it; template orders removed |
| Register applied mechanically (hypothesis table) | Every register line concatenated and turned into a register_followed_N question with no LOS gating | appliesTo per register line; Jev plan bundle asks "does this rule apply to this LOS" |
| Scope creep beyond the LOS (duration tables) | Family fragment names spread/duration in register and table shapes; budget check excludes tables; no "only the LOS verb" line | Scope guard in the plan; Jev los_verb_alignment Score; tables counted for over-scope |
| Raw floats in prose (3) | Fixtures injected as raw JSON; rounding forbidden | Round to decimalPlaces before injection; raw_float_in_prose finding |
| Faked calculator tables (27 of 30 blocks) | Two-step example; only the last display validated; severity minor so never repaired | ≥3 steps, input before CPT, sign rule, major severity, one table per block; deterministic TVM steps from fixtures |
| Level II constructed response; 2-child item sets; symbolic keys | Family formats not level-gated in the plan validator; children rule topic-only; app stricter than API | API schema = app schema (landed); canonical chapter practice per level (landed) |
| "Summary only; chapter pending" on every link (3 of 3) | Writer never sees chapter names; v3 has no link resolution against the run's own chapters | Resolve against run chapters and LOS key first |
| Lowercase LOS headings (13) | "verbatim" vs "sentence case" conflict; normaliser landed after these lessons | Normaliser (landed), Jev heading question removed |
| Reviewer flags source gaps, not defects (8 of 8) | Instruction asks for source support; schema carries v2 coverage fields; packet fed as evidence | Reviewer scores the 12-criterion rubric with quoted spans; provenance only |
Prompt rewrite: what each stage gets instead
Planner
- Identity, LOS with its command verb, chapter id and name, usable flag.
- Family block trimmed to templates, level formats, visual kinds, traps; register lines and table shapes only where their
appliesTomatches the LOS verb or template. - Task: one unit with fixtures, calculator ids, practice {formats, count, itemSetChildren} from the canonical per-level rule, examFraming, commonError (renamed from trapCallout), and a scopeGuard list of tempting off-LOS material to exclude or push to Go deeper.
- Strict zod that strips
sections,coverage,workedExamples: the plan can no longer carry a prose draft. - No template orders. No planner call at all for topic lessons (revision notes are assembled in code).
Writer, chapter (≈4–5k tokens instead of 13k)
- Identity, LOS, On-the-exam text.
- A literal Markdown template with placeholders for the fixed headings and the rule "no other headings; write the knowns, conventions and substitution flow as prose with at most two tables".
- Teaching brief: intuition before the formula, only what the LOS verb asks, worked example distinct from the walk-through, symbols before use, the scope guard.
- Applicable register lines only.
- Fixtures as a compact rounded table, not raw JSON.
- Fence schemas with the real constraints: item-set children 4–6, one
____, a six-step calculator example with inputs and the sign convention. - Notation rules merged into one block. Deleted: plan dump, packet, Jev names, static code ids, template order, trap callout.
Reviewer
- Scores the audit's 12 criteria 0–2 with a quoted span per score; publishable at ≥16 with no zero on LOS fidelity, worked example, practice validity, clean copy.
- Keeps the numerical-claims block for the arithmetic checker.
- Schema loses coverage, sourceEvidence and staticFindings; receives provenance, not excerpts; receives the Jev-flagged sections with the question text.
Item regeneration
- Includes level, allowed option keys, the whole block (no duplicates), rounded fixture values, the Jev probabilities as feedback, the misconception-per-distractor rule, and the children and blank constraints.
- Regenerates only the failing option when the distractor bundle says so, not the whole item.
What the critics dropped from the prompt plan
- Keep the source packet in every stage. Deleting it was going to save about $2 across the entire 1,233-lesson corpus, because input is 5% of spend and the planner's copy is a cache read. It is also the only factual grounding the corpus has, and it is why Ethics scores highest in the hand audit. The justification for the rewrite is the repair rate, not the token count: 20 of 36 successful lessons needed an extra full-reasoning writer pass.
- Do not gate register rules by the outcome's verb. A quarter of the 1,030 outcomes carry more than one verb, and the prompt prints only the first. Gating "describe, compare, and interpret returns" to "describe" would delete the compare and interpret work. Scope creep gets fixed instead by a planner-authored scope guard that names the concrete off-outcome material, plus printing the full verb list.
- Do not feed topic lessons their sibling chapters. The runner has no per-module barrier, so the dependency does not exist in the pipeline the prompt runs in. That is exactly why the revision-notes design needs a two-phase runner first; until it lands, topic lessons stay self-contained.
- Do not replace the fixtures with rounded values. The practice validator drops items whose numeric key misses the compiled value by more than a hair, and a rounded fixture misses by far more. Fixtures get two columns instead: the exact value for binding, the display value for prose.
- Level gating is already closed in the validator and the planner, so no config edit is needed, only an assertion test.
Jev: today and proposed
Jev (TypeSafe System One) returns typed probabilities, not text: $0.042 per million input tokens, output free, ~0.4 s per call, 64k context. Today it costs $0.0006 per lesson and decides almost nothing.
Today
- 11 presence booleans per chapter (30 per topic) over the whole plan plus lesson; means 0.96–0.97, they never fire.
- Topic units get chapter questions: "LOS heading verbatim" fails on 16 of 16 topic lessons because overviews have no LOS heading, and the repair pushes the chapter scaffold into them.
- Item QA asks four things and acts on one; 59 answers below 0.5 on "rationales explain each option" and "tests the stated LOS" were stored and ignored, including the correct verdict on the bond lesson's non-calculating item (0.39).
- Lessons that scored 12–14 of 24 by hand passed every Jev gate.
Proposed: five bundles, section-scoped state
- B0 plan bundle (after the planner, ~3k tokens): visual kind Choice per unit, "practice formats can exercise the LOS verb", "overview unit is a teaching unit", register line applicability Nouls.
- B1 unit bundle (per unit, ~4k tokens): Scores with concrete 0/1/2 levels for LOS verb alignment, exam framing usefulness, teaching-not-reference, example distinctness, changed-condition quality, trap specificity, scaffold leak, visual usefulness, calculator plausibility, compression.
- B2 item bundle (per item, ~0.7k): key Choice over option keys as an independent answer check, a Noul per wrong option "would a candidate with a specific misconception pick this", "item exercises the LOS verb", self-answering check, Bloom Score reconciled with the declared label.
- B3 evidence bundle: citation Choice {supports, contradicts, not addressed} per tagged claim against source excerpts; contradictions go to the expert-gap list.
- B4 snapshot bundle: per-bullet "is this a tested point taught in the unit", coverage Score.
Cost ≈ $0.001 to $0.003 and about a second per lesson.
Three rules the critics imposed
- Jev never removes a stage. The proposal had a high Jev composite skip the reviewer. Rejected: the reviewer is the only factual, sign and arithmetic check, and the staging publisher requires a review result. Jev narrows what the reviewer looks at instead.
- Everything ships advisory first. A question becomes a gate only after it hits precision 0.8 and recall 0.6 on a held-out labelled set. The eight hand-audited lessons cannot calibrate anything: two criteria have no zeros and one scores full marks in all eight, so the set needs synthetic degradations (delete the worked example, swap a calculate body for a describe body, restate the substitution as the example) plus fresh labels.
- Gate on the level, never on the blended score. TypeSafe's own documentation says the score is a probability-weighted position and must not be read as a magnitude, so gating uses the top level or the probability of level zero, with confidence routing the middle band to the reviewer.
Also removed from the catalogue: anything a deterministic rule already owns (heading leaks, duplicated traps, calculator shape, dangling figures, self-answering grids, word budgets) and the noisiest questions (one asked about derivations produced 59 near-misses). Repairs get a hard budget of one batched, section-scoped call per lesson. Added: conceptual-correctness checks Jev is actually good at, such as an approximation presented as exact, a case-specific conclusion stated as a universal rule, or a clean/dirty and IFRS/US GAAP distinction used without labelling which is which.
Jev inside generation, not only after it
- Plan-time gating stops wrong practice formats, mechanical register lines and teaching-in-overview before four minutes of writing.
- Scenario choice at plan time: the planner emits two or three worked-example scenario stubs bound to the same fixture, and Jev picks the one that reads as a real exam vignette rather than a formula walk-through. This replaces best-of-N drafting, which was rejected because it multiplies exposure to the least reliable stage (seven writer deaths in sixty lessons).
- Repair targeting: repair the section with the lowest Score instead of rewriting the lesson, saving a 30k-token writer pass on the lessons that need one.
- Register applicability: per (register line, LOS) Nouls decide which rules enter the writer prompt, so the Quant "state hypotheses and tails" rule no longer produces a hypothesis table in a returns lesson.
Topic lessons become module revision notes
Why not just write it
Three designs were compared. Writing it fresh repeats the failure we already have: the writer re-teaches, and today's three surviving topic lessons run 2,529 to 3,023 words against a 2,500 cap with teaching units hidden under "Why this module matters". Assembling it from what the chapters already produced gives consistency by construction: every rule, formula, trap and exam line traces to a chapter section. One bounded model call then compresses JSON prose fields only, never Markdown and never formulas, and code renders the document. If that call fails twice, the deterministic render of the raw source ships anyway, so the layer has full availability.
What the critics changed
- No all-chapters gate. At the measured 82.5% chapter success rate, requiring every chapter would leave about 113 modules with no notes at all, and the biggest modules, where consolidation matters most, would almost never qualify. Instead a failed chapter renders one "chapter pending" block, and the notes re-assemble automatically when it lands (keyed by a hash of the chapter inputs).
- Snapshots alone are too thin. The Ethics chapter snapshot names 10 provisions where the outcome says "including the subsections of each standard", and the current topic lesson enumerates all 22. So the assembler also reads the chapter body's tables, and rule-family outcomes render as a Standard / required conduct / trap table instead of a prose block. A per-outcome check asks whether everything the outcome text demands is present, not merely whether the snapshot was copied.
- The drill is written fresh and spans outcomes. Recycling chapter items tests recognition, and single-outcome item sets cannot integrate. Chapter stems are passed as a do-not-reuse list; at Level II the item set must span at least two outcomes, at Level III the case facts come from one chapter and the rule from another.
- Queue order is not a dependency barrier. The runner becomes two-phase: the chapter pool drains, then each module assembles.
- Word budgets scale with the module. Total cap 250 + 120 per outcome; the per-block ceiling is computed in code and passed to the polish call, so an 18-outcome module is not squeezed into the same envelope as a 2-outcome one.
Shape
<intro ≤ 100 words> code prepends the area's exam weight ## Must know ### <outcome text> [[Chapter link]] one block per outcome, or "chapter pending" - Rule: bullets kept verbatim when they carry a formula, number or provision - Formula: $…$ [must recall | can derive | interpret only] · Keys: … - Changes when: up to 3 bullets - Trap: ≤ 25 words - Exam angle: ≤ 20 words - Choose this when: (Level II model-selection cue) - Answer skeleton: (discuss / recommend / evaluate outcomes) verb → conclusion → case fact → consequence ## Formula and rule sheet table, ≤ 12 rows, formulas inserted by code ## Classic errors ≤ 8 bullets, clustered across chapters ## Ten-minute drill fresh cross-outcome practice
This needs three new fields on the chapter plan so the assembler has something structured to read: the changed conditions as a list, formula cards (LaTeX, meaning, must-recall tag, calculator id), and a Level II "choose this when" cue. A follow-up step assembles a subject-area roll-up from the module sheets, which is the actual final-week artifact candidates describe; module notes are the consolidation layer between readings and after mocks.
Calculator: what was built, and why the obvious fix is the wrong order
The contract scoped the calculator as a machine-checkable keystroke table bound to the worked example's fixture, rendered as a Keys / Display table, because the research ask was "BA II Plus keystrokes beside each applicable example". It was never interactive. The writer authors the keys freehand and only the last display is validated.
| Measured on the 30 generated blocks | Finding |
|---|---|
| Step count distribution | 27 blocks have a single step; one has three, one six, one nine. No block uses the cash-flow, bond or statistics worksheets |
| Calculator settings | Zero blocks set payments per year. The device ships at 12, and clearing the time-value registers does not reset it, so the published sequence silently computes a monthly bond |
| Sign convention | Two blocks in the same file print +947.58 and −947.58 for the identical register set. Both pass validation, because only the last non-empty display is compared |
| Arithmetic-in-words blocks | One prints an expression whose answer requires algebraic entry mode; the device's default chain mode returns a different number entirely |
| Silent non-compliance | 15 plans declare a calculator; 11 lessons emit one. The four gaps produce no heading, no block and no finding. Separately, two tables render under a practice heading instead of the calculator heading |
| Blocks the planner demands but cannot exist | The planner throws unless certain templates reference a calculator fixture, but the fixture compiler is a five-operator arithmetic evaluator that knows nothing about registers, so no valid sequence exists for those lessons |
The revised plan, in the order the evidence supports
| Phase | What ships | Why this order | Gate to the next phase |
|---|---|---|---|
| Correctness and visibility | The fixture declares which worksheet it maps to and names its registers in device units; the block declares its settings and the expected sign; the validator checks magnitude at fixture precision plus the sign independently; invalid blocks and their heading are dropped deterministically; a per-run calculator report makes coverage visible | Fixes the wrong-on-device problem and the silent gaps. Crucially it also touches the planner and the fixture contract, which neither of the original options did, so coverage can actually rise instead of collapsing to two blocks | ≥90% of emitted blocks pass on a 20-lesson re-bench |
| Device drill on the existing practice path | Fill-in items that ask the learner to key the sequence on their own device and enter the display. No new app code, no new format, graded by the tolerance rule that already exists | This is the "repeated practice" the research actually asked for, and it drills the real device rather than a substitute. It ships in days, not weeks | Telemetry from ≥200 graded attempts |
| Interactive register widget | Shared calculation engine, a second calculator block version, in-lesson widget | Deliberately last. The neutral-keypad requirement means the learner drills a layout they will never touch on exam day, and tapping a highlighted key removes the tested decision (which quantity is which register, what sign, what payment frequency) while keeping the untested motor step | Product owner decision on the drill telemetry |
| Full emulator | — | Skip permanently. Free play has no oracle, so the regression suite is unbounded, and bond date conventions plus the regression worksheets are an open-ended tail | — |
One trap avoided: raising the invalid-calculator finding to major would make the repair loop fire, and with a strict validator 28 of the 30 blocks fail. At corpus scale that flips roughly 206 chapters from "ships a bad table" to "the whole lesson fails", unfixably. The finding stays non-blocking; the drop becomes deterministic instead.
Verified on the published lessons
The audit fixes are not just committed, they were replayed over the 36 lessons already on staging to confirm they catch the real defects.
| Defect class | Before | After | How |
|---|---|---|---|
| Trap stated twice | 28 | 0 | The normaliser drops the callout when a Common error section exists, and promotes it when that is the only statement of the trap |
| Template step names used as headings | 23 | 0 | Same normaliser, with a major finding for anything that survives |
| Stray practice heading before the widget | 8 | 0 | Same normaliser |
| Fabricated calculator blocks | 27 of 30 | rejected | At least three steps, an input before any compute key, no repeated header row, every display traceable to a fixture value or its own keystrokes. The three genuine multi-step blocks still pass |
| Practice items the app cannot render | 15 of 36 lessons | rejected at source | The API item schema now matches the app schema, and both test suites parse the same example file |
| Raw floats in prose | 3 | fixed upstream | The writer now receives fixtures at display precision, so it never sees a raw float; a finding is the backstop |
| Dangling figure references | 1 | fixed upstream | A dropped figure takes its introducing sentence with it |
Two rules were narrowed after the adversarial reviewer caught them. The self-answering grid test now requires the offending column to actually be the row's answer, because the first version deleted a legitimate equities grid where a column label merely contained the word "rights". And the Level III constructed-response rule records a mismatch rather than throwing, because the first version would have failed 8 of 11 Level III chapter blocks outright.
Roadmap
Ordered so that each stream unblocks the next. Everything before the first horizontal rule is already committed on the API branch.
| Stream | What lands | Blocks | Effort |
|---|---|---|---|
| 0. Landed and verified | Practice validator matched to the app schema with shared examples both suites parse; copy normalisers; fixture rounding at prompt-injection time; calculator step rules; dangling-figure and visual-misuse handling. 386 tests, typecheck clean | — | done |
| 1. Remaining from the audit | Chapter-link resolution against the run's own chapters, and the reviewer repointed at the 12-criterion rubric | Topic links that say "chapter pending" when the chapter exists; the reviewer flagging learner-facing defects | ~1 day |
| 2. Reliability and provenance | Revert to max effort, record the effort actually used, transport retry separate from content repair, per-stage resume checkpoints, run lock, spend cap, fail fast before the review | Every measurement and experiment after it | ~1 day |
| 3. Repair redesign | Patch-scoped repairs with the plan and fixtures in context, spliced by heading anchor; regression gate failing any repair that drops 15% of words, a heading or a fixture value; normalisers applied to Jev repairs; item regeneration before the review | The largest quality lever; also the speed one | 2–3 days |
| 4. Prompt rewrite | Planner loses the template orders and the legacy prose fields and gains a scope guard; writer drops to a literal template at about 4.5k tokens; reviewer scores the 12-criterion rubric with quoted spans; item regeneration gets level, keys, fixtures and the real feedback | Defect classes at source rather than by normaliser | 2 days |
| 5. Jev bundles | Plan bundle (register applicability, scenario choice), unit bundle (rubric Scores), item bundle (key check, distractor quality) — all advisory first, with a labelled set including synthetic degradations, then promotion to gates at precision ≥ 0.8 and recall ≥ 0.6 | Repair targeting and reviewer focus | 5–7 days plus labelling |
| 6. Module revision notes | Chapter plan gains changed conditions, formula cards and the Level II cue; assembler, polish and drill stages; two-phase runner; new findings; app "Revise" segment with collapsible blocks | Replaces the topic overview entirely | 7–9 days |
| 7. Calculator | Correctness first (worksheet-aware fixtures, settings and sign in the block, deterministic drop, coverage report), then the device drill on the existing practice path, then the widget only if the drill telemetry justifies it | Coverage rises instead of collapsing | ~3 days, then ~2, then gated |
| 8. Re-benchmark and re-audit | Fresh 60-lesson run at head, concurrency measured at 30/45/60, hand audit of a new sample of eight against the rubric | The go/no-go for the 1,233-lesson corpus | 2 days |
Decisions the product owner owns
- Overnight at full quality, or faster now? An all-max corpus pass is about 8 hours at 30 streams and roughly $175. Cutting effort saves about 3.4 hours per pass but measurably thins plans and reviews. Recommendation: pay for quality, earn the cuts through gated experiments.
- Who scores the blind rubric samples until the reviewer is repointed at the rubric? About eight lessons and two hours per experiment.
- The nine staging lessons generated at high effort: regenerate (recommended) or keep with corrected metadata.
- Is "Revise" the default tab for a CFA topic, or does Deep Dive stay default with a "Revise this module" button?
- Interactive calculator: the recommendation is now correctness, then a drill on the learner's own device, and the widget only if the drill data justifies it. Confirm that ordering, or ask for the widget sooner.
- A separate model key for benchmark runs, isolated from production, before any 30-stream run.