Deck · Pipeline & prompts · The simple version · Before and after · Luna vs Muse · Theme
The final CFA lesson pipeline
One lesson, station by station: who does the work, what comes out, how long it takes, and what happens when it fails. Then how 1,233 lessons get through it and into staging. Numbers are from the 22 September run of 20 lessons (10 topic overviews, 10 chapters, Levels I to III, nine subjects), all 20 in flight at once.
1. One lesson, station by station
Timings are per-lesson means from the run above. "Repair" means the exact problem is sent back with the text, at most once per section, and a repair that shrinks the work, changes a heading, or touches a fence it was not asked to change is refused.
Per lesson
Cost per lesson
Throughput
Pass rate
2. What the section review did on those 20 lessons
| Measure | Count | Reading |
|---|---|---|
| Sections judged | 117 | 5.9 per lesson |
| Sections with a gated failure | 79 | advisory questions excluded |
| Repairs attempted / applied / reverted / refused by a guard | 96 / 75 / 13 / 8 | refusals were "shrank", "fence changed", "heading changed" |
| "Talks to the candidate" failures before → after | 16 → 4 | the voice repair works |
| Practice item rationale failures before → after | 15 → 4 | fixed once practice fences were allowed to change |
| "Numbers from fixtures" before → after | 26 → 23 | half real: a changed-condition section invented a fresh scenario (10,000,000; 800,000) that no fixture holds, which a rewrite cannot fix. Half is Jev being conservative on tables. |
| "Overview stays an overview" before → after | 15 → 13 | the Learning outcomes section keeps teaching; a rewrite does not remove content. This needs a template change, not a repair. |
"Nothing repeated" was asked on every prose section and failed nearly all of them, repaired or not; it is now recorded but never a trigger until calibrated.
3. Against the previous flow
| Previous (whole-lesson Luna reviewer) | Now (Jev per section + section repairs) | |
|---|---|---|
| Wall time per lesson | 530 s at concurrency 5 | 529 s at concurrency 20 |
| Cost per lesson | ≈ $0.12 | $0.133 |
| Throughput measured | 26 / h | 88 / h |
| Reviewer output used for anything | No: 8 majors on 8 lessons, none fixed | Yes: 75 section repairs applied |
| Writer share of time | 49% | 58%; the writer's reasoning effort is the one lever left |
4. What keeps a long run alive
- Retries with backoff. Throttling, 429, 5xx, dropped sockets, connect timeouts and cut streams retry up to six times (3 s doubling to 90 s). Everything else fails the lesson at once. Retries are logged and counted per batch.
- Cut streams are failures, not drafts. The SDK reports a mid-stream cut as an error part and resolves the text with what arrived; the stage now throws so the retry runs it again. Found when the network dropped 24 streams at once during a run; the next run had zero incidents.
- Ref'd deadlines and a heartbeat. A 30-minute deadline per stage on a timer that keeps the process alive, a heartbeat every minute with in-flight lessons and retry counts, a warning if the loop ever drains with work pending.
- Resume. Every lesson checkpoints; a run restarts from progress and skips finished lessons. Resume refuses if code or inputs changed, so a fix means a new run name.
- Evidence. Raw outputs, hashes, plans, section verdicts, arithmetic checks and full step traces per lesson; the cost report is computed from them.
5. From one lesson to all 1,233
sealed run cfa-v3-all: 203 topic overviews + 1,030 chapter lessons (2027 curriculum, staging hierarchy)
batch ladder --review-mode jev-sections --max-lessons N --concurrency C --batch-label cC → batches.jsonl
20 done (88/h, no throttling); next 50 and 100 to find the ceiling
continue at the best concurrency with zero throttling, --resume until done; Jev token refreshed every 12 h
publish publish-staging-v3 --run-name cfa-v3-all [--apply | --verify]
per lesson, one transaction: lock the stub → concept groups (one per H2), blocks, mappings →
assets to the notes bucket (sha256 paths) → assets + concept_group_assets → lesson metadata
{stub:false, curriculum ids, cfaRevision} → read back → COMMIT. Dry-run is read only.
later migrate-to-prod: copy by cfaRevision.runName, re-upload assets, remap URLs
6. Decisions locked
| Decision | Why |
|---|---|
| Luna writes; Luna repairs | Muse Spark matched Luna on the judge's rubric at a tenth of the cost, but read in full its quantitative drafts are thin, it broke table syntax, and its reviews miss issues. Its voice is the target, not the model. GLM equalled Luna; MiMo failed two calls in three. |
| Jev judges, per section, with a rubric | About a second and a fifth of a cent per call; precise because it sees one section against the plan and fixtures; its answer is a location for a repair, not a verdict to file. |
| Whole-lesson Luna reviewer | 99 s and $0.02 per lesson for a verdict nothing acted on. Its one irreplaceable job, spotting finance errors like a conflated forward value, is not covered by Jev; that is the open item below. |
| Repairs per section, in parallel, guarded | A whole-draft rewrite for one defect lost teaching 11 times in 37 on the benchmark. A section rewrite cannot lose what it never saw. |
| Deterministic before AI | Tables, headings, template labels, step lists and cut streams are handled in code, so no model call is spent on them. |
| Mermaid for diagrams and charts, illustrations for pictures, one per lesson | Chart numbers are checked against fixtures; an image model is never asked for a number. |
7. Still open, in order
- Finance-error check. Nothing in the new flow reads a chapter for a wrong sign convention or a conflated definition. Options: a narrow Luna pass that only lists factual errors with quotes (about 50 s) feeding the same section repair, or a Jev question set per section built from the subject register rules. The register rules are already asked; the factual pass is not.
- Illustrations. Only 6 of 20 lessons got one: the visual floor counts figures in total, so enough Mermaid satisfies it. A per-kind floor in the plan validator is a small change.
- Overviews that teach. "Learning outcomes" keeps becoming a lesson; the template should cap it to the LOS table plus two sentences per outcome, enforced by code.
- Invented scenario numbers. When the changed condition invents inputs, the fix is a fixture, not a rewrite: the section repair should ask the planner for the fixture or delete the numbers.
- Writer latency. 308 s at max effort is 58% of the lesson; measure high effort on the same 20 lessons before the full run.
- The three old static collisions (prose runs, malformed table) that the bounded repair cannot clear.
- App side: pin Mermaid 11, load SN Pro, apply the class definitions and theme.