How a CFA lesson gets made

The simple version: what each step does, what the words mean, what went wrong, and what we change.

1. It is an assembly line with four stations

Three of the stations are the AI writing. One is our own code checking. Every lesson goes through all of them, in order.

AI · planPlannerReads the official learning goal and our teaching rules. Writes a plan: which sections, which examples, which trap to warn about.
AI · writeWriterTurns the plan into the actual lesson a learner reads — headings, worked example, common error, quiz.
code · checkChecksOur code reads the lesson like a strict editor: right headings? numbers from the approved list? quiz items valid?
AI · judgeReviewerA separate AI acts as a tutor: is this actually good teaching? Which numbers should a human double-check?
REPAIR Whenever a check fails, we send the exact error back to the AI and ask for a corrected version. Once or twice, then we stop and record the failure.

Timing, from a real lesson we recorded: planner 50 s, writer 160 s, checks under 1 s, reviewer 42 s. Almost all the wait is the AI thinking.

2. The words, in plain English

LOS
The official one-line goal from CFA Institute that a lesson must teach.“compare the organizational forms of businesses”
Plan
The planner's blueprint, written as structured data: section list, examples to use, trap to warn about, quiz formats.
Fixtures
The numbers for worked examples, computed by our code, not the AI. The writer must use them exactly. This is how we stop the AI inventing wrong arithmetic.Like a cooking show where the ingredients are pre-measured before the chef walks on.
Charts and pictures
Three kinds. Mermaid: the AI writes plain diagram code — decisions and processes only — then a separate “stylist” call redraws it the way the medical lessons look (vertical layout, a title plus one or two detail lines per box, colours by meaning: blue inputs, yellow decisions, purple rules, red traps, green outcomes) and our code accepts the redraw only if it says exactly the same thing. The app draws it. Charts: also Mermaid (its xychart type). The AI writes the bar or line values into the chart text, and our code checks every value against the numbers it computed before the lesson can pass; a stylised curve must say “Illustrative” in its title. Cash flows over time are tables. Illustration: the AI asks an image model for a picture — structures, shapes, a calculator keypad — and is forbidden from asking it for any number.
Static checks
Rules our code applies with no AI involved — a spell-checker for structure. Fast and deterministic.“A topic lesson may only use its five headings.”
Jev
A small, cheap AI judge (from TypeSafe). It answers yes/no questions about the lesson with a confidence.“Is there a trap callout?” → 0.92 yes
Reviewer
The third AI call. Reads the finished lesson as a tutor and lists teaching problems and numeric claims to verify. It reports; it does not rewrite.
Repair
Sending a failed plan or lesson back to the AI with the exact error, asking for a fixed version. “Bounded” means at most one or two tries.
Calculators, media, practice
Code steps that turn special blocks in the lesson into BA II Plus keystrokes, charts, and quiz items — and validate each one.
Trace
A recording of exactly what every step received and returned. That is what the clickable table in the deck shows.
Bedrock
AWS's service that hosts the GPT-5.6 model for us. Same model as before, different door. openai.gpt-5.6-luna is just Bedrock's name for it.

3. What the recording caught

A repair is asked to “return the complete corrected plan”. So the AI is free to return less. One tiny error cost a plan most of its teaching.

The planner's first attempt had one problem: an extra field the checker did not recognise (mermaid, in the wrong place). We sent that one error back. The second attempt was valid — and had quietly thrown away most of the plan.

Attempt 1 — one bad field

Sections8
Worked examples2
Coverage note1
Revision points5

Attempt 2 — “fixed”

Sections0
Worked examples0
Coverage note0
Revision points4

We had already measured the same thing on lesson rewrites: 11 of 37 repairs lost more than a fifth of their words. Now we have it on the planner too, with both attempts saved side by side.

The second thing it caught: the topic lesson (the overview of a whole module) used up both repairs and still failed. 13 of its 15 remaining errors were the same one — the writer keeps adding chapter-style sub-headings to a format that forbids them. Asking again does not fix a habit.

4. The fix, in one line each

Now

Fix the typo by rewriting the essay. The AI gets the error and returns the whole thing again.

- send error + whole plan
- ask for "complete corrected plan"
- accept whatever comes back

For topic lessons, stop asking the AI to free-write an overview at all. Our code assembles the revision notes from the chapter lessons that already passed — the rule, the formula, the trap, the exam angle for each goal — plus a formula sheet and a fresh quiz. Nothing to repair, because nothing is rewritten.

5. Where things stand

Done
Every AI call runs on Bedrock. Every step is recorded. The deck shows real prompts and real outputs, click any row.
Done
Six of the checker's earlier blind spots closed and verified on the 36 published lessons (duplicate trap 28 → 0, template headings 23 → 0, fake calculators 27 of 30 rejected).
Next
Patch-scoped repairs with the content gate. Topic lessons as assembled revision notes. Then re-run 60 lessons and re-audit 8 (target: 7 of 8 publishable, up from 4 of 8).