How a CFA lesson gets made
The simple version: what each step does, what the words mean, what went wrong, and what we change.
1. It is an assembly line with four stations
Three of the stations are the AI writing. One is our own code checking. Every lesson goes through all of them, in order.
Timing, from a real lesson we recorded: planner 50 s, writer 160 s, checks under 1 s, reviewer 42 s. Almost all the wait is the AI thinking.
2. The words, in plain English
- LOS
- The official one-line goal from CFA Institute that a lesson must teach.“compare the organizational forms of businesses”
- Plan
- The planner's blueprint, written as structured data: section list, examples to use, trap to warn about, quiz formats.
- Fixtures
- The numbers for worked examples, computed by our code, not the AI. The writer must use them exactly. This is how we stop the AI inventing wrong arithmetic.Like a cooking show where the ingredients are pre-measured before the chef walks on.
- Charts and pictures
- Three kinds. Mermaid: the AI writes plain diagram code — decisions and processes only — then a separate “stylist” call redraws it the way the medical lessons look (vertical layout, a title plus one or two detail lines per box, colours by meaning: blue inputs, yellow decisions, purple rules, red traps, green outcomes) and our code accepts the redraw only if it says exactly the same thing. The app draws it. Charts: also Mermaid (its xychart type). The AI writes the bar or line values into the chart text, and our code checks every value against the numbers it computed before the lesson can pass; a stylised curve must say “Illustrative” in its title. Cash flows over time are tables. Illustration: the AI asks an image model for a picture — structures, shapes, a calculator keypad — and is forbidden from asking it for any number.
- Static checks
- Rules our code applies with no AI involved — a spell-checker for structure. Fast and deterministic.“A topic lesson may only use its five headings.”
- Jev
- A small, cheap AI judge (from TypeSafe). It answers yes/no questions about the lesson with a confidence.“Is there a trap callout?” → 0.92 yes
- Reviewer
- The third AI call. Reads the finished lesson as a tutor and lists teaching problems and numeric claims to verify. It reports; it does not rewrite.
- Repair
- Sending a failed plan or lesson back to the AI with the exact error, asking for a fixed version. “Bounded” means at most one or two tries.
- Calculators, media, practice
- Code steps that turn special blocks in the lesson into BA II Plus keystrokes, charts, and quiz items — and validate each one.
- Trace
- A recording of exactly what every step received and returned. That is what the clickable table in the deck shows.
- Bedrock
- AWS's service that hosts the GPT-5.6 model for us. Same model as before, different door.
openai.gpt-5.6-lunais just Bedrock's name for it.
3. What the recording caught
The planner's first attempt had one problem: an extra field the checker did not recognise (mermaid, in the wrong place). We sent that one error back. The second attempt was valid — and had quietly thrown away most of the plan.
Attempt 1 — one bad field
Attempt 2 — “fixed”
We had already measured the same thing on lesson rewrites: 11 of 37 repairs lost more than a fifth of their words. Now we have it on the planner too, with both attempts saved side by side.
The second thing it caught: the topic lesson (the overview of a whole module) used up both repairs and still failed. 13 of its 15 remaining errors were the same one — the writer keeps adding chapter-style sub-headings to a format that forbids them. Asking again does not fix a habit.
4. The fix, in one line each
Now
Fix the typo by rewriting the essay. The AI gets the error and returns the whole thing again.
- send error + whole plan
- ask for "complete corrected plan"
- accept whatever comes back
Proposed
Fix only the broken piece, and refuse a fix that shrinks the work.
+ send error + only the broken piece
+ ask for that piece, corrected
+ our code splices it back in
+ reject if the result lost content
(sections, examples, or > 10% of words)
For topic lessons, stop asking the AI to free-write an overview at all. Our code assembles the revision notes from the chapter lessons that already passed — the rule, the formula, the trap, the exam angle for each goal — plus a formula sheet and a fresh quiz. Nothing to repair, because nothing is rewritten.
5. Where things stand
- Done
- Every AI call runs on Bedrock. Every step is recorded. The deck shows real prompts and real outputs, click any row.
- Done
- Six of the checker's earlier blind spots closed and verified on the 36 published lessons (duplicate trap 28 → 0, template headings 23 → 0, fake calculators 27 of 30 rejected).
- Next
- Patch-scoped repairs with the content gate. Topic lessons as assembled revision notes. Then re-run 60 lessons and re-audit 8 (target: 7 of 8 publishable, up from 4 of 8).