In my system, a multi-year analysis could legitimately mean at least three different things (tracking the same students’ growth across years, comparing this year’s cohort against last year’s, or trending school-wide results over time). So I defined the three ways we do it and wrote selection rules based on the shape of the district’s data. That handles the ambiguity machines can settle.

For the ambiguity they can’t: when a renewal-report request hits a district running our courses in multiple subjects, the system stops and asks me which subject I mean (or whether I want all of them) instead of guessing.

Ambiguity resolves through a defined menu of interpretations with selection rules — and when the rules can’t decide, the system asks the operator instead of guessing.

Enumerate the legitimate interpretations before the model ever meets them. Encode the rules for choosing. Gate whatever remains on a human. The model generates within a chosen interpretation — it never chooses silently.

I audited one end-to-end run of my report skill: 164 structured events emitted, six distinct drift modes — with three correct reference docs sitting in the working directory the whole time. Separately, a 1,000-line skill spec that ran clean when I supervised it interactively started silently skipping steps the moment it ran as a background agent. Same spec. Same code. Same data.

Anything deterministic must be code, not prose the model re-interprets on every pass. The longer the run, the more this matters.

A validating helper the model calls with arguments — it never writes the fixed shape itself. The clearer your prose, the more tempting it is to trust it. Don’t. Better prose isn’t the answer; not-prose is.

In that same 164-event audit: the subtype sql_query was promoted into the type: field on 39 of 164 events; an invented entry: START/END field appeared on 34; nine invented subtypes fell outside the canonical enum; the journal landed in the wrong directory.

The producer of a fixed shape validates structure and rejects unknowns. The model passes data — never format.

A small CLI owns the schema: it allocates IDs, stamps timestamps, hardcodes the path, validates every enum, rejects invented fields. Five of my six drift modes disappeared by construction — the model can’t drift a file it no longer writes.

While registering a new SQL query with verified logic, the model hand-typed the expected result of that query instead of running it. The typed value was 530x off from what the warehouse actually returned.

This one incident put a hard gate on expectation strings (e.g. “this query should return X”): computed, never authored.

No value reaches output unless execution produced it during this run. Result strings bind to the query that made them — never to memory.

Separate compute from narrate. Registering a metric forces the query to execute at registration time, so the recorded expectation is real by construction. The narrator may reference computed values by name; it can never mint one.

The fix: narrative templates carry tokens instead of numbers. A sentence is authored as “{funnel_enrolled} students enrolled in ChalkTalk,” where {funnel_enrolled} names a registered SQL query or Python calculation. At render time it becomes “12,651 students enrolled in ChalkTalk.”

Interpolated values render visibly distinct from authored prose — a reviewer sees at a glance which words are data, and can click a number to land on the query that produced it.

Every claim traces to a recorded event — and the trace is inspectable. Data-bound text and authored prose must be distinguishable at a glance.

One read model for narrative and evidence: numbers live as tokens bound at render time, never retyped. A claims file with source and reasoning per claim; a halt when a narrative number matches no recorded event. Provenance that exists but can’t be read is provenance that doesn’t exist.

I built a whole validation layer because of this zone. The checks: grain (each row unique on declared columns, catching fan-out), rollup (recomputing the same number from its finest-grained rows gives the same total — one query checked against its own math), scope (no rows from outside the declared district, curriculum, and year), source-consistency (the declared table matches what the SQL actually joins), and reconciliation (separate outputs checked against each other — the counts published in one section must sum to the headline in another, and independent sources must agree within tolerance).

Declare invariants on the output shape — uniqueness, parts-equal-whole, no leakage — and mechanically verify them at every step. Verify the middle, not just the ends.

The invariant rides as a declaration on the data model itself and is checked per-query at runtime. Severity is a product decision per constraint: fan-out and broken rollups halt; a possible scope leak proceeds but gets flagged where the human and the narrator both see it.

A report cover read: “reaching 334 of 440 enrolled students and a path to doubling the paired-evidence base.” Every number was right — the doubling meant 109 going to roughly 220. But put “doubling” next to “334 of 440” and the reader’s eye computes an impossible 668. Correct data, false reading.

Check how a number will be read, not only whether it is right. A multiplier must bind explicitly to its base.

A render-time check for perception traps — mine detects a multiplier landing near an unrelated base and halts unless the sentence carries its own binding: “doubling (from 109 to ~220).” Data-correctness gates can’t catch this class. It needs its own gate.

My version: phase 2’s exact numbers reaching phase 3 as lossy conversational summary, then a hollow phase poisoning everything downstream while the final report read beautifully. Until each phase boundary became a checkable artifact, localizing a cascade meant re-reading a 90-minute transcript by hand.

Deep orchestration must decompose into phases that each prove their own correctness — or a coherent-sounding whole hides a broken part you cannot find.

Checkpoints between phases — downstream reads the file, never the prior prose. Per-phase execution counts. Reconciliation at phase boundaries, so a dead layer surfaces after phase 1 instead of after the run. Orchestration depth raises the stakes of per-phase proof; it never lowers them.

One run executed every phase inline instead of dispatching them: substantive-looking deliverables, zero telemetry, and about 115 false alarms from confused downstream checks. An earlier run had silently dropped an entire analysis phase and two customer PDFs — nothing checked for them, so nothing noticed. The hardening that followed cut detection from ~90 minutes to ~30 seconds.

Every mandated step proves it ran. A step that cannot show execution evidence halts the run. Zero evidence is a failure — never a clean run.

A completion contract (a machine-readable list of what each phase must produce) plus per-phase execution counts means a hollow phase halts before the next one builds on it. The elegant endgame: the retry path re-dispatches the phase, and a dispatched subagent physically cannot run inline — the architecture cures its own failure mode.

An empty environment variable made my logging hook silently no-op. Hooks installed mid-session didn’t fire until restart: one district’s run captured 0 of 26 expected events and finished “successfully.” The report was right. The telemetry was fiction — and nothing knew.

An end-of-run assertion that the telemetry captured reality — and the checker shares code with the thing it checks, so the two can never drift apart.

Reconstruct what should have been recorded from an independent source — the session transcript — and diff it against what was recorded; hard-fail on zero-capture. A detail I’m proud of: the gate imports the recorder’s own decision function instead of re-implementing it. The anti-drift lesson, applied to the anti-drift tool.

When the system ran unattended, the steps it skipped were never the judgment calls — they were the boilerplate. Every run ends with the same four wrap-up commands. Each prose handoff was one more chance for attention to wander — exactly where the skips clustered. One 90-minute run wasted about 50 minutes on work it had abbreviated and then had to redo.

Code drives sequencing. The model is invoked only where the next step requires judgment.

A driver script picks the next step from state. It calls the model surgically — choosing the story angle, drafting narrative, talking to the operator — and runs helpers directly for everything else. Skipping becomes structurally impossible, and you stop paying inference prices for couriering.

I run all three shapes in one system, deliberately. Heavy data phases fan out — context isolation and parallelism. Judgment moments and operator gates stay inline, where the orchestrator can see and ask. Choosing wrong in either direction cost me: sequential drift on long runs, and contained subagents that couldn’t surface the question they should have asked.

Orchestration shape is a first-class design decision trading accuracy × performance × control — chosen per step, never globally.

Fan out for isolation and independent parallel work. Stay inline wherever a clarification gate or a human question lives. A driver mediates between the two. This is more than a telemetry choice — it decides what the system can notice and what it can ask.

My first skill was a 600-line monolithic file — every rule, every reference, loaded up front. By line 300 the agent had forgotten the constraints from line 50. It performed worse than having no skill at all.

Context loads progressively: start small, navigate to detail on demand, stop at the shallowest layer that answers the question.

Layered structure — a small entry point with dispatch logic, phase files that load when their phase fires, references that load when pointed to. The budget you’re actually managing is attention, not tokens.

Mid-run, phase 2 produced exact metrics — paired_students: 247. The conversation compacted between phases. Phase 3 read “about 250 paired students” from the summary and built the customer-facing artifacts on it. The report was off by single-digit percentages everywhere, for no visible reason.

Persist load-bearing facts to disk the moment they exist. Files survive compaction; conversation prose does not.

A checkpoint file written at every phase end, in structured YAML, so a value like paired_students: 247 is data, not a sentence a summarizer can soften into “about 250.” Narrative may be summarized; hard numbers are written to disk before they are ever presented. Downstream phases read the file, never the prior prose.

The same branch, path, and formatting mistakes recurred session after session until each became a written rule in a file the agent loads at startup. Then the sharper proof: I came back from my two-week honeymoon and couldn’t remember some of the finer details of how the “shared learnings & rules” system I had built four weeks earlier worked — which file fed which reviewer, where a lesson was supposed to live.

Every correction becomes a rule in a known home that the right reader loads automatically — at write-time and at review-time.

Route the lesson by who needs it: personal habits into per-user memory; repo standards into committed rule files mirrored into the AI reviewer’s config. One lesson, two readers — the writer never makes the mistake, and the reviewer catches it if it slips through anyway.

I shipped a redesign, self-reviewed the diff, and it looked clean. A different vendor’s model (Codex reviewing Opus’s work, in my case) then returned 9 findings — 7 were real bugs that would have gone to production. The disagreement between models was the signal.

A second, different model reads every change before it ships — and you validate its findings against source, because it is confident when wrong, too.

Cross-model review as a standing gate: one model writes, a different one reviews, a human adjudicates with the source open. Model diversity — different training, different priors — catches what repetition can’t.

My skills share a library of capabilities — telemetry, shared Python helpers, reference data files. Each skill declares which capabilities it adopts and which it opts out of, with a justification, so one canonical implementation serves many skills and a linter can check each declaration against reality.

The linter’s first version emitted an advisory note for every declared opt-out — which it never actually verified. Any skill could copy an opt-out clause from a neighbor, forget to update the wording, and pass forever. The fix made the check enforceable, with a test for exactly the copy-and-forget case.

Every declaration is machine-checked against reality. An unverified promise is treated as false.

The dial-and-machine pattern: a declaration you write (the dial) is only meaningful when automatic machinery reads it and fails when it doesn’t match reality (the machine). A dial with no machine is decoration.

I built a baseline-drift mechanism to catch numbers shifting between runs. Across weeks of real runs: one baseline ever committed, zero drift comparisons performed. I deprecated it — and deleting it is what forced the better mechanism (binding results to live execution) to exist at all.

Build the gate when a failure proves it’s needed. Delete dormant machinery — its presence implies a coverage that isn’t there.

Keep an explicit ledger: what’s automated, what’s still prose, what’s a promotion target. Promote on proven pain. Retire on proven disuse. The ledger of what you haven’t hardened is as load-bearing as the hardening.