Lesson 16The Build Playbooks
Welcome to Module 11: Real-World Skills — the final module. Notice what it can't contain: new theory. There is none left. Every topic today is assembly — course pieces snapped into the product shapes that actually exist in the world, plus the judgment layer that decides what's worth building. This is the module where you stop being someone who understands AI systems and become someone who ships them.
Topic 78: Building Chatbots — the canonical assembly
The most-built, most-commoditized, most instructive product shape — because a good chatbot is the entire course in one stack:
The build sequence that works, in order: (1) Scope before anything — write the system prompt's escape hatches first (Topic 55): what this bot refuses, deflects, and escalates. Undefined scope is the root cause of most chatbot embarrassments. (2) Ground it — RAG over the real knowledge (Module 6), citations on, "say so if it's not in context" (Topic 48), golden set built the same week (Topic 75). (3) Manage the conversation — query rewriting for follow-ups (Topic 52, non-negotiable), history summarization before the quadratic cost curve bites (Topics 53/77). (4) Engineer the exits — the most underrated feature in the entire category: graceful human escalation. A bot that recognizes its limits and hands off with full context beats a bot that's 5% smarter and traps users; escalation rate and quality belong on the dashboard. (5) Capture feedback — 👍/👎 on every response: it's your eval signal (Topic 74) and your KTO training data (Topic 28) accumulating for free.
The chatbot-specific failure modes to design against: scope drift (users treat every chatbot as general-purpose ChatGPT — your refund bot will be asked for poetry and medical advice; the escape hatches are load-bearing), injection via retrieved content (Topic 57 — your own indexed documents are an attack surface), and measuring vanity — the metrics that matter are resolution rate, escalation rate, cost per conversation, and CSAT, never "messages sent" (engagement with a support bot is often a failure signal: users message more when unresolved).
And the honest market note: the chat interface is fully commoditized — anyone can ship one in a weekend. Differentiation lives entirely in layers 3–6 of the diagram: knowledge depth, integration into real systems (a bot that can actually check the order status via tools beats one that talks about it), and the eval-driven quality grind nobody sees.
Summary
Chatbot = the full course stacked: scoped constitution, grounded RAG, managed conversation, engineered escalation, captured feedback — measured by resolution and cost, differentiated by knowledge and tools, never by the chat window.
Mental model
A new support hire: worth exactly their training binder (RAG), their standing orders (system prompt), and their judgment about when to get the manager (escalation) — not their friendliness.
Mistakes to avoid
shipping without escalation ("the bot handles everything" is a promise you'll break publicly); and celebrating engagement metrics on a task-completion product.
Exercise
Assemble it — you've built every layer as separate exercises (Topics 48, 50, 52, 53, 55, 75). Wire them into one chatbot for a domain you know, with escalation as a literal escalate_to_human tool (Topic 56). Run your golden set through it. This is the course's first complete product.
Topic 79: Building AI Copilots — intelligence inside the workflow
A chatbot is a destination; a copilot is a passenger — it lives inside an existing tool (editor, inbox, CRM, spreadsheet), sees what the user is doing, and proposes; the user disposes. The category's defining law:
Context is the product. Two copilots with identical models differ entirely in what they can see — the open document, the selected code, the ticket thread, the cursor position. Copilot engineering is 80% context assembly (Topics 5, 53, 58): deciding, per invocation, which slice of the user's world goes on the whiteboard.
The interaction patterns, each with its own physics: inline completion (Topic 66's FIM — a hard <500 ms budget that mandates SLMs, Topic 63, and makes this the one pattern where model choice is dictated by latency, not quality); selected-text actions ("rewrite," "explain," "fix" — burst-shaped, mid-size models); side-panel chat with context (the workhorse — a chatbot that can see the workspace); and background drafts awaiting review (pre-written email replies, suggested ticket responses — latency-free because asynchronous, so the biggest model is affordable here; note the elegant inversion: the pattern users see instantly gets the smallest model, the one they see later gets the largest).
The structural advantage nobody should waste: every suggestion is accepted, edited, or rejected — which means a copilot generates labeled preference data as exhaust (Topic 16's triplets: shown vs. what-the-user-kept), plus a free north-star metric: acceptance rate, trackable per feature, per context type, per model version — a production eval (Topic 75) running itself. This flywheel is why copilots that ship early and instrument well become uncatchable: their training data is their usage.
Trust design completes the shape, straight from Topic 57's consequence tiers: suggestions are previewed, provenance-marked, one-keystroke-undoable, and consequential actions (send, commit, delete) are never auto-applied. A copilot that once silently mangles a document loses more trust than a hundred good suggestions earned — design for the error case first.
Summary
Copilot = context assembly + latency-tiered interaction patterns + suggestion-not-action trust design, with acceptance rate as the north star and accept/reject exhaust as a preference-data flywheel.
Mental model
A brilliant colleague reading over your shoulder who only ever hands you sticky notes — never touches your keyboard, and learns your taste from which notes you keep.
Mistakes to avoid
using one model for all patterns (the 500 ms completion path and the async draft path are different animals — Topic 38's matrix applies within one product); and discarding the accept/reject stream — that's your moat leaving through the logs.
Exercise
Spec a copilot for a tool you use daily (your editor, email, or issue tracker): list the five context sources you'd assemble per invocation and their token budgets, assign each interaction pattern a model tier with its latency budget, and define what "accepted" means telemetrically for each. One page — the copilot PRD.
Topic 80: AI Automation — humans mostly out of the loop
Chatbots and copilots serve a present human. Automation runs while nobody watches: ticket triage, document extraction, data enrichment, report generation, monitoring. Different presence, different design center — with no human catching errors in real time, reliability engineering is the whole game, and everything follows from Topic 58's compounding math:
Workflows over agents (Topic 59's rule at maximum strength — automation tasks are usually knowable structures: chain them, gate them, keep the model inside code-decided rails). Verify everything verifiable (schema validation via constrained decoding — Topic 54 — is the automation workhorse; execution checks where possible). And the pattern that makes automation shippable at all — confidence-based triage, Topic 38's router applied to autonomy itself:
Auto-handle the high-confidence cases; queue the low-confidence ones for humans. A system that's 90% reliable overall is unshippable as full automation — but often 99.5% reliable on the 70% of cases it's confident about, which ships beautifully as "automation with a review queue."
That yields the autonomy spectrum every automation should be consciously placed on: human-in-the-loop (approve each action — for consequential/irreversible work), human-on-the-loop (review queues, sampled audits, exception handling — the workhorse tier), human-out-of-the-loop (reserved for verified-or-cheaply-reversible operations only). Moving rightward is earned by eval evidence (Topic 75), and the deployment gate is beautifully concrete: automation ships when its measured error rate on the auto-handled slice beats the human baseline error rate — humans make mistakes too, and "better than the status quo, measured" is the honest bar, not "perfect."
Two boring-but-decisive notes: classic automation engineering applies unchanged — idempotency, retries with backoff, dead-letter queues for poisoned inputs; the LLM is just a fallible worker in a pipeline your existing engineering instincts already know how to harden. And economics: most automation is offline and latency-indifferent → Topic 36's batch tiers cut the bill in half by default. The market note: this is the unsexy, enormous category — document processing and categorization at scale won't demo virally, and it's where a large fraction of real AI value (and contract revenue) actually concentrates.
Summary
Automation = workflows with gates, constrained outputs, and confidence-triaged autonomy: auto-handle the confident slice, queue the rest, ship when measured error beats the human baseline, run it on batch economics with classic pipeline hardening.
Mental model
A mailroom that sorts 70% of parcels automatically with near-perfect accuracy and places every uncertain parcel on the supervisor's desk — versus one that guesses on everything and misroutes silently. Only one of these gets trusted with more mail.
Mistakes to avoid
framing the decision as "automate or don't" instead of "automate which confidence slice"; and skipping the human-baseline measurement — you can't beat a bar you never measured, and humans are worse than teams assume.
Exercise
Pick one recurring drudge task you actually do (triaging notifications, categorizing expenses, extracting fields from documents). Design its automation: the workflow steps with gates, the confidence signal you'd triage on, where it starts on the autonomy spectrum, and the eval that would earn it a rightward move. Then build the first chain with constrained outputs — it's a Topic 59 pattern plus a schema, one evening of work.
Topic 81: AI SaaS Workflows — the business shapes
The engineering is settled; now the commercial container. Four shapes: AI-native products (the AI is the product), AI features inside existing SaaS (the largest category by revenue), vertical AI (one industry's workflow, owned deeply), and internal tooling/services (unsold, enormously valuable). Two design forces govern all four:
Force 1 — unit economics as a design constraint (Topic 77, now existential): pricing must survive your cost curve. Per-seat pricing + heavy usage = margin erosion; usage-based or credit pricing tracks COGS honestly; and the margin-protection stack — routing, caching, caps, batch — isn't optimization, it's what makes the pricing page true. Cost-per-customer metering (Topic 70) from day one, because in AI SaaS your heaviest user can be your least profitable.
Force 2 — the moat question, deserving the honest treatment: "it's just a wrapper" is the standard sneer, and the standard rebuttal is real: the model was never the moat — models commoditize on Topic 11's 6–12-month lag, for everyone, symmetrically. Durable advantage in AI SaaS comes from exactly four places: the data flywheel (next diagram), workflow depth (integrations, permissions, the fifty unglamorous edge cases that make switching painful), distribution, and trust/proprietary context (being the system that already holds the customer's data). Here's the first one — the compounding engine this entire course quietly built the parts for:
Decode the flywheel with course vocabulary and notice you own every gear: usage → data is Topic 74's feedback capture and Topic 79's accept/reject exhaust; data → improve is Topic 75's regression cases plus Topics 16/28's preference pairs feeding DPO/KTO; improve → better product is the eval-gated ship loop; and competitors starting today can copy your UI in a week but cannot copy your accumulated eval set and preference corpus — the one asset that only time-in-market produces. Corollary for small teams: vertical beats horizontal — a niche workflow's flywheel spins faster because the data is concentrated and the workflow depth is reachable; and the validated build sequence is concierge → semi-automated → productized: do the workflow manually for five customers first (you're building the golden set and discovering the edge cases — Module 2's data craft as market research), automate second. Premium-tier idea you're uniquely equipped to offer: per-customer fine-tunes via Topic 22's multi-adapter serving — a 50 MB adapter per client on one shared base is a feature competitors without Module 3 can't quote.
Summary
AI SaaS = engineering wrapped in unit economics (price to survive your cost curve, meter per customer) and moated by flywheel, workflow depth, distribution, and trust — never by the model. Go vertical, concierge first, and let the eval/preference corpus compound.
Mental model
The model is the espresso machine — every café on the street has the same one. The moat is the regulars, the barista who knows their orders (data flywheel), and the fact that the office upstairs runs a tab (workflow depth).
Mistakes to avoid
defending "wrapper" accusations by hiding the model instead of building the flywheel — the accusation is only true of products that don't accumulate data advantages; and pricing per-seat with unmetered usage, then discovering your best customer is your biggest loss.
Exercise
For a SaaS idea you'd actually build: write the flywheel concretely (what data does usage produce? what does it improve, mechanically — eval cases? DPO pairs? retrieval corpus?), name the moat among the four, and draft the pricing model with a Topic 77 margin check at 10× expected usage. One page: the difference between "an idea" and "a business shape."
Topic 82: AI Coding Workflows — using the tools like a professional
Module 8 explained why coding models got good; this topic is the user side — the workflow discipline that separates engineers who 10× with these tools from those who generate plausible bugs faster. The tool tiers first: autocomplete (FIM, ambient), chat-assisted (explain/draft/debug in conversation), agentic (Claude Code-class: reads the repo, edits files, runs commands — Topic 58's loop pointed at your codebase), and background agents (issue in, PR out, review async). The disciplines that matter run across all tiers:
1. Spec first — the prompt is a spec (Topic 54, applied to engineering). "Fix the auth bug" produces guesses; "login fails for emails with +; the validator in auth/validate.ts rejects them; fix the regex, add test cases for plus-addressing, don't touch the session logic" produces the fix. And the repo-level version: rules files (CLAUDE.md and kin) are system prompts for your codebase (Topic 55) — conventions, commands, architecture notes, forbidden patterns — the highest-leverage file-per-character in a modern repo.
2. Verification first — tests are the contract. Topic 66's whole thesis, wielded: the strongest agentic pattern going is write (or generate and review) the tests, then let the agent iterate until green — converting Topic 58's compounding-failure regime into retry-until-verified. Your role shifts from writing the solution to specifying the checkable success condition.
3. Decompose and review. Big-bang refactors are the compounding trap (0.95³⁰ again) — small scoped diffs, sequentially. And the line that must never move: read every diff before it merges. AI-generated code is confident in exactly the Topic 21 way — fluent, plausible, occasionally subtly wrong — and "AI wrote it" will never be an accepted root cause. (AI review as a second pass is excellent; as a replacement for your reading, it's the blind leading the blind.)
Where the tools are strong today: boilerplate, tests, migrations, glue, unfamiliar-API navigation, "make it pass." Where your judgment stays load-bearing: novel architecture, subtle concurrency, performance-critical paths, security boundaries. The honest skill shift: value migrates from typing code to specifying, decomposing, reviewing, and verifying it — which is to say, the senior-engineer skills appreciated and the junior mechanical ones depreciated. Plan your own development accordingly.
Summary
Professional AI coding = spec-quality prompts + rules files as repo constitutions + test-anchored verification loops + small diffs + non-negotiable human review — with your leverage moving up the stack from writing to specifying and verifying.
Mental model
You've been promoted to tech lead of a tireless, fast, occasionally-overconfident team: your output is now specs, task decomposition, and review quality — and the team is exactly as good as your tickets.
Mistakes to avoid
vague tasks to agentic tools, then blaming the model for guessing (garbage spec, garbage diff — the amnesiac-contractor standard applies to code tasks doubly); and merging unread diffs because the tests pass — tests verify what tests cover, and you specified the tests.
Exercise
Add a rules file to a real repo of yours: conventions, build/test commands, architecture in five lines, three "never do" rules. Then run the same non-trivial task through your coding agent with and without it, and diff the diffs. The delta is Topic 55's lesson, measured on your own code.
Topic 83: AI Orchestration Systems — Module 7 at production scale
One workflow is a script; a company's AI runs hundreds of workflows, thousands of executions a day, for weeks-long lifecycles. Orchestration is the layer that makes that reliable, and it's Module 7's concepts plus classic distributed-systems hygiene:
Durable execution — the keystone concept: long-running workflows (a document pipeline with a human-approval step might live for days) must survive process crashes, deploys, and restarts, resuming from their last completed step. Recognize it: Topic 26's checkpointing, generalized from training state to workflow state — every step's inputs/outputs persisted, the workflow replayable to its frontier. Temporal/Inngest-class engines productize this; a job table in Postgres plus idempotent steps is the artisanal version; the property, not the vendor, is the requirement.
Around that keystone: queues and schedulers (event-driven beats cron; backpressure when the model gateway rate-limits), per-step hardening (retries with backoff, timeouts, circuit breakers — an LLM step is a flaky network dependency and gets treated like one), the model gateway underneath everything (Topic 70's shell as shared infrastructure — one place for keys, routing, fallbacks, and fleet-wide rate-limit governance, so forty workflows don't independently DDoS your provider quota), and tracing (Topic 60's lesson industrialized: every execution a browsable trace of steps, prompts, tool calls, costs — the difference between debugging and archaeology).
Two orchestration-specific insights worth the price of the topic: humans are a service. At scale, human-in-the-loop (Topics 57/80) means review queues as first-class workflow steps — with assignment, SLAs, and escalation — and once modeled that way, humans slot into your architecture as another dependency with latency (hours) and error rates (Topic 74's 70% agreement), plannable like any other. And version everything together: a workflow's behavior = code × prompts × model versions × eval baselines; deploy them as one versioned unit (Topic 75's loop at fleet scale), or live the special misery of "the prompt changed but the eval baseline didn't."
Summary
Orchestration = durable execution (checkpointed workflow state), hardened steps, shared model gateway, full tracing, humans-as-a-service review queues, and prompts/models/evals versioned as one deployable unit. Concepts over frameworks, as always.
Mental model
Air-traffic control for your workflows: every flight (execution) tracked, resumable after a radar blip (durability), routed through shared corridors with flow control (gateway), with human controllers in the loop as scheduled capacity — not heroic interventions.
Mistakes to avoid
in-memory workflow state ("it's just a script") — the first deploy mid-execution teaches durability the expensive way; and letting each workflow call providers directly, discovering rate limits as a fleet-wide outage instead of a gateway policy.
Exercise
Take your Topic 80 automation design and make it durable on paper: define each step's persisted input/output, what happens if the process dies between steps 3 and 4, the retry/timeout policy per step, and where the human queue sits with its SLA. If you can answer "kill -9 at any moment — what happens?" for every arrow in your diagram, you've understood orchestration.
Topic 84: AI Product Thinking — the judgment layer
The final topic of the course — no mechanisms, only judgment: what to build, and the meta-skills that survive every model release.
Build where fallibility is affordable. Models are probabilistic (Lesson 1, never repealed). Products that thrive absorb errors structurally: drafts a human sends, suggestions a user accepts, triage a queue backstops, search with citations. Products that require determinism — payments, compliance math, anything where one error is catastrophic — want rules and SQL, with AI at most proposing. The maturity test of an AI product isn't its accuracy; it's what happens on the wrong answer — Topic 21's layered-defense instinct, elevated to product strategy. Twin principle: build where verification is cheap (Topic 58's asymmetry as a market map — it predicted coding's boom; apply it forward: which domains have cheap checkers? those are next).
Build on the trendline, not the snapshot. Capabilities improve relentlessly and costs fall roughly an order of magnitude every year or two for equivalent quality. Two corollaries in tension, both true: "barely works today" means "works next year" — the too-early product beats the impossible one — and thin layers get absorbed: if your product is one prompt-shaped gap in the platform's capabilities, the next model release is your extinction event. The test: does your product get stronger when models improve (flywheel, workflow depth, integrations — the tide lifts you) or redundant (you were patching a capability gap — the tide covers you)? Build the former, always.
Respect the demo-to-product gap. A demo is p50; a product is p99 (Topic 76's percentile lesson as strategy). The last 10% of reliability is 90% of the work, and Module 10's eval discipline is the only vehicle that crosses it — which is why "we have a great demo" and "we have a product" are separated by precisely the golden set, the regression battery, and six months of the Topic 75 loop.
And the most senior judgment of all: knowing when not to use AI. If a regex, a SQL query, or three business rules solve it — the LLM is expensive, slow, nondeterministic tech debt wearing a trendy jacket. The engineer who says "this doesn't need a model" earns more trust than the one who models everything; restraint is the credential.
Summary
Build where errors are absorbable and verification is cheap; build for where models are going, in shapes the tide strengthens rather than absorbs; cross the demo–product gap with evals; and keep the confidence to not use AI at all. Ship small, instrument everything, let the flywheel compound.
Mental model
Surfing, not swimming: the wave (model progress) supplies the power — your judgment is position (which problems), timing (trendline, not snapshot), and balance (fallibility-absorbing design). Paddling against the wave, or standing where it breaks, are both losing strategies regardless of skill.
Mistakes to avoid
building your product's core value on patching a current model weakness (you are short the entire AI industry's R&D); and equating a wowed demo audience with product-market fit — the p99 grind, not the demo, is the product.
Exercise · the course's final one
Write your one-page thesis: a problem you're positioned to solve with AI. State — where errors get absorbed; what's verifiable; why the trendline strengthens it; the flywheel's gears; the moat among Topic 81's four; the three numbers (quality/speed/cost, Module 10) you'd track from day one; and what part deliberately doesn't use AI. Then build the concierge version. That page plus this course is everything you need.
Curriculum Complete
Look back at the distance: you started with "an LLM is autocomplete" and you're ending with unit-economics frontiers, durable orchestration, and product theses. The arc, in one breath: tokens → attention → parameters → data as the spec → LoRA and quantization → the bandwidth law → serving economics → retrieval and memory → tools, loops, and guards → the model zoo → the five deployment addresses → measurement as discipline → judgment. Eighty-four topics, and the deepest pattern across all of them: the same dozen ideas kept returning wearing new clothes — compression, verification asymmetry, cheap-wide-then-expensive-precise, everything-is-prompt-assembly, measure-don't-vibe. You didn't memorize a field; you learned its grammar. That was the deal made in Lesson 1.
What now: the exercises you skipped are the course you haven't finished — the Lesson 6 fine-tune, the Topic 75 eval suite, and the Topic 78 chatbot assembly are the three that convert knowledge into portfolio.
Next: Module 12 (Bonus) — The Career Layer. The eleven modules above make you an engineer. The bonus module makes that legible to the people who hire and fund. It is the module most curricula are too polite to include, and it is blunter than the others, because career advice that hedges is worthless.