Building with LLMs

Everything around the model, which is where the engineering actually is

About this course

Treating the model as fixed: retrieval, tool use, agent loops, evaluation, guardrails, cost and observability — the system that surrounds an LLM.

What you will be able to do

  • Design a retrieval pipeline and diagnose which stage failed
  • Expose tools to a model as a typed interface, and handle failure
  • Build an agent loop with state, stopping conditions and error recovery
  • Construct an eval harness, including LLM-as-judge, and audit the judge
  • Operate the system — guardrails, latency, cost and observability

What this course covers

  • Frozen weights vs. controllable levers
  • Context window as a fixed budget with competing claimants
  • History growth and the cliff-edge failure mode
  • Pricing the levers in a common currency
  • Contract length and compound conformance failure
  • Capstone case and course roadmap
  • The item set: questions, slices, and provenance
  • The aggregate score: true but useless
  • The harness is a choice: four honest sets, three answers
  • Two blindnesses: size and aggregation
  • Invariant checks when no labels exist
  • Capstone: every evaluation number is a joint fact
  • The fit step: slot, budget, and selection
  • Stop-at-first-overflow fit policy and its consequences
  • Skip-and-continue as an alternative fit policy
  • Sufficient, grounded, and correct as distinct properties
  • Conditioning cost is a per-request expenditure
  • Worked diagnostic case: silent selection failure
  • Logging assembled context as a system requirement
  • Two currencies: cost units vs. operations
  • The bill: repricing the context-window table
  • The levers: two rankings of the same four moves
  • The tail: P99 as an order statistic
  • The trade: fast-path optimisation worsens the tail
  • Operations-to-latency transfer theorem and reliability break-even
  • LLM-as-judge setup and headline agreement
  • The constant judge and the base-rate illusion
  • Chance-corrected agreement coefficient
  • Slice analysis and Simpson's-paradox-style reversal
  • Swap consistency audit for pairwise judgements
  • Repeat-consistency audit and stability vs. correctness
  • Reference fallibility and the limits of any agreement metric
  • The four-artifact chain and nesting structure
  • Five failure causes taxonomy
  • The grounding blind spot on live traffic
  • Retrieval metric gap and the corpus-gap cell
  • Pricing repair levers against the request
  • End-to-end score as a lossy projection
  • Agent as a State Machine
  • Reachability and Decidability
  • Termination Arguments
  • Quadratic Cost of Carrying State Forward
  • Transcript Compaction: Trade-offs
  • Fixed Points and State Saturation
  • Four Stopping Conditions Compared
  • Observability as a pre-incident decision
  • The world vs. the trace: projection and answerability
  • Which fields, not how many: the schema lattice
  • The field that answers nothing: the inert-field scan
  • Sampling and outcome-conditioned retention
  • Trace-level answerability and the forward-only fix
  • Embeddings and cosine similarity defined
  • Three similarity formulas compared
  • Boilerplate inversion (concept two)
  • Negation blindness (concept three)
  • Threshold non-transitivity and non-portability (concept four)
  • Capstone case: support-assistant failure modes
  • Five exact limits of scalar similarity — closing summary
  • Provenance and the five vocabulary terms
  • The channel: provenance dies at assembly
  • Placement as projection: the three-guard comparison
  • What a placement can possibly decide: the block argument
  • Carrying provenance as the engineering conclusion
  • The price of saying no: rejection, constraint, and the route-around
  • Monday checklist: four operational questions
  • The Assembly Problem: Four Citation Checks
  • The Walk: Fourteen-Stage Request Pipeline
  • The Twins: Coincidence vs. Inheritance in Shared Renderings
  • Scope of Forbidden Lists and Guard Validation
  • The Ledger: What This Course Could Not Do
  • Course Spine: Shapes vs. Digits as the Takeaway
  • Multiplicative composition of trajectory reliability
  • Why retrying works only when failures are independent
  • Recovery edges and the three-state reliability chain
  • Five-policy, three-fault enumeration of a concrete loop
  • Correctness conditions distinct from stopping conditions
  • The retry-then-reroute counterweight anti-pattern
  • Exhaustive search as the definition of correctness
  • Recall at K as an exact fraction with a denominator
  • The N-probe dial: cost vs. recall trade-off
  • The miss: a named neighbour in an unopened cell
  • The price: diminishing returns per click of the dial
  • Graph-based search and the shared failure mechanism

Syllabus

  1. The Fixed Model, and Representation

    1. L1 · The Model Is Fixed — What That Leaves You
    2. L2 · Embeddings — What a Vector Encodes, and What It Does Not
    3. L3 · Vector Search — Exact, Approximate, and the Price of Speed

  2. Retrieval

    1. L5 · RAG — Retrieval as a Conditioning Step
    2. L6 · When RAG Fails, and How You Know Which Part Broke

  3. Tools and Agents

    1. L8 · Agents I — A Loop With State and a Stopping Condition
    2. L9 · Agents II — Compounding Error and Recovery

  4. Evaluation and Guardrails

    1. L10 · The Eval Harness — Building the Test Set You Do Not Have
    2. L11 · LLM-as-Judge — Auditing the Thing That Grades You
    3. L12 · Guardrails as a System Property

  5. Running It in Production

    1. L13 · The Bill and the p99
    2. L14 · Deployment and Observability
    3. L15 · The Whole System, End to End