Skip to content

Week 12 — Ship, measure, publish

Why this week: finish properly. A finished product that people use, with honest numbers, beats three half-built ones, for employers and for you. This week adds no features. It checks that nothing broke, makes the repo readable by a stranger in five minutes, and turns twelve weeks of work into a case study, a CV and applications.

By Friday you have: the full eval suite and red-team suite re-run against the live setup, with any regressions fixed; a README a stranger can follow, with an architecture diagram and the numbers at the top; a public case study; an updated CV and GitHub profile; and your final round of applications sent.

Main reading: How to Write Better with The Why, What, How Framework by Eugene Yan, and Make a README.

Concepts

Regression. Something that used to work and now doesn't. Weeks 9 to 11 changed a lot underneath the agent: a new way of calling Claude, possibly a different model, caching, routing, a fallback model, a time limit. Each could have moved recall or let an old attack back in. Your week 7 and week 8 suites exist to catch exactly this.

Who a README is for. Not you. A hiring manager gives it a minute, and an engineer gives it five. In that time they want to know what it does, whether it works (a live link, numbers), how it's built (a diagram), and whether they could run it. Everything else is detail further down.

Architecture diagram. A box-and-arrow picture of the parts and what flows between them. Mermaid lets you write it as text in the README, and GitHub draws it, so it stays in git and changes alongside the code.

Case study: why, what, how. Eugene Yan's structure. Why: the problem, in the reader's terms (scams cost people money, and many check messages alone on a phone). What: what good looks like, as measures and limits (recall on real scams, false alarms, cost per check, time to answer, no personal data stored). How: the design and the decisions, with the numbers that justified each one. Most engineers start with How; readers need Why first.

A number with its denominator. "Recall 0.91" means little. "Caught 91 of 100 scams in a labelled set I built from public advisories, with 6 false alarms on 50 genuine messages" can be checked and defended. Every number in the case study gets its denominator and its source.

Monday — Re-run everything

Learn (30 min): re-read your week 7 evals/thresholds.json, your week 8 red-team report, and the promptfoo red team quickstart if you need a reminder of the commands.

Build:

  1. Run the full eval suite against the code as deployed (API key, current model, caching and routing on), exactly as CI does: scripts/run_eval.py and your evals/ checks and judge. Estimate the cost first from week 11's cost per check; the judge adds its own calls.

  2. Run the week 8 red-team suite from promptfooconfig.yaml the same way. Add the attacks that only became possible after week 9: a 2,000-character message stuffed with instructions; a message designed to look "easy" to is_easy (no link, no digits) that is still a scam; a message that tries to get the explanation step to repeat a phone number.

  3. Your turn: compare with the numbers you recorded in weeks 7, 8 and 11. For each one that got worse, find the commit that caused it (git log, and a re-run on the commit before), fix it, and add a test or eval case so it can't come back quietly. If nothing got worse, write that down too, with the numbers: it's evidence your CI gate works.

  4. Check the live site the way a stranger would: on your phone, on mobile data, with the five test messages from week 10.

Done when: a table in your README shows this week's eval and red-team numbers next to weeks 7, 8 and 11, and every regression has a fix and a test.

Stuck?
  • Recall dropped after week 11: the usual suspect is routing. Re-run cost_latency.py in cached mode (no routing) to see whether it's routing or something else.
  • The judge's scores moved but your labels didn't: the judge model may have changed in week 9. Re-measure its agreement with your labels before trusting it.
  • A red-team attack now gets through the explanation step: the week 10 EXPLAIN_SYSTEM prompt never went through week 8's guardrails. Run the explanation through your output guardrail too.

Tuesday — A README for strangers

Learn (30 min): Make a README, and GitHub's page on creating diagrams.

Build:

  1. Rewrite README.md from the top, in this order:

    • One sentence on what it is, and who it's for.
    • The live links: the page, and the API's /docs.
    • Three headline numbers, each with its denominator: recall and false alarms, cost per check, p95 time.
    • A screenshot or short GIF of a check on a phone.
    • "How it works": the diagram below, and five lines of explanation.
    • "Run it yourself": the exact commands, from git clone to a working check, with .env values described but never shown.
    • "How it's tested": unit tests, the eval suite and its CI gate, the red-team suite.
    • "Limits and privacy": what it can't do, what's stored (stripped text), what isn't, and that it's not a guarantee.
    • Links to the case study, the cost report and your three write-ups.

  2. The architecture diagram. Paste this into the README and adjust it to match what you actually built:

    flowchart LR
    person["Person with a suspicious message"] --> page["React page<br/>GitHub Pages"]
    page -->|"POST /check/stream"| api["FastAPI on Cloud Run"]
    api --> agent["Agent loop<br/>tools, retrieval, reviewer"]
    agent -->|"call_claude()"| claude["Claude API<br/>Haiku, Sonnet as fallback"]
    agent --> tools["Link expander, domain age,<br/>blocklist, scam-report graph"]
    agent --> patterns[("Scam patterns<br/>Postgres + pgvector")]
    api -->|"stripped text only"| history[("Check history<br/>Neon Postgres")]
    api -->|"tokens, cost, time"| traces["Langfuse traces"]

    GitHub draws it when you view the README. Square brackets make a box, [(...)] a database cylinder, and text between pipes labels an arrow. The Mermaid flowchart syntax page has the rest.

  3. Your turn: the stranger test. Ask one developer friend, or a fresh Claude session given only your repo link, to follow "Run it yourself" from a clean machine. Every place they get stuck is a bug in the README. Fix each one.

Done when: someone other than you has run the project from the README alone, and the diagram on GitHub matches the code.

Wednesday — The case study

Learn (45 min): read Eugene Yan's why, what, how again, slowly, then one write-up you admire from the course reading, such as Writing effective tools for agents, and notice how it uses numbers.

Build:

  1. Write docs/case-study.md, about 1,500 words, with these headings:

    # Building an anti-scam message checker: what worked, with numbers

## Why
The problem in plain words: who receives scam messages, what it costs them, why a
second opinion on a phone matters. One or two sourced facts from public advisories.

## What good looks like
Measures: recall on scams, false alarms on genuine messages, cost per check, p95 time.
Limits: no personal data stored, no guarantee, works on a phone, survives an outage.

## How it works
The diagram. The agent loop and its tools. Retrieval over known scam patterns.
The graph signal from linked scam reports.

## How I know it works
The keyword baseline versus the checker (week 1). Error analysis on 100 labelled
messages and what the top failure was (week 5). The judge and its agreement with my
labels, and the CI gate (week 7). The red-team results (week 8 and this week).
What ten real users struggled with, and what I changed (week 10).

## What it costs and how fast it is
The before-and-after table from the cost report (week 11).

## What I'd do next
Three things, each with the number it should move.

  2. Every number gets its denominator and the week it came from. Every claim you can't back with a number gets cut or softened.

  3. Where your fraud and trust & safety background changed a decision, such as how you ranked failure types by harm, or which attacks you thought of first, say so plainly. It's the part of the story nobody else applying has.

  4. Your turn: publish it where people read, such as your blog, LinkedIn or a dev.to post, with a link back to the repo and the live page. Then send the link to Claude with the Friday check-in questions and ask which three claims an interviewer would push on hardest.

Done when: the case study is public, and every number in it can be traced to a file in the repo.

Thursday — CV, GitHub profile, applications

Learn (30 min): read your notes/interview-answers.md from start to finish. It holds twelve weeks of answers to interviewers' questions, and it's the raw material for today.

Build:

  1. Rewrite your CV's top third around what you built, not the course. One line on the product, with the live link; three bullets with your numbers, such as "built an agent-based scam checker that caught 91 of 100 labelled scams at about US$0.001 per check" (with your real figures); then your ten years of fraud and trust & safety work, framed as the domain knowledge behind the product's design.

  2. Your GitHub profile. Pin the scam-checker repo, with a one-line description and the live link, and write a short profile README that links the case study and your three write-ups.

  3. The final applications push. Go back through the roles you listed at the start of the course: small-team AI engineer roles, AI evaluation roles, and anti-fraud and anti-scam engineering teams. For each, tailor the CV's three bullets to the posting, and in the cover note link the one artefact that matches it best: the case study for AI engineer roles, the evaluation write-up for evaluation roles, the PySpark and graph work for anti-fraud teams.

  4. Your turn: send at least five applications today, and log each one in notes/progress.md with the date and the artefact you linked.

Done when: the CV, profile and pinned repo all lead with the product, and at least five applications are sent and logged.

Friday — Demo day, and a look back

Demo day (9:00–12:00):

  1. Record a 5-minute walkthrough you'd be happy to send an employer: the problem in one sentence, a live check on a phone, the diagram, the three headline numbers, and one hard decision with the evidence behind it. Keep it; it's a portfolio piece.
  2. Write five lines on what you learned this week, and five on the whole course.
  3. Send both to Claude with the Friday check-in. Ask, too: "Based on my progress log and check-ins, which week should I repeat in week 13?"

Friday afternoon: the cloud side track, or a break. You've earned it.

Deliverable checklist

  • Eval and red-team suites re-run, compared with weeks 7, 8 and 11, regressions fixed with tests
  • A README with live links, headline numbers, a Mermaid diagram and run-it-yourself steps that a stranger has followed
  • A public case study in why, what, how order, every number with its denominator
  • CV and GitHub profile leading with the product
  • At least five applications sent and logged
  • Demo video, five lines, and the Friday check-in sent

If you have time left: record a 60-second version of the demo for the top of the README, and caption it. Most people who open the repo will watch that and nothing else.