Skip to content

Week 5 — Evals I: error analysis

Why this week: you have an agent, tools and retrieval, and a 30-message comparison from week 1. That's not enough to say where the checker fails or whether a change helps. This week you build a proper test set, run the checker on all of it, read every mistake, and turn what you find into a ranked list of failure types. It's the core skill for evaluation roles, and the one where your fraud background counts most: you already know what a costly miss looks like.

By Friday you have: 100 messages you labelled yourself, including tricky genuine bank and government messages; a script that runs the agent on all 100, four at a time, and survives a crash; a viewer for reading results one at a time; a ranked list of failure categories; grades for the agent's tool use; precision and recall with honest error bars; and one fixed failure category, with before-and-after numbers.

Main reading: the AI Evals FAQ by Hamel Husain and Shreya Shankar, the error-analysis and annotation questions. Each day lists which ones.

Concepts

Labelled test set. A fixed set of inputs with the right answer written down by a person who knows the domain, here you. It's the ruler you measure every change against, so it must stay the same from run to run and be labelled before you see what the checker says.

Trace. The full record of one run: the message, every tool the agent called and what came back, and the final answer. Your runs/ files from week 3 are traces. Error analysis means reading them, not just counting verdicts.

Error analysis. Reading the checker's outputs and writing down what went wrong, before deciding what to measure or fix. Precision and recall tell you how often it's wrong; error analysis tells you how and why. Without it you fix whatever you happen to notice first.

Open coding. The first pass: for each failure, a short free-text note in your own words, such as "flagged a real DBS OTP message because it mentions a transfer". No categories yet. The term comes from qualitative research, the same idea as coding interview transcripts. Note the first thing that went wrong in each trace; later problems often follow from it.

Axial coding. The second pass: group the notes into a small set of categories (a "failure taxonomy"), such as "genuine bank message flagged" or "invented phone number". Then count them.

Frequency × harm. Not every failure costs the same. A missed scam can cost someone their savings; a false alarm costs some trust. You give each category a harm score and rank by count × harm. It's the cost-weighted confusion matrix you've used in fraud work, applied to failure types.

Concurrency and threads. Most of a call_claude() call is waiting for a reply. A thread is a separate line of work inside one Python program; with four threads, four calls wait at the same time, so 100 messages take about a quarter of the time. ThreadPoolExecutor hands out the work to a fixed number of threads (a "pool") and gives you each result as it finishes.

Why 4 and not 50. Every call counts against your subscription's usage limit, whether it runs alone or alongside others. Fifty at once would hit the limit within a minute; every call would then fail, retry and fail again, and you'd lock yourself out of Claude Code for a while too. Each call also starts its own claude program on your laptop. Four is fast enough and gentle on both.

Resumable batch job. A long run will be interrupted: a usage limit, a closed laptop, a bug in message 63. Writing each result to the CSV the moment it arrives, and skipping finished rows on the next start, means you never lose work or pay twice for the same call.

Trajectory grading. Checking how the agent reached its answer, not only the answer: did it call the right tools, and did it stop in a sensible number of steps? A right answer reached by skipping the pattern lookup is luck, and it won't cite anything.

Confidence intervals at n = 100. With about 50 scams, one more caught scam moves recall by two points. A 95% interval for recall of 0.80 on 50 scams runs from about 0.67 to 0.89. So "0.80 versus 0.84" is probably noise. The Wilson interval is a standard way to compute this that behaves well near 0 and 1, where the textbook formula doesn't.

Monday — Label 100 messages

Learn (75 min):

Build:

  1. Write your labelling rules first, in the README, before any labels. A few lines is enough:

    • Scam: tries to get money, credentials, codes or an app install under false pretences.
    • Not a scam: genuine, even if it's marketing, spam or annoying. Spam isn't fraud.
    • scam_type: the knowledge base id it matches (job_scam, bank_phishing and so on), other for a scam your knowledge base doesn't cover, none for genuine messages.
    • When unsure: decide, and write why in notes. A label you can't decide on is useful information; it often marks a message the checker will also struggle with.

  2. Gather 100 messages. Aim for about half scams and half genuine, with at least 15 tricky genuine ones:

    Where from How many Notes
    Your week 1 data/eval30.csv and data/rag_queries.csv 20–30 Already written; relabel with a scam_type
    The SMS Spam Collection from week 1 up to 10 Only a few: it's UK messages from 2012
    Public advisories (ScamShield, police news) 20–30 Write messages based on the examples they describe, in your own words
    Genuine messages on your own phone 15–25 Rewrite them: change every name, number, amount and account detail
    Messages you write the rest Fill gaps: each pattern twice, plus scams your knowledge base doesn't cover

    Tricky genuine messages are the most valuable rows in the file. Think of: a real bank OTP that mentions a transfer amount; a card transaction alert; an SMS from the gov.sg sender ID; a real courier saying a parcel arrives today; a clinic asking you to reply to confirm; a friend who really did change their number, with no request attached.

    Never real people's private data

    Don't paste a real message from a friend, relative or stranger as it is, and never include a real person's name, phone number, NRIC, address or account number. Change every detail so no one could be identified. Your repo is public, and in week 9 these messages may pass through a cloud service. This is the same rule as working with customer data at your old job.

  3. Save them as data/labelled100.csv with five columns: id, text, is_scam, scam_type, notes. The id is a number you never change, so results can be matched back to labels even if you reorder the file. The first rows of the rehearsal file:

    id,text,is_scam,scam_type,notes
1,"Hi mum, I dropped my phone, this is my new number. Can you help me pay a bill urgently?",True,fake_friend_call,written from ScamShield advisory
2,"SingPost: your parcel is on hold due to unpaid fees. Pay $1.99 at singpost-redelivery.com",True,parcel_delivery_phishing,written from ScamShield advisory
3,"Part-time job: earn $300/day liking videos. Contact our HR on WhatsApp.",True,job_scam,week 1 set
6,"DBS: Your OTP for the PayLah! transfer of $25.00 is 482913. Never share your OTP with anyone. If you did not request this, call the number on the back of your card.",False,none,tricky: genuine bank OTP message with a money amount
9,"Hi it's Mei, I've changed my number, this is the new one. See you at Ah Ma's birthday dinner on Sat!",False,none,"tricky: 'changed my number' but no request"
10,"Your parcel from Shopee will be delivered today between 2pm and 6pm. Track it in the Shopee app.",False,none,"tricky: parcel message, no link, no fee"

    Put quotes around any text containing a comma. Write none in lower case: pandas reads None and NA as missing values.

  4. Check the file before spending any calls on it, with scripts/check_labels.py:

    """Sanity-check data/labelled100.csv before you spend calls on it."""

import pandas as pd

from scam_checker.retrieval import load_patterns

df = pd.read_csv("data/labelled100.csv")
known = {p.pattern_id for p in load_patterns()} | {"other", "none"}

problems = []
if list(df.columns) != ["id", "text", "is_scam", "scam_type", "notes"]:
    problems.append(f"columns are {list(df.columns)}")
if df["id"].duplicated().any():
    problems.append(f"duplicate ids: {df.loc[df['id'].duplicated(), 'id'].tolist()}")
if df["text"].duplicated().any():
    problems.append("some messages appear twice")
unknown = set(df["scam_type"]) - known
if unknown:
    problems.append(f"scam_type values not in the knowledge base: {unknown}")
genuine_with_type = df[~df["is_scam"] & (df["scam_type"] != "none")]
if len(genuine_with_type):
    problems.append(
        f"genuine messages with a scam_type: {genuine_with_type['id'].tolist()}"
    )
scam_without_type = df[df["is_scam"] & (df["scam_type"] == "none")]
if len(scam_without_type):
    problems.append(f"scams with scam_type 'none': {scam_without_type['id'].tolist()}")

print(
    f"{len(df)} messages: {df['is_scam'].sum()} scams, {(~df['is_scam']).sum()} genuine"
)
print(df["scam_type"].value_counts().to_string())
print("\n".join(problems) or "No problems found.")

    uv run python scripts/check_labels.py

    Fix everything it reports, and look at the counts per type: any pattern with fewer than three messages needs more.

  5. Your turn: check your own consistency. Tomorrow morning, before anything else, hide the is_scam and scam_type columns, relabel 20 random rows, and compare. If you disagree with yourself on more than two or three, your rules in step 1 aren't clear enough; sharpen them and fix the labels. Write the agreement rate in the README.

Done when: check_labels.py prints No problems found. for 100 rows, with at least 15 tricky genuine messages marked in notes, and the file is committed.

Stuck?
  • KeyError: 'scam_type' or the columns check fails: a stray space in the header row, or the file was saved from Excel with a different separator. Open it in a text editor and check the first line.
  • is_scam comes back as text, so ~df["is_scam"] fails: some rows say TRUE, yes or 1. Use exactly True and False.
  • scam_type values not in the knowledge base: {nan}: an empty scam_type, or one written as None. Use none.
  • Running out of ideas for genuine messages: read your own SMS inbox for a week's worth of bank, delivery, clinic and government messages, and rewrite each with made-up details.

Tuesday — Run the checker on all 100, four at a time

Learn (45 min):

  • Python's concurrent.futures docs, the ThreadPoolExecutor section and its example (25 min). Ignore ProcessPoolExecutor
  • Python's csv docs, DictWriter only (10 min)
  • What is a trace? (5 min)
  • Re-read the "Concurrency and threads" and "Why 4 and not 50" concepts above

Build:

  1. Work out why you need this. One message through the agent takes about 10–20 seconds (two or three call_claude() calls). One at a time, 100 messages take around 25 minutes, and a single crash at message 80 throws it all away. The script below runs four at a time and writes each result to disk as it arrives.

  2. If your week 3 code saves runs by appending to one shared file in runs/, make it safe for four threads at once. Two threads appending a long line at the same moment can, rarely, mix the lines together. A lock lets one thread write at a time:

    import threading

RUNS_LOCK = threading.Lock()  # one thread writes to runs/ at a time

        with RUNS_LOCK, open("runs/runs.jsonl", "a") as f:
        f.write(json.dumps(line) + "\n")

    Change the file name and the line variable to whatever your save function uses. If it writes one file per run instead, you don't need this.

  3. The runner, scripts/run_eval.py:

    """Run the agent on every labelled message, a few at a time, saving as it goes.

Usage: uv run python scripts/run_eval.py [results file]
If it stops halfway, run the same command again: finished rows are skipped.
"""

import csv
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import date
from pathlib import Path

import pandas as pd

from scam_checker.agent import run_agent

WORKERS = 4  # calls running at once; more just hits your usage limit sooner
FIELDS = [
    "id",
    "verdict",
    "matched_pattern",
    "red_flags",
    "advice",
    "tools",
    "n_steps",
    "seconds",
]

labels = pd.read_csv("data/labelled100.csv")
default_path = f"evals/results_{date.today().isoformat()}.csv"
out_path = Path(sys.argv[1] if len(sys.argv) > 1 else default_path)
out_path.parent.mkdir(exist_ok=True)

done = set(pd.read_csv(out_path)["id"]) if out_path.exists() else set()
todo = labels[~labels["id"].isin(done)]
print(f"{out_path}: {len(done)} already done, {len(todo)} to run")


def check(row_id: int, text: str) -> dict:
    """Run the agent on one message and flatten the answer into one CSV row."""
    ...  # your turn, step 4


write_header = not out_path.exists()
with open(out_path, "a", newline="") as f, ThreadPoolExecutor(WORKERS) as pool:
    writer = csv.DictWriter(f, fieldnames=FIELDS)
    if write_header:
        writer.writeheader()
    futures = [pool.submit(check, row.id, row.text) for row in todo.itertuples()]
    try:
        for i, future in enumerate(as_completed(futures), start=1):
            try:
                row = future.result()
            # One bad message shouldn't stop the run, whatever went wrong.
            except Exception as exc:  # noqa: BLE001
                print(f"{i}/{len(todo)} failed: {exc!r}")
                continue
            writer.writerow(row)
            f.flush()  # on disk now, so a crash can't lose it
            print(
                f"{i}/{len(todo)} id {row['id']}: {row['verdict']} ({row['seconds']}s)"
            )
    except KeyboardInterrupt:
        print("Stopping. Run the same command again to carry on.")
        pool.shutdown(cancel_futures=True)

    How it works, top to bottom:

    • Resume: if the results file exists, read the ids already in it and only run the rest.
    • pool.submit(check, ...) queues every message at once; the pool runs four at a time. Each call returns a future, a placeholder for a result that isn't ready yet.
    • as_completed(futures) hands back each future as soon as it finishes, in whatever order. That's why the progress lines show ids out of order.
    • Only the main loop writes to the CSV, one row at a time, so the four threads never write to the file at once. f.flush() pushes the row to disk immediately instead of waiting in memory.
    • A failed message is printed and left out of the file, so the next run tries it again. Ruff normally objects to catching every kind of Exception; the noqa comment says you mean it here, and the line above says why.
    • Ctrl-C cancels the queued messages, lets the four running ones finish, and exits. Run the same command to carry on.

  4. Your turn: write check(). It runs run_agent(text), times it with time.perf_counter(), and returns a dict with exactly the keys in FIELDS. Lists don't fit in a CSV cell, so join red_flags with " | " and the tool names from run.steps with ";". Use "" when matched_pattern is None.

    Stuck? One version of check() 
    def check(row_id: int, text: str) -> dict:
    """Run the agent on one message and flatten the answer into one CSV row."""
    start = time.perf_counter()
    run = run_agent(text)
    return {
        "id": row_id,
        "verdict": run.result.verdict,
        "matched_pattern": run.result.matched_pattern or "",
        "red_flags": " | ".join(run.result.red_flags),
        "advice": run.result.advice,
        "tools": ";".join(step.tool for step in run.steps),
        "n_steps": len(run.steps),
        "seconds": round(time.perf_counter() - start, 1),
    }

    step.tool assumes each step in your week 2 AgentRun.steps stores the tool name in a field called tool. If yours is called something else, change that one place.

  5. Rehearse the crash. Make sure the database is up (docker compose up -d), start the run, let about five messages finish, then press Ctrl-C:

    uv run python scripts/run_eval.py

    Run the same command again. The first line should now say 5 already done, 95 to run (or however many had finished). That's the resume working.

  6. Let it finish. In the rehearsal, 10 messages took 54 seconds with four workers, so 100 take about 10 minutes. If a usage limit stops it, wait and re-run; nothing is lost.

  7. The shared code for this week's scripts, src/scam_checker/eval_data.py. It joins your labels with one results file and names each mistake:

    """Join your labels with one eval run, and name each mistake."""

from pathlib import Path

import pandas as pd

LABELS_PATH = Path("data/labelled100.csv")


def latest_results() -> Path:
    """The newest evals/results_*.csv (the date in the name sorts correctly)."""
    return max(Path("evals").glob("results_*.csv"))


def failure_type(row: pd.Series) -> str:
    """Name the mistake in one row, or return '' if the answer was right."""
    if row["is_scam"]:
        if row["verdict"] == "not_scam":
            return "missed scam"
        if row["verdict"] == "unsure":
            return "unsure on scam"
        if row["scam_type"] != "other" and row["matched_pattern"] != row["scam_type"]:
            return "wrong pattern"
        return ""
    if row["verdict"] == "scam":
        return "false alarm"
    if row["verdict"] == "unsure":
        return "unsure on genuine"
    return ""


def load_joined(results_path: Path) -> pd.DataFrame:
    """One row per labelled message that has a result, with a `failure` column."""
    labels = pd.read_csv(LABELS_PATH)
    results = pd.read_csv(results_path, keep_default_na=False)
    df = labels.merge(results, on="id", how="inner")
    df["failure"] = df.apply(failure_type, axis=1)
    return df

    keep_default_na=False keeps empty cells as "" rather than turning them into NaN, so "no pattern" compares cleanly. "Wrong pattern" counts as a mistake: the verdict was right but the citation wasn't.

  8. Add a confidence interval to src/scam_checker/metrics.py. Put import math at the top, then:

    def wilson_interval(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """95% confidence interval for a proportion such as recall, honest at small n."""
    if n == 0:
        return 0.0, 1.0
    p = successes / n
    centre = (p + z**2 / (2 * n)) / (1 + z**2 / n)
    half = z * math.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / (1 + z**2 / n)
    return max(0.0, centre - half), min(1.0, centre + half)

    Check it: wilson_interval(40, 50) should give about (0.67, 0.89). See Binomial proportion confidence interval if you want the derivation.

  9. Score the run with scripts/score_eval.py:

    """Precision, recall and mistake counts for one eval run.

Usage: uv run python scripts/score_eval.py [results file]   (default: the newest)
"""

import sys
from pathlib import Path

from scam_checker.eval_data import latest_results, load_joined
from scam_checker.metrics import precision_recall, wilson_interval

path = Path(sys.argv[1]) if len(sys.argv) > 1 else latest_results()
df = load_joined(path)

truth = df["is_scam"].tolist()
flagged = (df["verdict"] == "scam").tolist()  # "unsure" counts as not flagged
p, r = precision_recall(truth, flagged)
tp = sum(t and f for t, f in zip(truth, flagged))
p_low, p_high = wilson_interval(tp, sum(flagged))
r_low, r_high = wilson_interval(tp, sum(truth))

print(f"{path}: {len(df)} messages, {sum(truth)} scams")
print(
    f"precision {p:.2f}  (95% CI {p_low:.2f}-{p_high:.2f}, from {sum(flagged)} flagged)"
)
print(f"recall    {r:.2f}  (95% CI {r_low:.2f}-{r_high:.2f}, from {sum(truth)} scams)")
print(f"unsure on {(df['verdict'] == 'unsure').sum()} messages")
print()
print(df["failure"].replace("", "correct").value_counts().to_string())

    uv run python scripts/score_eval.py

    Write the numbers in the README. They're your "before". Note how wide the intervals are.

Done when: evals/results_<today>.csv has 100 rows, you've seen the resume work after a Ctrl-C, and the README has precision and recall with their intervals.

Stuck?
  • Many rows fail with a usage-limit message: you've hit your subscription's limit. Stop (Ctrl-C), wait, and re-run later; finished rows are kept. Don't raise WORKERS to make up time.
  • psycopg.OperationalError on every row: the database isn't running, so lookup_scam_pattern fails. docker compose up -d, then re-run.
  • pandas.errors.EmptyDataError at the start: the results file exists but is empty (the run died before writing the header). Delete it and start again.
  • AttributeError: 'Step' object has no attribute 'tool': your step objects name the field differently. Print one run.steps[0] and use its field name in check().
  • Ctrl-C seems to do nothing for a while: it's waiting for the four calls already running. Give it up to a minute.
  • A second run on the same day says 100 already done, 0 to run: with no file name, the script treats it as a resume of today's file. Pass a new name, such as uv run python scripts/run_eval.py evals/results_2027-01-12_sonnet.csv.

Wednesday — Read every failure: open coding

Learn (45 min):

Build:

  1. A viewer that shows one result at a time and saves your note, evals/viewer.py:

    """Read eval results one message at a time and write a note on each.

Run with: uv run streamlit run evals/viewer.py
"""

from pathlib import Path

import pandas as pd
import streamlit as st

from scam_checker.eval_data import latest_results, load_joined

NOTES_PATH = Path("evals/annotations.csv")
NOTE_COLUMNS = ["results_file", "id", "note", "category"]

results_path = latest_results()
df = load_joined(results_path)
notes = (
    pd.read_csv(NOTES_PATH, keep_default_na=False)
    if NOTES_PATH.exists()
    else pd.DataFrame(columns=NOTE_COLUMNS)
)
notes = notes[notes["results_file"] == results_path.name]

st.title("Eval viewer")
show_all = st.toggle("Show correct answers too", value=False)
rows = df if show_all else df[df["failure"] != ""]
if rows.empty:
    st.success("No failures in this run.")
    st.stop()

if "i" not in st.session_state:
    st.session_state.i = 0
st.session_state.i = min(st.session_state.i, len(rows) - 1)
left, middle, right = st.columns([1, 2, 1])
if left.button("Previous") and st.session_state.i > 0:
    st.session_state.i -= 1
if right.button("Next") and st.session_state.i < len(rows) - 1:
    st.session_state.i += 1
row = rows.iloc[st.session_state.i]
middle.write(f"{st.session_state.i + 1} of {len(rows)} · `{results_path.name}`")

st.subheader(f"Message {row['id']}")
st.text(row["text"])
st.write(
    f"**Your label:** {'scam' if row['is_scam'] else 'genuine'}"
    f" ({row['scam_type']}). {row['notes']}"
)
st.write(f"**Checker said:** {row['verdict']}, pattern `{row['matched_pattern']}`")
if row["failure"]:
    st.error(row["failure"])
st.write(f"**Red flags:** {row['red_flags']}")
st.write(f"**Advice:** {row['advice']}")
st.write(f"**Tools ({row['n_steps']} steps):** {row['tools'] or 'none'}")

saved = notes[notes["id"] == row["id"]]
with st.form(key=f"note-{row['id']}"):
    note = st.text_area(
        "What went wrong, in your own words",
        value=saved["note"].iloc[0] if len(saved) else "",
    )
    category = st.text_input(
        "Category (fill in later)",
        value=saved["category"].iloc[0] if len(saved) else "",
    )
    if st.form_submit_button("Save note"):
        all_notes = (
            pd.read_csv(NOTES_PATH, keep_default_na=False)
            if NOTES_PATH.exists()
            else pd.DataFrame(columns=NOTE_COLUMNS)
        )
        same = (all_notes["results_file"] == results_path.name) & (
            all_notes["id"] == row["id"]
        )
        new = pd.DataFrame([[results_path.name, row["id"], note, category]])
        new.columns = NOTE_COLUMNS
        pd.concat([all_notes[~same], new]).to_csv(NOTES_PATH, index=False)
        st.success("Saved.")

    uv run streamlit run evals/viewer.py

    Streamlit reruns the whole script on every click, so st.session_state.i is how it remembers which message you're on. st.form groups the two boxes so typing doesn't trigger a rerun until you press Save. Notes are stored per results file, so Friday's re-run starts with a clean slate.

  2. Your turn: open coding. Go through every failure. For each, write one or two sentences in the note box about the first thing that went wrong, in plain words, as you'd explain it to a colleague. Don't fill in the category yet. Good notes are specific:

    • Weak: "wrong".
    • Good: "Flagged a genuine DBS OTP message as a scam because it mentions a transfer; it ignored 'never share your OTP', which is what real banks say."
    • Good: "Right verdict, but the advice gives a police phone number that isn't in the message or the knowledge base, and it's wrong."

    The failure label in red is only the automatic kind of mistake. Your note is the why.

  3. Then switch on "Show correct answers too" and read at least 20 of the answers marked correct. Right verdicts hide problems the counts can't see. In the rehearsal, one "correct" answer told the reader to call a police number that was wrong by one digit, and another gave a SingPost number that appears nowhere in the message or the knowledge base, so nothing backs it up. Note those too.

  4. For any trace that puzzles you, open the matching line in runs/ to see what each tool actually returned. A wrong pattern often turns out to be a retrieval problem, not a model problem.

Done when: every failure, plus at least 20 correct answers, has a note in evals/annotations.csv.

Stuck?
  • ValueError: max() iterable argument is empty: there's no evals/results_*.csv yet. Run Tuesday's script first, from the project root.
  • The page says "No failures in this run": either the checker is perfect (check the labels loaded) or the ids in the results don't match labelled100.csv. Look at both files' id columns.
  • Notes vanish when you click Next: you didn't press "Save note" first. The form only saves on the button.
  • ModuleNotFoundError: scam_checker: run Streamlit from the project root with uv run, as above.
  • It's slow going: that's normal. Budget 2–3 minutes per failure. Reading is the work; the AI Evals FAQ calls error analysis the most important activity in evals.

Thursday — Categories, ranking and the agent's trajectory

Learn (40 min):

Build:

  1. Your turn: axial coding. Read all your notes in one go (open evals/annotations.csv in a spreadsheet or the viewer). Group them into 4–8 categories with short, specific names, and type each row's category into the viewer or the CSV. In the rehearsal the categories were:

    • skipped pattern tool: answered without calling lookup_scam_pattern, so no citation
    • invented contact details: advice gives a phone number that isn't in the message or the knowledge base
    • genuine bank message not trusted: a real OTP or transaction alert marked unsure or scam

    A good category points at a fix. "Model is bad at banks" doesn't; "treats any amount plus 'transfer' as a scam signal, even in an OTP message" does.

  2. Rank them with scripts/rank_failures.py. Set HARM to your own judgement for each of your categories:

    """Rank your failure categories: how often each happens, times how much it hurts.

Usage: uv run python scripts/rank_failures.py [results file]   (default: the newest)
"""

import sys
from pathlib import Path

import pandas as pd

from scam_checker.eval_data import latest_results, load_joined

# Your judgement, per category: 3 = the reader could lose money,
# 2 = the reader is misled but probably safe, 1 = annoying or untidy.
HARM = {
    "skipped pattern tool": 1,
    "invented contact details": 3,
    "genuine bank message not trusted": 2,
}

path = Path(sys.argv[1]) if len(sys.argv) > 1 else latest_results()
df = load_joined(path)
notes = pd.read_csv("evals/annotations.csv", keep_default_na=False)
notes = notes[notes["results_file"] == path.name]
df = df.merge(notes[["id", "category"]], on="id", how="inner")
df["category"] = df["category"].replace("", "(no category yet)")

ranking = df.groupby("category").agg(count=("id", "size"), ids=("id", list))
ranking["harm"] = [HARM.get(c, 0) for c in ranking.index]
ranking["priority"] = ranking["count"] * ranking["harm"]
ranking = ranking.sort_values(["priority", "count"], ascending=False)

print(f"{path.name}: {len(df)} annotated messages\n")
print(ranking[["count", "harm", "priority", "ids"]].to_string())
missing = [c for c in ranking.index if c not in HARM]
if missing:
    print(f"\nGive these a harm score in HARM: {missing}")

    uv run python scripts/rank_failures.py

    On the 10-message rehearsal it printed:

    results_2026-10-17.csv: 4 annotated messages

                                  count  harm  priority     ids
category
invented contact details              1     3         3     [4]
skipped pattern tool                  2     1         2  [1, 2]
genuine bank message not trusted      1     2         2     [6]

    Copy your table into the README with one sentence per category. Be ready to defend each harm score in an interview: why is an invented phone number worse than a missing citation?

  3. Grade the trajectory with scripts/grade_trajectory.py. It reads the tool names and step counts that run_eval.py saved:

    """Grade how the agent worked, not just what it answered.

Usage: uv run python scripts/grade_trajectory.py [results file]   (default: the newest)
"""

import sys
from pathlib import Path

from scam_checker.baseline import URL_PATTERN
from scam_checker.eval_data import latest_results, load_joined

LINK_TOOLS = {"extract_links"}  # use the names from your own TOOLS registry
MAX_STEPS = 4

path = Path(sys.argv[1]) if len(sys.argv) > 1 else latest_results()
df = load_joined(path)
tools = df["tools"].str.split(";").apply(lambda t: [x for x in t if x])

checks = {
    "looked up a scam pattern": tools.apply(lambda t: "lookup_scam_pattern" in t),
    "checked links when there were any": [
        not URL_PATTERN.search(text) or bool(LINK_TOOLS & set(t))
        for text, t in zip(df["text"], tools)
    ],
    f"stopped within {MAX_STEPS} steps": df["n_steps"] <= MAX_STEPS,
    "no tool called twice": tools.apply(lambda t: len(t) == len(set(t))),
}

print(f"{path.name}: {len(df)} runs\n")
for name, passed in checks.items():
    failed = df.loc[[not p for p in passed], "id"].tolist()
    print(f"{name:36} {len(df) - len(failed):3}/{len(df)}   failed: {failed}")
print(f"\nsteps per run: mean {df['n_steps'].mean():.1f}, max {df['n_steps'].max()}")

    Put your week 2 link tools (expanding short links, checking domain age, checking a blocklist) in LINK_TOOLS. URL_PATTERN is your week 1 keyword baseline's link rule, reused. The rehearsal printed:

    results_2026-10-17.csv: 10 runs

looked up a scam pattern               7/10   failed: [1, 2, 6]
checked links when there were any      9/10   failed: [2]
stopped within 4 steps                10/10   failed: []
no tool called twice                  10/10   failed: []

steps per run: mean 0.7, max 1

    Your turn: add one check of your own that matters for your agent. For example: a message that claims to be from a government agency should trigger your government-sender tool. Add any category the checks reveal to your ranking.

  4. Pick the top category and plan its fix. Write in the README: the category, why it's top, what you'll change, and what number should move. Start the fix this afternoon if there's time.

Done when: every annotated row has a category, the README has the ranked table and the trajectory results, and you've written down the fix you'll make.

Stuck?
  • FileNotFoundError: evals/annotations.csv: you haven't saved any notes yet. Do Wednesday's open coding first.
  • A category shows harm 0 and a message at the bottom: it isn't in HARM. Add it with a score.
  • AttributeError: Can only use .str accessor with string values, not floating: the tools column was read as NaN. load_joined reads results with keep_default_na=False; check you didn't drop that.
  • Categories keep multiplying: merge them. More than eight usually means some are the same failure seen from different sides.

Friday — Fix the top failure, re-run, and demo day

Learn (30 min): re-read your own README table from yesterday and the "Confidence intervals at n = 100" concept. Then skim How do I know if I can trust my automated eval?, a preview of week 7.

Build (before 9:00 or after demo day):

  1. Your turn: fix the top category. Prefer fixes in code over fixes in the prompt when the rule is clear-cut; code does the same thing every time. Typical fixes:

    • Skipped a tool the agent must always use: call it in code before the loop and put its result in the first prompt, rather than asking the model to remember.
    • Genuine bank or government messages flagged: add what genuine ones look like to the knowledge base or the prompt (for example, ScamShield says banks never send clickable links by SMS, so a bank SMS without a link isn't suspicious for that reason).
    • Invented contact details: remove them in code. The worked example below.

  2. Worked example from the rehearsal, where "invented contact details" ranked first. The check doesn't need a model, so it lives in src/scam_checker/guards.py:

    """Checks on the checker's answer that don't need a model."""

import re

from scam_checker.models import CheckResult
from scam_checker.retrieval import load_patterns

# 4+ digits, maybe with spaces or hyphens (1799, 6123 4567, 1800-123-4567),
# but not prices or decimals such as $1.99 or 12.50
PHONE = re.compile(r"(?<![\d$])(?<!\d\.)\+?\d(?:[\s-]?\d){3,}(?!\d|\.\d)")
SAFE_ADVICE = "If you're unsure, call the ScamShield Helpline on 1799."


def numbers_in(text: str) -> set[str]:
    return {re.sub(r"\D", "", m) for m in PHONE.findall(text)}


def known_numbers(message: str) -> set[str]:
    """Numbers the advice may mention: ones in the message or the knowledge base."""
    kb = " ".join(p.body for p in load_patterns())
    return numbers_in(message) | numbers_in(kb)


def unknown_numbers(advice: str, message: str) -> set[str]:
    return numbers_in(advice) - known_numbers(message)


def remove_unknown_numbers(result: CheckResult, message: str) -> CheckResult:
    """Drop any advice sentence with a phone number we can't vouch for."""
    allowed = known_numbers(message)
    sentences = re.split(r"(?<=[.!?])\s+", result.advice)
    kept = [s for s in sentences if numbers_in(s) <= allowed]
    if len(kept) == len(sentences):
        return result
    advice = " ".join(kept + [SAFE_ADVICE])
    return result.model_copy(update={"advice": advice})

    1799 is the ScamShield Helpline, and it's in your knowledge base, so it's allowed. If you want the advice to be able to give another official number, such as the police hotline, add it to a knowledge base file with its source, and the guard will allow it too. In run_agent, just before it returns, pass the final answer through the guard:

    run.result = remove_unknown_numbers(run.result, text)

    And test it, in tests/test_guards.py:

    from scam_checker.guards import numbers_in, remove_unknown_numbers
from scam_checker.models import CheckResult


def test_finds_phone_numbers_but_not_prices():
    assert numbers_in("Call 6123 4567 or 1800-123-4567. Pay $1.99.") == {
        "61234567",
        "18001234567",
    }


def test_drops_sentences_with_invented_numbers():
    result = CheckResult(
        verdict="scam",
        red_flags=[],
        advice="Don't pay. Call the police on 1800-000-0000.",
    )
    fixed = remove_unknown_numbers(result, "Pay $5 now")
    assert "1800" not in fixed.advice
    assert fixed.advice.startswith("Don't pay.")
    assert "1799" in fixed.advice


def test_keeps_numbers_from_the_message():
    result = CheckResult(verdict="unsure", red_flags=[], advice="Call 6123 4567.")
    assert remove_unknown_numbers(result, "Call us on 6123 4567") == result

    These tests caught a real bug in the first version of the pattern: a full stop straight after a number ("call 6123 4567.") cut the number short. That's why the end of PHONE says "not followed by a digit, or by a dot and a digit" rather than "not followed by a dot".

  3. Turn the category into a number you can track on every run. Add two lines to score_eval.py, the import with the others and the count after the "unsure" line:

    from scam_checker.guards import unknown_numbers

    invented = sum(bool(unknown_numbers(a, t)) for a, t in zip(df["advice"], df["text"]))
print(f"advice with a phone number not in the message or knowledge base: {invented}")

    Whatever your top category is, do the same: if code can spot it, write the check, so the category becomes a count instead of something you have to reread.

  4. Re-run into a new file and score both runs:

    uv run python scripts/run_eval.py evals/results_2027-01-15_fix1.csv
uv run python scripts/score_eval.py evals/results_2027-01-12.csv
uv run python scripts/score_eval.py evals/results_2027-01-15_fix1.csv
uv run python scripts/grade_trajectory.py evals/results_2027-01-15_fix1.csv

    Use your own dates. In the rehearsal (10 messages), the invented-number count went from 2 to 0 and precision and recall stayed at 1.00. But two other things moved even though the fix didn't touch them: "wrong pattern" went from 2 to 3, and the "unsure" verdict moved from one genuine message to another. Same code, different run. That's run-to-run noise, and it's why you judge a fix by the category it targeted, and treat small moves elsewhere with suspicion.

  5. Write the before-and-after in the README:

    Before After fix 1
    Precision (95% CI)
    Recall (95% CI)
    Top category count
    Looked up a pattern

    Then one paragraph: did the target category drop, did anything else get worse, and is the change bigger than the noise? With overlapping intervals, the honest answer to "did recall improve?" is often "not measurably", and saying so is a strength.

Demo day (9:00–12:00):

  1. Record a 3-minute screen video: the labelled file and your labelling rules, the viewer with one interesting failure and your note on it, the ranked table, and the before-and-after numbers.
  2. Write five lines on what you learned this week.
  3. Send both to Claude with the Friday check-in. Under "My numbers" put precision and recall with their intervals, and the top category's count before and after.

Apply now: this is the evidence evaluation roles ask for, including the NTU evaluation role on your list. Send applications this afternoon. In the application or cover note, lead with the work, not the tools: "I labelled 100 messages including tricky genuine bank and government messages, read every failure, built a failure taxonomy ranked by frequency and harm, and fixed the top category with a code check; precision X and recall Y (95% CI ...)". Link the repo and the README section. Your ten years of deciding what counts as fraud is the domain expertise these roles need; say so.

Deliverable checklist

  • data/labelled100.csv: 100 messages with scam_type and notes, at least 15 tricky genuine ones, labelling rules and a self-agreement check in the README
  • scripts/run_eval.py running four at a time, resuming after a stop, writing evals/results_<date>.csv
  • evals/viewer.py and notes on every failure plus 20 correct answers
  • A ranked table of failure categories (count × harm) in the README
  • Trajectory grades: pattern lookup, link checks, step counts
  • Precision and recall with 95% intervals, before and after fixing the top category
  • A code check for the top category, with tests
  • Demo video, five lines, the Friday check-in, and applications sent

If you have time left:

  • Measure your noise floor: run the unchanged checker twice into two files and compare. Any difference between them is noise, and tells you how big a real improvement has to be.
  • Run the 100 with model="sonnet" passed through to call_claude in your agent, into its own results file, and compare cost, time and the failure counts with Haiku. Watch your usage limit; that's at least 200 calls.
  • Fix the second category and re-run. Keep every results file; by week 7 they become the history your CI gate compares against.