Skip to content

Week 1 — Checker v0: rough but working

Why this week: day one is AI work. By Friday you have a checker that already does something useful, with numbers showing it beats a simple keyword rule.

By Friday you have: a scam-checker repository on GitHub with a keyword baseline, an LLM checker that returns structured answers and fails safely, a comparison on 30 messages, and a one-page app running on your laptop.

Main online course: Anthropic Academy's Building with the Claude API, up to and including structured output. The course uses an API key in its exercises; watch and read it for the ideas, then do the exercises with call_claude() from Calling Claude from Python.

Concepts

Repository and commit. A repository (repo) is a project folder whose history git records. A commit is a saved snapshot with a message saying what changed. Small, frequent commits let you undo mistakes and show your progress.

Virtual environment and uv. Each project gets its own private copy of Python packages, so projects can't break each other. uv creates that environment, installs packages and records exact versions in uv.lock, so the project works the same on another machine.

Linter and formatter. ruff check finds likely bugs and bad patterns; ruff format rewrites your code into one consistent layout. You stop arguing with yourself about style.

Baseline. The simplest reasonable solution, measured first. If an LLM can't beat a keyword list, it isn't worth its cost. Every claim you make in an interview should be "X beats baseline Y by Z".

Structured output. Instead of free text, you make the model reply in a fixed JSON shape (verdict, red flags, advice). Your code can then use the answer without guessing how to read it. Pydantic checks the shape for you.

Precision and recall, for scams. Precision: of the messages you flagged, how many were scams. Recall: of the real scams, how many you caught. For a scam checker, a missed scam (low recall) usually hurts more than a false alarm, but too many false alarms and people stop listening.

Retries, backoff and fallback. Calls fail: timeouts, usage limits, malformed replies. You retry a few times, waiting longer each time (backoff) with a little randomness (jitter) so retries don't pile up. If every attempt fails, you return a safe answer instead of crashing.

Monday — Set up the repo

Learn (75 min):

  • uv: Creating projects, the "Packaged applications" section (20 min)
  • Ruff tutorial (15 min)
  • Pro Git, chapter 2.2 "Recording Changes to the Repository" (30 min), if commits are new to you
  • Start the Building with the Claude API course

Build:

  1. Make the project. In a terminal, in the folder where you keep projects:

    uv init --package scam-checker
cd scam-checker
uv add --dev ruff

    --package gives you the src/scam_checker/ layout. It keeps your code separate from scripts and data, which matters once you add tests next week.

  2. Copy the helper from Calling Claude from Python into src/scam_checker/llm.py, then check it works:

    uv run python -m scam_checker.llm

    You should see a short greeting and a line of token counts.

  3. Lint and format everything:

    uv run ruff format .
uv run ruff check .

  4. Keep downloaded data out of git. Add this line to .gitignore:

    data/raw/

  5. Make the first commit and put the repo on GitHub:

    git add .
git commit -m "Set up project with uv, ruff and the Claude helper"
gh repo create scam-checker --public --source . --push

    A public repo is part of your portfolio. Never commit passwords or keys to it; there are none yet, and week 9 shows where secrets go.

  6. Copy this week's Friday checklist (bottom of this page) into README.md as your plan for the week.

Done when: uv run ruff check . prints All checks passed! and your repo is visible at github.com/<your-username>/scam-checker.

Stuck?
  • uv: command not found: install uv with brew install uv, then open a new terminal.
  • gh: command not found or not logged in: brew install gh, then gh auth login.
  • ModuleNotFoundError: scam_checker: run commands from the scam-checker folder, with uv run in front.

Tuesday — Data and the keyword baseline

Learn (60 min): skim the SMS Spam Collection page: where the data comes from, and its licence. Then read pandas: Getting started only if read_csv and filtering feel rusty.

This dataset is 5,574 UK text messages from around 2012, labelled spam or ham (genuine). Spam isn't the same as a scam, and it isn't Singapore. It's a fine start for v0: week 5 replaces it with 100 messages you label yourself.

Build:

  1. Download the data and add pandas:

    mkdir -p data/raw
curl -L -o data/raw/smsspam.zip "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
unzip data/raw/smsspam.zip -d data/raw
uv add pandas

  2. A loader, src/scam_checker/data.py. The file is tab-separated with no header, and some messages contain quote marks, so quoting is switched off:

    """Load the SMS Spam Collection as a DataFrame."""

import csv
from pathlib import Path

import pandas as pd

RAW_PATH = Path("data/raw/SMSSpamCollection")


def load_sms() -> pd.DataFrame:
    df = pd.read_csv(
        RAW_PATH, sep="\t", header=None, names=["label", "text"], quoting=csv.QUOTE_NONE
    )
    df["is_scam"] = df["label"] == "spam"
    return df

  3. Precision and recall, written out by hand in src/scam_checker/metrics.py. You'll reuse this every week:

    """Precision and recall, written out so you can see what they count."""


def precision_recall(truth: list[bool], predicted: list[bool]) -> tuple[float, float]:
    tp = sum(t and p for t, p in zip(truth, predicted))
    fp = sum(p and not t for t, p in zip(truth, predicted))
    fn = sum(t and not p for t, p in zip(truth, predicted))
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return precision, recall

  4. Your turn: write the keyword baseline in src/scam_checker/baseline.py. Start from this, then read 30 spam messages and grow the list:

    """A keyword-rule checker: the number the LLM version has to beat."""

import re

SUSPICIOUS_PHRASES = [
    "you have won",
    "claim",
    "urgent",
    "verify",
    "click",
]
URL_PATTERN = re.compile(r"https?://|www\.|\.com\b|\.ly\b", re.IGNORECASE)


def red_flags(text: str) -> list[str]:
    lowered = text.lower()
    flags = [f"contains '{p}'" for p in SUSPICIOUS_PHRASES if p in lowered]
    if URL_PATTERN.search(text):
        flags.append("contains a link")
    return flags


def keyword_check(text: str, threshold: int = 2) -> bool:
    """True if the message trips at least `threshold` rules."""
    return len(red_flags(text)) >= threshold

  5. Score it on the whole dataset with scripts/eval_baseline.py:

    from scam_checker.baseline import keyword_check
from scam_checker.data import load_sms
from scam_checker.metrics import precision_recall

df = load_sms()
predicted = [keyword_check(t) for t in df["text"]]
p, r = precision_recall(df["is_scam"].tolist(), predicted)
print(f"{len(df)} messages, {df['is_scam'].sum()} spam")
print(f"keyword baseline: precision {p:.2f}, recall {r:.2f}")

    uv run python scripts/eval_baseline.py

    Change the phrase list and the threshold a few times and watch both numbers move. Notice the trade-off: catching more spam usually costs more false alarms.

  6. Build the 30-message test set you'll use all week, scripts/make_eval_set.py: 12 spam, 12 genuine, and 6 Singapore-style messages you write yourself from public advisories, such as the ScamShield site:

    """Build data/eval30.csv: 12 spam and 12 ham from the dataset, plus 6 you write."""

import pandas as pd

from scam_checker.data import load_sms

df = load_sms()
spam = df[df["is_scam"]].sample(12, random_state=42)
ham = df[~df["is_scam"]].sample(12, random_state=42)

own = pd.DataFrame(
    {
        "text": [
            "Hi mum, I dropped my phone, this is my new number. Can you help me pay a bill urgently?",
            "SingPost: your parcel is on hold due to unpaid fees. Pay $1.99 at singpost-redelivery.com",
            "Part-time job: earn $300/day liking videos. Contact our HR on WhatsApp.",
            "DBS: your account has been suspended. Verify your details at dbs-secure-login.com",
            "Your Grab ride is arriving in 3 minutes. Driver: Tan, SJK1234Z.",
            "Reminder: your dental appointment is on Tue 10 Dec at 3pm. Reply C to confirm.",
        ],
        "is_scam": [True, True, True, True, False, False],
    }
)
eval_set = pd.concat([spam[["text", "is_scam"]], ham[["text", "is_scam"]], own])
eval_set.to_csv("data/eval30.csv", index=False)
print(eval_set["is_scam"].value_counts())

    Replace the six example messages with ones you write. Include at least one tricky genuine message, like the dental reminder that says "Reply C".

    random_state=42 makes the sample the same every run, so your numbers are comparable from day to day.

Done when: your baseline's precision and recall on the full dataset are written in README.md, data/eval30.csv exists, and both are committed.

Stuck? A fuller baseline

One version that gets roughly precision 0.98 and recall 0.52 on the full dataset:

SUSPICIOUS_PHRASES = [
    "you have won", "you've won", "winner", "claim", "prize", "free",
    "urgent", "act now", "verify", "account suspended", "account locked",
    "click", "call now", "txt", "reply", "cash", "guaranteed",
]
MONEY_PATTERN = re.compile(r"[£$]\s?\d|\d+\s?(?:sgd|usd|dollars)", re.IGNORECASE)

Add if MONEY_PATTERN.search(text): flags.append("mentions money") inside red_flags. High precision, low recall is typical of keyword rules: they're sure when they fire, and they miss anything phrased differently.

Wednesday — One LLM call, with structured output

Learn (75 min):

  • The structured output lessons in the Building with the Claude API course
  • Structured outputs (docs, 30 min): read for the idea. Your call_claude(..., schema=...) does the same job through the command line.

Build:

  1. Add Pydantic, which turns a Python class into a JSON schema and checks replies against it:

    uv add pydantic

  2. The answer's shape, src/scam_checker/models.py:

    """The shape of every answer the checker gives."""

from typing import Literal

from pydantic import BaseModel, Field


class CheckResult(BaseModel):
    verdict: Literal["scam", "not_scam", "unsure"]
    red_flags: list[str] = Field(description="Specific warning signs found in the message")
    advice: str = Field(description="One or two plain sentences on what the reader should do")

    The description text is sent to the model as part of the schema, so write it as an instruction.

  3. A first checker, src/scam_checker/checker.py. No error handling yet; that's tomorrow:

    """Check one message with Claude."""

from scam_checker.llm import call_claude
from scam_checker.models import CheckResult

SYSTEM_PROMPT = """You check text messages for scams, for ordinary people in Singapore.

Decide whether the message is a scam, not a scam, or unclear.
List the specific warning signs you see, quoting the words that worry you.
Give one or two sentences of advice a non-technical reader can act on.
Use "unsure" when the message could plausibly be genuine."""


def check_message(text: str) -> CheckResult:
    reply = call_claude(
        f"Message to check:\n\n{text}",
        system=SYSTEM_PROMPT,
        schema=CheckResult.model_json_schema(),
    )
    return CheckResult.model_validate(reply.data)

  4. Try it on five messages from eval30.csv, including two genuine ones:

    uv run python -c "from scam_checker.checker import check_message; print(check_message('Your OTP is 482913. Do not share it with anyone.'))"

  5. Your turn: read each answer as a sceptical user would. Are the red flags specific? Is the advice something your parents could act on? Did it invent facts, such as a phone number? Change the system prompt once to fix the worst problem, and note the before and after in your README.

Done when: check_message returns a CheckResult for any message you give it, and you've made one prompt change for a reason you can explain.

Stuck?
  • ValidationError: the reply didn't match the schema. Print reply.data to see what came back.
  • reply.data is None: the schema wasn't passed. Check you used schema=, not system=.
  • The model invents helpline numbers: tell it in the system prompt not to give phone numbers or links unless they are in the message.

Thursday — Fail safely, then compare

Learn (60 min):

  • Claude API errors (docs, 15 min): which errors are worth retrying (overloaded, rate limited, timeouts) and which aren't (a bad request). The command line surfaces the same failures as a ClaudeError.
  • Read your own llm.py again and find each place it raises ClaudeError.

Build:

  1. Your turn: make check_message survive failures. It should:

    • retry when call_claude raises ClaudeError or Pydantic raises ValidationError
    • try at most 3 times, waiting about 2 seconds, then 4, plus a random fraction of a second
    • log each failure with the logging module
    • return a safe fallback answer if every attempt fails, never crash

    Write it yourself first; the hint below has one version.

  2. Test the failure path on purpose: temporarily call call_claude(..., timeout=0.01) so every attempt times out. You should see three warnings, then the fallback answer. Put the timeout back afterwards.

  3. Compare the two checkers on the 30 messages, scripts/compare.py:

    """Run the baseline and the LLM checker on the 30-message set and compare."""

import pandas as pd

from scam_checker.baseline import keyword_check
from scam_checker.checker import check_message
from scam_checker.metrics import precision_recall

df = pd.read_csv("data/eval30.csv")
truth = df["is_scam"].tolist()

baseline = [keyword_check(t) for t in df["text"]]

verdicts = []
for i, text in enumerate(df["text"], start=1):
    verdicts.append(check_message(text).verdict)
    print(f"{i}/{len(df)} {verdicts[-1]}")
llm = [v == "scam" for v in verdicts]

for name, predicted in [("keyword baseline", baseline), ("LLM checker", llm)]:
    p, r = precision_recall(truth, predicted)
    print(f"{name:17} precision {p:.2f}  recall {r:.2f}")
print(f"LLM said 'unsure' on {verdicts.count('unsure')} of {len(df)}")

df["llm_verdict"] = verdicts
df["baseline"] = baseline
df.to_csv("data/eval30_results.csv", index=False)

    uv run python scripts/compare.py

    It takes about 4–5 minutes, one call at a time. Here "unsure" counts as not flagged; decide whether you agree, and say so in the README.

  4. Open data/eval30_results.csv and read every message where the two checkers disagree. Write two sentences in the README about what the LLM catches that keywords miss, and one thing it gets wrong.

Done when: a table in your README shows precision and recall for both checkers on the same 30 messages, and the failure test returned the fallback answer.

Stuck? One version of the retrying checker 
import logging
import random
import time

from pydantic import ValidationError

from scam_checker.llm import ClaudeError, call_claude
from scam_checker.models import CheckResult

logger = logging.getLogger(__name__)

FALLBACK = CheckResult(
    verdict="unsure",
    red_flags=[],
    advice=(
        "We couldn't check this message. Don't click its links or share codes; "
        "contact the organisation using a number you find yourself."
    ),
)


def check_message(text: str, max_attempts: int = 3) -> CheckResult:
    """Return a verdict, retrying failures and falling back to a safe answer."""
    for attempt in range(1, max_attempts + 1):
        try:
            reply = call_claude(
                f"Message to check:\n\n{text}",
                system=SYSTEM_PROMPT,
                schema=CheckResult.model_json_schema(),
            )
            return CheckResult.model_validate(reply.data)
        except (ClaudeError, ValidationError) as exc:
            logger.warning("attempt %d of %d failed: %s", attempt, max_attempts, exc)
            if attempt < max_attempts:
                time.sleep(2**attempt + random.random())  # 2s, 4s, ... plus jitter
    return FALLBACK

Keep SYSTEM_PROMPT from Wednesday above this function. To see the warnings in a script, add logging.basicConfig(level=logging.INFO) at its top.

Friday — A page people can try, and demo day

Learn (45 min): Streamlit: Get started, the "Fundamentals" pages.

Build (before 9:00 or after demo day):

  1. Add Streamlit and write app.py in the project root:

    uv add streamlit

    """A one-page checker. Run with: uv run streamlit run app.py"""

import streamlit as st

from scam_checker.checker import check_message

st.title("Is this message a scam?")
st.caption("A learning project. It can be wrong; when in doubt, don't click.")

text = st.text_area("Paste the message here", height=150)

if st.button("Check", type="primary") and text.strip():
    with st.spinner("Checking..."):
        result = check_message(text)

    if result.verdict == "scam":
        st.error("Likely a scam")
    elif result.verdict == "not_scam":
        st.success("Probably genuine")
    else:
        st.warning("Not sure")

    if result.red_flags:
        st.subheader("Warning signs")
        for flag in result.red_flags:
            st.write(f"- {flag}")
    st.subheader("What to do")
    st.write(result.advice)

  2. Run it and try your six own messages:

    uv run streamlit run app.py

    It opens in your browser at localhost:8501.

Why there's no public link yet

The original plan puts this page on Streamlit Community Cloud this week. That would need an API key, because a subscription login can't run an app for other people. You get a public link in week 9 instead, once the checker has a proper backend. Until then, demo it from your laptop.

Demo day (9:00–12:00):

  1. Record a 3-minute screen video: the README numbers, then the app catching a scam and passing a genuine message.
  2. Write five lines on what you learned this week.
  3. Send both to Claude with the Friday check-in.

Deliverable checklist

  • Repo on GitHub, set up with uv and ruff, with a commit every weekday
  • Keyword baseline with precision and recall on the full dataset
  • A 30-message test set including 6 messages you wrote
  • An LLM checker with structured output, retries and a safe fallback
  • A README table comparing both checkers on the same 30 messages
  • A Streamlit page running locally
  • Demo video, five lines, and the Friday check-in sent

If you have time left: run compare.py with model="sonnet" passed through to call_claude and see whether the bigger model changes the numbers enough to justify being slower.