Skip to content

Week 4 — RAG over known scam patterns

Why this week: so far the checker relies on whatever the model happens to know about scams. This week it looks up known Singapore scam patterns first, and names the one it matched, with a source. That makes answers easier to trust and easier to check. It also needs a database, which brings your first Docker.

By Friday you have: Postgres with pgvector running under docker compose; a knowledge base of 12 or so Singapore scam patterns written from public advisories; hybrid search (keywords plus meaning) that returns the closest patterns for any message; recall numbers for 2 chunk sizes × 2 embedding models on 30 test messages you wrote; and an agent whose answers cite the matched pattern and its source.

Main reading: Anthropic's Introducing Contextual Retrieval and the pgvector-python hybrid search example. There's no single course this week; each day lists its own reading.

Concepts

Container and image. An image is a packaged program plus everything it needs to run (here: Postgres, already set up with the pgvector extension). A container is one running copy of an image. It's isolated from your laptop, so you don't install Postgres on your Mac at all; you start and stop a container. Docker is the tool that downloads images and runs containers.

Docker Compose. A file, docker-compose.yml, that describes the containers your project needs and how they're set up. docker compose up reads it and starts them. Anyone who clones your repo gets the same database with one command. In week 9 your API joins the same file.

A database as a service. Until now your data lived in CSV files that Python read directly. Postgres is different: it's a separate program that keeps running and waits for connections. Your Python code connects to it, sends SQL, and gets rows back. Several programs can use it at once.

Port. A numbered door on a computer that a program listens on. Postgres listens on port 5432. The container has its own ports, so compose "publishes" the container's 5432 on your laptop's 5432. Your Python code then connects to localhost:5432, meaning "this laptop, door 5432".

Volume. Containers are meant to be thrown away and recreated. A volume is storage Docker keeps outside the container, so your database survives a restart. Delete the volume and the data goes with it.

Environment variable. A named setting that a program reads from its surroundings instead of from its code, such as DATABASE_URL. Your code uses a sensible default for your laptop; in week 9 the cloud server sets a different value and the code doesn't change.

RAG (retrieval-augmented generation). Before the model answers, you search your own documents and give it the most relevant pieces. The model then answers from those pieces, not from memory alone, and can say where the answer came from.

Embedding. A list of numbers (here 384 of them) that represents the meaning of a piece of text. Texts with similar meaning get similar lists, even with no words in common: "earn money liking videos" lands near "paid to rate apps from home". An embedding model turns text into these numbers. You'll use small free models that run on your laptop.

Chunking. Splitting documents into pieces before embedding them. Small chunks are precise but can lose context ("this chunk is about which scam?"). Large chunks keep context but blur several ideas together. You'll measure two sizes rather than guess.

Keyword search and vector search. Keyword (full-text) search finds chunks that share words with the message; it's good at exact names like "SingPost" or "AnyDesk". Vector search finds chunks with the closest embeddings; it's good at the same idea in different words. Each misses things the other catches.

Hybrid search and reciprocal rank fusion (RRF). Run both searches, then merge the two ranked lists. RRF gives each chunk a score of 1 / (60 + its rank) in each list and adds them up. A chunk near the top of both lists wins. It uses ranks, not raw scores, so you never have to compare a keyword score with a vector distance.

Recall@k. For each test message you know the right pattern. Recall@5 is the share of test messages where the right pattern appears in the top 5 results. It's the same recall you know, applied to search: of the things you should have found, how many did you find?

Monday — Postgres and pgvector under Docker

Learn (75 min):

Build:

  1. Check Docker is installed and running. You may already have Docker Desktop; if not, install it from Install Docker Desktop on Mac (free for personal use). Open the Docker app, wait for it to say it's running, then:

    docker --version
docker compose version
docker run --rm hello-world

    The last command downloads a tiny image and prints "Hello from Docker!". If it does, Docker works.

  2. Write docker-compose.yml in the project root:

    services:
  db:
    image: pgvector/pgvector:0.8.7-pg18-trixie
    environment:
      POSTGRES_USER: scam
      POSTGRES_PASSWORD: scam
      POSTGRES_DB: scam_checker
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql

volumes:
  pgdata:

    Line by line:

    • image: the official pgvector image, which is Postgres 18 with pgvector 0.8.7 already installed. Pinning the exact version means your database doesn't change under you. The tags are listed on Docker Hub.
    • environment: environment variables the image reads the first time it starts, to create a user, a password and an empty database.
    • ports: laptop port 5432 → container port 5432.
    • volumes: keep the data in a volume called pgdata. From Postgres 18 the image expects it at /var/lib/postgresql; older guides say /var/lib/postgresql/data, which is for version 17 and below.

    A password in a file?

    scam/scam is fine for a database that only exists on your laptop and holds public advisories. Never reuse it for anything real. Week 9 moves real secrets into a secret manager.

  3. Start it and check it's up:

    docker compose up -d
docker compose ps
docker compose logs db

    -d runs it in the background. ps should show the db service running. Near the end of the logs you should see database system is ready to accept connections. The first up downloads the image, which takes a minute or two.

  4. Open a SQL prompt inside the container and switch on pgvector:

    docker compose exec db psql -U scam -d scam_checker

    CREATE EXTENSION IF NOT EXISTS vector;
SELECT extversion FROM pg_extension WHERE extname = 'vector';
\q

    You should see 0.8.7. docker compose exec db means "run this command inside the db container", so you don't need Postgres installed on your Mac.

  5. Connect from Python. Add the Postgres driver (psycopg, version 3) and pgvector's Python helper:

    uv add "psycopg[binary]" pgvector
uv run python -c "import psycopg; print(psycopg.connect('postgresql://scam:scam@localhost:5432/scam_checker').execute('SELECT version()').fetchone()[0])"

    You should see a line starting PostgreSQL 18. Read the connection string as: postgresql:// user scam, password scam, at localhost port 5432, database scam_checker.

  6. Learn the four commands you'll use every day:

    Command What it does
    docker compose up -d Start the database (or do nothing if it's already running)
    docker compose stop Stop it; the data stays
    docker compose down Remove the container; the data stays in the volume
    docker compose down -v Remove the container and the volume: an empty database next time

    Docker Desktop must be running for any of them. After a laptop restart, open Docker Desktop, then docker compose up -d.

  7. Commit docker-compose.yml, pyproject.toml and uv.lock, and copy this week's checklist into the README.

Done when: docker compose ps shows the database running, SELECT extversion ... returns 0.8.7, and the Python one-liner prints a Postgres version.

Stuck?
  • Cannot connect to the Docker daemon or failed to connect to the docker API: Docker Desktop isn't running. Open it from Applications, wait until it says it's running, and try again.
  • port is already allocated or address already in use on 5432: something else is using the port, often a Postgres installed with Homebrew. See what it is with lsof -i :5432. Either stop it (brew services stop postgresql) or change the compose line to "5433:5432" and use localhost:5433 in your connection string. From Tuesday, your code reads the connection string from the DATABASE_URL environment variable if it's set, so you can run export DATABASE_URL=postgresql://scam:scam@localhost:5433/scam_checker in the terminal instead of editing code.
  • extension "vector" is not available: the image is plain postgres, not pgvector/pgvector. Fix the image: line, then docker compose down -v and up -d again.
  • password authentication failed for user "scam": the user and password are only set the first time the volume is created. If you changed them afterwards, docker compose down -v to start fresh (you lose the data, which is fine this early).
  • The container stops straight away and docker compose logs db starts Error: in 18+, these Docker images are configured to store database data in a format which is compatible with "pg_ctlcluster": the volume is mounted at /var/lib/postgresql/data, as older guides show. For Postgres 18 use /var/lib/postgresql, then down -v and up -d.
  • connection refused from Python straight after up -d: Postgres takes a few seconds to start. Wait for the "ready to accept connections" log line.

Tuesday — The knowledge base, chunks and embeddings

Learn (75 min):

Build:

  1. Collect the knowledge base, by hand. ScamShield's Understanding scam types page links to about a dozen scam types (job, phishing, government officials impersonation, investment and so on), each with how it works, red flags and what to do. The police news room has advisories on newer variants.

    Collect it responsibly:

    • Read the pages in your browser. Don't write a scraper; a dozen pages don't need one.
    • Write each pattern in your own words, in a few short sections. ScamShield's terms of use allow personal use and don't allow reproducing or scraping its content, and your repo is public. Summarising the facts (how the scam works, the signs, the advice) in your own words, with a link, is the safer approach.
    • Put the source URL in every file, so every answer can cite it.
    • Note the page's "last updated" date in your README. Scam advice changes.

    One file per pattern in data/knowledge_base/, named after its id. Here is data/knowledge_base/job_scam.md:

    # Job scam

id: job_scam
source: https://www.scamshield.gov.sg/i-want-protection-from-scams/learn-to-recognise-scams/job-scams/

## How it works

The victim is offered an online job they can do from home, such as liking social media posts, reviewing hotels or restaurants, completing surveys, rating apps or "boosting" product listings, for a commission. Some "jobs" ask the victim to move money through bank accounts the scammer provides. Small commissions may be paid at first. Then the victim is asked to pay in larger amounts for higher earnings, and can't withdraw.

## Tell-tale signs

- Easy money for simple tasks done from home
- You must pay upfront, or use your own money to complete tasks
- You are asked to use your personal bank account for the job
- You are asked for your Singpass details, or to scan a Singpass QR code, to apply

## What to do

Check the job offer by contacting the company directly. Never pay to get or do a job. Call the ScamShield Helpline on 1799 if unsure.

    Aim for 12 files of 130–200 words each. Use the same ids as your week 3 data/scam_patterns.json where they overlap, so nothing else has to change. Splitting phishing into bank_phishing and parcel_delivery_phishing is worth it: they look different in an SMS.

    Is this knowledge base too small for RAG?

    Yes, honestly. The Contextual Retrieval article says a knowledge base under about 200,000 tokens can simply go into the prompt whole, and yours is about 2,500. You're building retrieval anyway because it's the skill (week 6 gives you thousands of synthetic scam reports, which won't fit), and because a separate search step is something you can measure on its own.

  2. Add the embedding library. It brings PyTorch with it, so the download is large the first time:

    uv add sentence-transformers

  3. The first half of src/scam_checker/retrieval.py: read the files, chunk them, embed the chunks and store them in Postgres.

    """Hybrid search over known scam patterns: Postgres full-text search plus pgvector."""

import os
from dataclasses import dataclass
from functools import cache
from pathlib import Path

import psycopg
from pgvector.psycopg import register_vector
from sentence_transformers import SentenceTransformer

KB_DIR = Path("data/knowledge_base")
DATABASE_URL = os.environ.get(
    "DATABASE_URL", "postgresql://scam:scam@localhost:5432/scam_checker"
)
DEFAULT_MODEL = "BAAI/bge-small-en-v1.5"
DEFAULT_CHUNK_WORDS = 150


@dataclass
class Pattern:
    pattern_id: str
    name: str
    source: str
    body: str


@dataclass
class Match:
    pattern_id: str
    name: str
    source: str
    chunk: str  # the best-matching piece of text
    score: float


def load_patterns(kb_dir: Path = KB_DIR) -> list[Pattern]:
    """Read every .md file: a '# Name' line, 'id:' and 'source:' lines, then the body."""
    patterns = []
    for path in sorted(kb_dir.glob("*.md")):
        lines = path.read_text().splitlines()
        name = lines[0].removeprefix("# ").strip()
        fields = {}
        body_start = 1
        for i, line in enumerate(lines[1:], start=1):
            if line.startswith("## "):
                body_start = i
                break
            if ":" in line:
                key, value = line.split(":", 1)
                fields[key.strip()] = value.strip()
        body = "\n".join(lines[body_start:]).strip()
        patterns.append(Pattern(fields["id"], name, fields["source"], body))
    return patterns


def chunk_words(text: str, size: int, overlap: int = 10) -> list[str]:
    """Split text into pieces of about `size` words, each overlapping the last."""
    ...  # your turn, step 4


@cache
def get_model(model_name: str = DEFAULT_MODEL) -> SentenceTransformer:
    """Load an embedding model once; the first call downloads it."""
    return SentenceTransformer(model_name)


def connect() -> psycopg.Connection:
    conn = psycopg.connect(DATABASE_URL, autocommit=True)
    conn.execute("CREATE EXTENSION IF NOT EXISTS vector")
    register_vector(conn)
    return conn


def build_index(
    chunk_size: int = DEFAULT_CHUNK_WORDS,
    model_name: str = DEFAULT_MODEL,
    add_title: bool = False,
) -> int:
    """Rebuild the chunks table from the knowledge base. Returns the number of chunks."""
    model = get_model(model_name)
    rows = []
    for p in load_patterns():
        for chunk in chunk_words(p.body, chunk_size):
            content = f"{p.name}: {chunk}" if add_title else chunk
            rows.append((p.pattern_id, p.name, p.source, content))
    embeddings = model.encode([r[3] for r in rows], normalize_embeddings=True)

    with connect() as conn:
        conn.execute("DROP TABLE IF EXISTS chunks")
        conn.execute(
            f"""CREATE TABLE chunks (
                id bigserial PRIMARY KEY,
                pattern_id text,
                name text,
                source text,
                content text,
                embedding vector({model.get_embedding_dimension()})
            )"""
        )
        conn.execute(
            "CREATE INDEX ON chunks USING GIN (to_tsvector('english', content))"
        )
        with conn.cursor() as cur:
            cur.executemany(
                "INSERT INTO chunks (pattern_id, name, source, content, embedding)"
                " VALUES (%s, %s, %s, %s, %s)",
                [(*row, emb) for row, emb in zip(rows, embeddings)],
            )
    return len(rows)


if __name__ == "__main__":
    print(build_index(), "chunks indexed")

    What the less obvious parts do:

    • os.environ.get("DATABASE_URL", ...): use the environment variable if it's set, otherwise the laptop default.
    • @cache: loading a model takes a few seconds, so do it once and reuse it.
    • register_vector(conn): teaches psycopg to send a NumPy array as a pgvector vector.
    • normalize_embeddings=True: scales every embedding to length 1, so cosine distance behaves well.
    • vector(...): a column of 384 numbers, the size asked from the model rather than typed in, so swapping models can't silently break it.
    • The GIN index on to_tsvector('english', content): makes keyword search fast. to_tsvector turns text into stemmed words ("scams" → "scam") and drops words like "the".
    • build_index drops and rebuilds the table each time. With 12 documents that takes seconds, and it means the table always matches the files.

  4. Your turn: write chunk_words. It should split text on whitespace, take size words at a time, and start each new chunk overlap words before the previous one ended, so a sentence cut at a boundary still appears whole in one chunk. With 100 words, size=40 and overlap=10, you should get chunks of 40, 40, 40 and 10 words, the second starting at word 30.

    Stuck? One version of chunk_words 
    def chunk_words(text: str, size: int, overlap: int = 10) -> list[str]:
    """Split text into pieces of about `size` words, each overlapping the last."""
    words = text.split()
    step = max(size - overlap, 1)
    return [" ".join(words[i : i + size]) for i in range(0, len(words), step)]

    range(0, len(words), step) gives the start of each chunk: 0, 30, 60, 90. max(..., 1) stops an endless loop if someone sets the overlap bigger than the size.

  5. Build the index and look at it:

    uv run python -m scam_checker.retrieval

    With 12 files of about 150 words you'll see roughly 18 chunks indexed. The first run downloads the model, about 130 MB. Then look inside:

    docker compose exec db psql -U scam -d scam_checker -c "SELECT pattern_id, left(content, 60) FROM chunks LIMIT 5;"

  6. Test the pieces that don't need a database, in tests/test_retrieval.py:

    from scam_checker.retrieval import chunk_words, load_patterns


def test_chunks_overlap_and_cover_every_word():
    text = " ".join(str(i) for i in range(100))
    chunks = chunk_words(text, size=40, overlap=10)
    assert [len(c.split()) for c in chunks] == [40, 40, 40, 10]
    assert chunks[1].split()[0] == "30"  # starts 10 words before chunk 0 ended


def test_every_pattern_has_an_id_and_a_source():
    patterns = load_patterns()
    assert len(patterns) >= 10
    assert all(p.pattern_id and p.source.startswith("https://") for p in patterns)
    assert len({p.pattern_id for p in patterns}) == len(patterns)

    uv run pytest -q

  7. Your turn: write the 30 test messages now, before search exists, so you can't tune them to what the search happens to find. data/rag_queries.csv has two columns, the message and the pattern id it should match, at least two per pattern:

    text,expected_pattern
"Hi! We are hiring part-time staff. Earn $200-$500 daily just by liking YouTube videos. No experience needed. Add our HR on WhatsApp",job_scam
"Congrats, you completed 3 tasks today! To unlock your commission of $680 please top up $300 to the merchant account first",job_scam
"SingPost: Your parcel could not be delivered due to an incomplete address. Update within 24h: singpost-sg.delivery",parcel_delivery_phishing
"Bro it's me, new number. Can lend me $2,000 for my renovation contractor? Pay you back Friday.",fake_friend_call

    Write them the way scammers do: short, misspelt, Singlish where it fits, and not using the words in your knowledge base files. A test set that copies the documents' wording measures nothing.

Done when: uv run python -m scam_checker.retrieval prints a chunk count, the tests pass, and data/rag_queries.csv has 30 rows covering every pattern.

Stuck?
  • KeyError: 'id' in load_patterns: a file is missing its id: line, or the line is below the first ## heading. The header lines must come before the first section.
  • psycopg.OperationalError: connection ... failed: the database isn't running. docker compose up -d.
  • A warning about HF_TOKEN or unauthenticated requests when the model downloads: harmless. You don't need a Hugging Face account for these models.
  • different vector dimensions 384 and 768: you changed DEFAULT_MODEL to a model with bigger embeddings but are searching a table built with the old one. Rebuild with uv run python -m scam_checker.retrieval.
  • The download is very slow: PyTorch is several hundred MB. Do it on the library Wi-Fi once; after that it's cached.

Wednesday — Hybrid search with reciprocal rank fusion

Learn (65 min):

Build:

  1. See the two searches separately first, so you know what you're fusing. scripts/two_searches.py:

    """Show keyword search and vector search side by side for one message."""

import sys

from scam_checker.retrieval import connect, get_model

text = sys.argv[1]
embedding = get_model().encode(text, normalize_embeddings=True)

with connect() as conn:
    print("Vector search (closest meaning):")
    for pattern_id, distance in conn.execute(
        "SELECT pattern_id, embedding <=> %s FROM chunks ORDER BY 2 LIMIT 5",
        (embedding,),
    ):
        print(f"  {distance:.3f}  {pattern_id}")

    query = conn.execute(
        "SELECT replace(plainto_tsquery('english', %s)::text, ' & ', ' | ')", (text,)
    ).fetchone()[0]
    print(f"\nKeyword search for {query}:")
    for pattern_id, rank in conn.execute(
        "SELECT pattern_id, ts_rank_cd(to_tsvector('english', content), %s::tsquery)"
        " FROM chunks WHERE to_tsvector('english', content) @@ %s::tsquery"
        " ORDER BY 2 DESC LIMIT 5",
        (query, query),
    ):
        print(f"  {rank:.3f}  {pattern_id}")

    uv run python scripts/two_searches.py "SingPost: your parcel is on hold due to unpaid fees. Pay \$1.99 at singpost-redelivery.com"

    On the example knowledge base this prints:

    Vector search (closest meaning):
  0.205  parcel_delivery_phishing
  0.367  ecommerce_scam
  0.379  loan_scam
  0.380  job_scam
  0.385  insurance_scam

Keyword search for 'singpost' | 'parcel' | 'hold' | 'due' | 'unpaid' | 'fee' | 'pay' | '1.99' | 'singpost-redelivery.com':
  1.100  parcel_delivery_phishing
  0.500  ecommerce_scam
  0.500  loan_scam
  0.300  job_scam
  0.100  investment_scam

    <=> is cosine distance: smaller is closer. Notice the words have been stemmed ("fees" became "fee"). Try three of your own test messages and find one where the two lists disagree.

    Why | instead of &

    The pgvector example uses plainto_tsquery, which joins the words with & (AND): a chunk must contain every word. That suits a two-word search like "growling bear". Your query is a whole SMS of 20–40 words, and no chunk contains all of them, so keyword search would find nothing. Swapping & for | (OR) finds chunks that share any words, and ts_rank_cd ranks the ones that share more of them higher.

  2. Add hybrid search to retrieval.py, below build_index and above the if __name__ block. The SQL is the pgvector example with two changes, marked in the comment:

    # Adapted from the pgvector-python RRF example. Two changes: the keyword query
# uses OR between words (a 30-word message rarely contains *every* word of a
# chunk), and we return 50 chunks so we can group them by pattern afterwards.
HYBRID_SQL = """
WITH or_query AS (
    SELECT replace(plainto_tsquery('english', %(query)s)::text, ' & ', ' | ')::tsquery AS query
),
semantic_search AS (
    SELECT id, RANK () OVER (ORDER BY embedding <=> %(embedding)s) AS rank
    FROM chunks
    ORDER BY embedding <=> %(embedding)s
    LIMIT 20
),
keyword_search AS (
    SELECT id, RANK () OVER (ORDER BY ts_rank_cd(to_tsvector('english', content), query) DESC)
    FROM chunks, or_query
    WHERE to_tsvector('english', content) @@ query
    ORDER BY ts_rank_cd(to_tsvector('english', content), query) DESC
    LIMIT 20
)
SELECT
    chunks.pattern_id, chunks.name, chunks.source, chunks.content,
    COALESCE(1.0 / (%(k)s + semantic_search.rank), 0.0) +
    COALESCE(1.0 / (%(k)s + keyword_search.rank), 0.0) AS score
FROM semantic_search
FULL OUTER JOIN keyword_search ON semantic_search.id = keyword_search.id
JOIN chunks ON chunks.id = COALESCE(semantic_search.id, keyword_search.id)
ORDER BY score DESC
LIMIT 50
"""


def search_patterns(
    text: str, k: int = 5, model_name: str = DEFAULT_MODEL
) -> list[Match]:
    """Return the k known scam patterns closest to `text`, best first."""
    embedding = get_model(model_name).encode(text, normalize_embeddings=True)
    with connect() as conn:
        rows = conn.execute(
            HYBRID_SQL, {"query": text, "embedding": embedding, "k": 60}
        ).fetchall()

    ...  # your turn, step 3

    Read the SQL as three steps. semantic_search ranks chunks by meaning, keyword_search ranks them by shared words, each keeping its top 20. The final SELECT joins the two lists; FULL OUTER JOIN keeps a chunk even if only one search found it, and COALESCE(..., 0.0) scores the missing side as zero. The %(k)s here is RRF's constant 60, not your k results. The extra JOIN chunks fetches each chunk's pattern and text.

  3. Your turn: finish search_patterns. The rows come back best first, one per chunk, so one pattern can appear several times. Return one Match per pattern, using its best chunk, and at most k of them.

    Stuck? The grouping step 
        matches: dict[str, Match] = {}
    for pattern_id, name, source, content, score in rows:
        if pattern_id not in matches:  # rows are best first, so keep the first
            matches[pattern_id] = Match(pattern_id, name, source, content, float(score))
    return list(matches.values())[:k]

    A Python dict remembers the order keys were added, so the result stays best first.

  4. Replace the if __name__ block at the bottom so it also searches:

    if __name__ == "__main__":
    print(build_index(), "chunks indexed")
    for m in search_patterns(
        "Earn $300 a day liking TikTok videos, contact HR on WhatsApp"
    ):
        print(f"{m.score:.4f}  {m.pattern_id:26}  {m.chunk[:60]}")

    uv run python -m scam_checker.retrieval

    The job scam pattern should come first. Scores near 0.033 are the maximum possible (1/61 + 1/61): top of both lists.

  5. Try five of your test messages through search_patterns in a Python prompt (uv run python). Write down one where hybrid beats vector-only and one where keyword matches drag in the wrong pattern (on the example knowledge base, "WhatsApp" and "video" pull in loan and government scams).

Done when: search_patterns(text) returns up to 5 distinct patterns, best first, and you can explain in two sentences what RRF does.

Stuck?
  • syntax error at or near "::": a cast like ...::tsquery can't sit directly in a FROM list. That's why the OR query lives in its own or_query step.
  • cannot adapt type 'ndarray' using placeholder '%s': register_vector(conn) didn't run, so psycopg doesn't know how to send an embedding. Always connect through connect().
  • Keyword search finds nothing for any message: check you swapped & for |. Print the query as two_searches.py does.
  • A harmless message such as "Hi, are you there?" still returns five patterns. That's expected: search always returns the closest patterns, even when none is close. Deciding whether a match actually fits is the model's job on Friday.
  • Everything matches the same pattern: check that normalize_embeddings=True is in both build_index and search_patterns, then rebuild.

Thursday — Measure retrieval: recall@5

Learn (40 min):

  • Hamel Husain and Shreya Shankar's How should I approach evaluating my RAG system? (15 min): measure retrieval as a search problem, separately from the answer
  • Back to the MTEB Leaderboard (15 min): find your two models and note their retrieval scores. Today you'll see whether that ranking holds on your own data
  • Re-read the "chunk size" part of the Pinecone article (10 min)

Build:

  1. Your turn: write scripts/eval_retrieval.py. For each of 2 models × 2 chunk sizes: rebuild the index, run every test message through search_patterns(text, k=5, model_name=...), and count how often the expected pattern is first (recall@1) and anywhere in the top 5 (recall@5). Start from this and fill in the inner loop:

    """Recall@5 for 2 chunk sizes x 2 embedding models, on data/rag_queries.csv."""

import pandas as pd

from scam_checker.retrieval import build_index, search_patterns

CHUNK_SIZES = [40, 150]
MODELS = ["sentence-transformers/all-MiniLM-L6-v2", "BAAI/bge-small-en-v1.5"]

queries = pd.read_csv("data/rag_queries.csv")
results = []
for model_name in MODELS:
    for chunk_size in CHUNK_SIZES:
        n_chunks = build_index(chunk_size=chunk_size, model_name=model_name)
        hits_at_1, hits_at_5 = [], []
        ...  # your turn: one True/False per query in each list
        results.append(
            {
                "model": model_name.split("/")[-1],
                "chunk_words": chunk_size,
                "chunks": n_chunks,
                "recall@1": sum(hits_at_1) / len(queries),
                "recall@5": sum(hits_at_5) / len(queries),
            }
        )

table = pd.DataFrame(results)
print(table.round(2).to_string(index=False))
table.to_csv("data/retrieval_results.csv", index=False)

    40 words is about one section of a pattern file; 150 is nearly the whole file.

    Stuck? The inner loop 
            for text, expected in zip(queries["text"], queries["expected_pattern"]):
            found = [
                m.pattern_id for m in search_patterns(text, k=5, model_name=model_name)
            ]
            hits_at_1.append(found[:1] == [expected])
            hits_at_5.append(expected in found)

    found[:1] == [expected] is safe even when nothing was found; found[0] would crash.

  2. Run it. It takes well under a minute after the models have downloaded:

    uv run python scripts/eval_retrieval.py

    On the example knowledge base (12 patterns) and 30 test messages, it gave:

                model  chunk_words  chunks  recall@1  recall@5
 all-MiniLM-L6-v2           40      64      0.70      1.00
 all-MiniLM-L6-v2          150      18      0.73      1.00
bge-small-en-v1.5           40      64      0.77      0.97
bge-small-en-v1.5          150      18      0.83      1.00

    Your numbers will differ, because your files and messages differ.

  3. Your turn: read your table like a sceptic and write a short paragraph in the README:

    • Recall@5 is nearly useless here. With 12 patterns, the top 5 is 40% of the whole knowledge base, so almost anything scores 1.00. That's why the script also reports recall@1, which still separates the configurations. Say this in interviews; it shows you understand the metric, not just the formula.
    • 30 messages is a small sample. One message is 3.3 percentage points. 0.77 against 0.83 is two messages. Before you trust a difference, open the messages that changed.
    • Pick a winner and say why. On the example run, bge-small with 150-word chunks was best on both measures. Set DEFAULT_MODEL and DEFAULT_CHUNK_WORDS at the top of retrieval.py to your winner and rebuild:
    uv run python -m scam_checker.retrieval

  4. Find the messages that failed at recall@1 for your winner and read them. Is the expected label wrong, is the message genuinely ambiguous (a fake parcel fee is both phishing and e-commerce), or is a knowledge base file missing a sign? Fix the files, not the test messages, unless the label was a mistake.

  5. Try the cheapest form of contextual retrieval: prefix every chunk with its pattern name, so a chunk that says "you must pay upfront" also says "Job scam". build_index(add_title=True) does it. Run your winner with and without it and add the row to your README. On the example knowledge base it moved bge-small at 40 words from 0.77 to 0.80 at recall@1, a single message. Small chunks gain the most, because they lose the most context.

Done when: your README has the 2 × 2 table, a paragraph on what it does and doesn't show, and the defaults in retrieval.py match the winner.

Stuck?
  • Every number is 1.00: your test messages copy the knowledge base's wording, or there are too few patterns. Rewrite five messages the way a real scammer would.
  • KeyError: 'expected_pattern': check the CSV header is exactly text,expected_pattern, with no spaces.
  • An expected pattern id has a typo, so it can never match: check with set(queries["expected_pattern"]) - {p.pattern_id for p in load_patterns()}, which should be empty.
  • The first build_index for MiniLM is slow: it's downloading the second model (about 90 MB). Later runs use the cached copy.

Friday — Cite the pattern in the answer, and demo day

Learn (30 min): Effective context engineering for AI agents (Anthropic, 30 min). Two ideas to take away: give the model "the smallest possible set of high-signal tokens" (three matched patterns, not the whole knowledge base), and let an agent fetch context "just in time" with a tool, which is exactly what lookup_scam_pattern does.

Build (before 9:00 or after demo day):

  1. Switch the agent's lookup_scam_pattern tool to retrieval. Find the function behind it in src/scam_checker/tools/ (week 3; your MCP server in mcp_server.py should call the same function, not a copy). Keep its name, argument and docstring style, and replace the body so it calls search_patterns:

    from scam_checker.retrieval import search_patterns


def lookup_scam_pattern(text: str) -> dict:
    """Find the known Singapore scam patterns that best match a message."""
    matches = search_patterns(text, k=3)
    return {
        "matches": [
            {
                "id": m.pattern_id,
                "name": m.name,
                "matched_text": m.chunk,
                "source": m.source,
            }
            for m in matches
        ]
    }

    If your other tools return JSON strings rather than dicts, return json.dumps(...) of the same thing. Update the tool's description in TOOLS to say it returns up to 3 patterns, each with an id, a name, matching text and a source URL. Three, not five: the agent needs the likely candidates, not a list to wade through.

  2. Add a field for the citation to src/scam_checker/models.py:

    class CheckResult(BaseModel):
    verdict: Literal["scam", "not_scam", "unsure"]
    red_flags: list[str] = Field(
        description="Specific warning signs found in the message"
    )
    advice: str = Field(
        description="One or two plain sentences on what the reader should do"
    )
    matched_pattern: str | None = Field(
        default=None,
        description="The id of the known scam pattern this message matches, "
        "copied from lookup_scam_pattern; null if none fits",
    )

    default=None keeps old code and old saved runs working: anything that doesn't set the field gets None.

  3. Add these lines to the agent's system prompt in agent.py:

    Call lookup_scam_pattern once for every message. If one of its matches fits the
message, copy that match's id into matched_pattern. If none fits, leave
matched_pattern null. A close match is not proof: a genuine message can look
like a pattern, so judge the message itself.

    The last sentence matters. Retrieval always returns something; a genuine Grab notification still has a "closest" pattern.

  4. Let code, not the model, write the citation. Add this to retrieval.py above the if __name__ block:

    def citation(pattern_id: str | None) -> str | None:
    """A sentence naming the matched pattern and its source, or None.

    The model only picks an id; this code writes the citation, so it can't
    invent a pattern or a link that isn't in the knowledge base."""
    patterns = {p.pattern_id: p for p in load_patterns()}
    if pattern_id not in patterns:
        return None
    p = patterns[pattern_id]
    return f"This matches a known scam pattern: {p.name}. Source: {p.source}"

    When rehearsing this, asking the model to start its advice with "This matches a job scam: ..." worked some of the time. Asking for an id and building the sentence in code works every time, and an invented id simply shows no citation. Add a test to tests/test_retrieval.py:

    def test_citation_only_for_known_patterns():
    assert "Job scam" in citation("job_scam")
    assert citation("made_up_pattern") is None
    assert citation(None) is None

    (add citation to the import line at the top).

  5. Show it in the app. In app.py, import it and add three lines after the advice:

    from scam_checker.retrieval import citation

        st.subheader("What to do")
    st.write(result.advice)

    cited = citation(result.matched_pattern)
    if cited:
        st.info(cited)

  6. Run the agent on three of your week 1 messages (a job scam, a parcel scam and the Grab message) through the app or a Python prompt. Check the saved runs in runs/: did it call lookup_scam_pattern? Did the Grab message get no citation? In the rehearsal the agent cited job_scam for the job message and nothing for the Grab message. It also sometimes skipped the tool entirely. Note that down; it's exactly what week 5 is for.

  7. uv run ruff format ., uv run ruff check ., uv run pytest -q, and commit.

Demo day (9:00–12:00):

  1. Record a 3-minute screen video: docker compose ps, your 2 × 2 recall table and what you concluded, then the app citing a pattern with its ScamShield link, and a genuine message with no citation.
  2. Write five lines on what you learned this week.
  3. Send both to Claude with the Friday check-in. Put recall@1 and recall@5 for your winner under "My numbers".

Deliverable checklist

  • docker-compose.yml running Postgres 18 with pgvector, committed
  • 12 or so knowledge base files in your own words, each with an id and a source URL
  • retrieval.py with build_index and hybrid search_patterns (full-text plus pgvector, fused with RRF)
  • 30 hand-written test messages in data/rag_queries.csv
  • A README table of recall@1 and recall@5 for 2 chunk sizes × 2 embedding models, with your reading of it
  • lookup_scam_pattern uses retrieval; answers carry matched_pattern and the app shows the citation
  • Tests pass; demo video, five lines, and the Friday check-in sent

If you have time left:

  • Try the full contextual retrieval method from the article: ask Claude for one line of context per chunk and prepend it before embedding. This script does the asking, and saves after every call so a crash loses nothing:

    """Contextual retrieval: ask Claude for one line of context per chunk.

Usage: uv run python scripts/add_context.py 40 3   (chunk size, how many chunks)
"""

import json
import sys
from pathlib import Path

from scam_checker.llm import call_claude
from scam_checker.retrieval import chunk_words, load_patterns

PROMPT = """<document>
{document}
</document>
Here is the chunk we want to situate within the whole document
<chunk>
{chunk}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else."""

chunk_size = int(sys.argv[1])
limit = int(sys.argv[2]) if len(sys.argv) > 2 else None
out_path = Path(f"data/chunk_context_{chunk_size}.json")
contexts = json.loads(out_path.read_text()) if out_path.exists() else {}

todo = [
    (p, chunk)
    for p in load_patterns()
    for chunk in chunk_words(p.body, chunk_size)
    if chunk not in contexts
][:limit]
for i, (p, chunk) in enumerate(todo, start=1):
    document = f"# {p.name}\n\n{p.body}"
    reply = call_claude(PROMPT.format(document=document, chunk=chunk))
    contexts[chunk] = reply.text.strip()
    out_path.write_text(json.dumps(contexts, indent=2))  # save after every call
    print(f"{i}/{len(todo)} {contexts[chunk][:80]}")

    The prompt is the one from the article. Run it on 3 chunks first, then all of them (about 64 calls at 40 words, so roughly 8 minutes and a share of your usage limit). Then change build_index to look up each chunk in the JSON file and prepend its context, and rerun eval_retrieval.py for 40-word chunks. Does it beat the free "prefix the title" trick?

  • Add vector-only and keyword-only rows to the table (two short SQL queries, as in two_searches.py), so you can say whether hybrid search earned its complexity on your data.

  • Read the Contextual Retrieval section on reranking, and write down how you'd test it.