Ask the Book, Not the Internet

Chunking: the paragraph is the wrong unit, and so is the chapter

The median paragraph in the main corpus is 91 tokens. Embed that and the vector describes its wording, not its idea. Embed a whole chapter and the vector sits close to everything and decisive about nothing. Several paragraphs, landing most chunks in a 550–750 token band, is where retrieval actually works.

Chunks never cross a heading. Several source files each contain 10–18 separate personal accounts. Chunking by file would have spliced the end of one person's story onto the start of another's. The file boundary is not the topic boundary; the heading is.

Overlap is whole paragraphs, never a token slice. A passage split across a boundary is otherwise retrievable from neither side. Carrying trailing paragraphs forward means no chunk ever opens mid-sentence, and because the duplicated region is always byte-identical, the reranker can later strip it with exact equality instead of a fuzzy match.

Result over the ingested corpus: 557 chunks, average 553 tokens, from ~308,000 tokens of source text across 60 sections and 764 printed pages.

Why hybrid search: the two arms fail in opposite directions

This is the single most important finding of the project, and it came from a real miss. One book contains the simile "We are like the passengers of a great liner the moment after rescue from shipwreck." Asking about "the great liner" did not return that passage in the top 50 vector results. Nothing errored. The system simply said the book doesn't mention it, confidently, and falsely.

The cause is structural: one vector represents ~700 tokens, so the chunk's vector describes its subject and a vivid phrase mentioned once contributes almost nothing to it. Dense retrieval averages away exactly the rare wording that makes a passage findable.

Three refinements turned "run both" into something that actually ranks well: a hand-rolled IDF score instead of Postgres' ts_rank (which has no notion of inverse document frequency), phrase-level scoring so stopword phrases like "one day at a time" become searchable addresses, and treating a decisively rare-term hit as an address rather than a vote, since plain reciprocal rank fusion can rank a chunk that's mediocre in both arms above the only chunk in the book containing the word that was actually asked about.

RRF_K = 60            # from the original paper; damps rank 1 vs rank 2
STRONG_LEXICAL = 4.5  # IDF at which a term identifies a passage, not describes it

def fuse(dense_ids, lexical):
    """Fuse two rankings — but let a decisively rare term beat consensus."""
    ranked_lexical = sorted(lexical, key=lexical.get, reverse=True)
    fused = defaultdict(float)
    for ranking in (dense_ids, ranked_lexical):
        for rank, pk in enumerate(ranking, start=1):
            # Each arm votes by position, not by score: cosine distance and
            # summed IDF share no scale, and blending the raw numbers is a
            # fudge factor pretending to be a weighting.
            fused[pk] += 1 / (RRF_K + rank)

    def rank_key(pk):
        score = lexical.get(pk, 0.0)
        strong = score >= STRONG_LEXICAL
        # A rare-term hit is an address, and IDF says so on an absolute scale.
        # It sorts first; RRF only orders what neither arm feels strongly about.
        return (strong, score if strong else 0.0, fused[pk])

    return sorted(fused, key=rank_key, reverse=True)

Keep it in Postgres

There's no separate vector database, no search service, and no external index. Source text, embeddings, full-text vectors, glossary, trigram indexes, chat history and the audit trail are all in one PostgreSQL instance.

The measurement that justified the design: computing to_tsvector per query was a sequential scan at 198ms against 557 chunks, so an ordinary eight-word question spent roughly 1.6 seconds just deciding which chunks held its words, the arm that exists to be the cheap one cost more than the embedding call it was helping. Persisting the vector in a generated column and indexing it with GIN turned that into an index lookup, ~30× faster immediately and asymptotically better as the corpus grows.

Grounding is a string problem, not a model problem

Verifying a quote doesn't need an LLM. The passage is on disk, the quote is a string, and whether one contains the other is a fact. The real difficulty is that "contains" isn't string equality when the text was typeset decades ago and the model types ASCII.

Both sides are folded through one normaliser, quote marks, the full dash range, whitespace, and both the model's "…" and the book's own ". . ." collapsed to a single sentinel, so a typographically different but honest quote passes, and an invented sentence under a correct citation still fails. That distinction is the one the whole system is built to make.

Retrieval is the product. Prompting is a rounding error.

Every meaningful quality jump came from the retrieval layer: hybrid search, IDF weighting, phrase matching, the rare-term override, window widening, the cross-encoder. Prompt changes moved tone. They never once rescued a bad set of passages, and nothing downstream of retrieval can.

Vector search fails silently, and silence is the dangerous failure.

A wrong vector width errors loudly at the column. A right width from the wrong model inserts happily and returns confident nonsense forever. A missed passage produces "the book does not mention that," which sounds authoritative and is false. I designed against silence specifically: stamping every vector with its model and a content checksum, refusing batches of the wrong width, and treating "found nothing" as a claim that has to be earned.

Two retrievers are not redundancy; they are coverage.

I expected the lexical arm to be a cheap fallback. It is not — it is the only arm that can find a phrase mentioned once, and the dense arm is the only one that can find an idea the reader paraphrased. They miss in opposite directions, which is the entire argument for having both.

Fusion can be worse than either input.

RRF quietly assumes its inputs are comparable rankings. When one arm produces an absolute signal, "this is the only chunk in the book with this word," flattening it into a rank throws away the strongest evidence available. Knowing when not to normalise two scores onto a common scale mattered more than any weighting I tried.

Thresholds cannot distinguish a typo from a corruption.

Measured against the glossary: sponser → sponsor scores 0.455, while good → make good and will → self will both score 0.500. The correction I wanted scored lower than the corruptions I had to avoid. The fix was not a better threshold but a different question: "is this word absent from the book at all?"

Verification belongs outside the model.

Asking a model whether it grounded its answer is asking the defendant for a verdict. Checking a string against a passage is deterministic, instant, free, and it caught the failure that every prompt-level instruction had missed: a fluent, invented sentence under a correct citation.

Labelling authority in the prompt is load-bearing.

Four kinds of text reach the model: the book, our definitions, our claim that the reader's word maps to the book's word, and our commentary. Merging any of the last three into the first produces an answer that cannot tell "the book says" from "we think this means."

Document the failing case, not the fix.

Nearly every non-obvious constant in this codebase sits next to the measurement or the specific broken query that set it. Months later, that is the difference between a number I can revisit and a number I am afraid to touch.

Domain structure beats model scale.

A heading boundary, a whole-paragraph overlap, a page label, a 229-term human-written glossary. None of it is machine learning, all of it is cheap, and collectively it outperformed every attempt to solve the same problems with a bigger model or a longer prompt.