‘What did I write about that contractor last August?’ — actually getting the answer

Engineering track. This post goes deep on Mneme’s recall pipeline — vector retrieval, BM25, Reciprocal Rank Fusion, provenance metadata. If you’re here for the consumer story of how recall feels in daily use, the portable-memory post and the contractor anecdote in the Obsidian post cover the same ground without the implementation depth.

The point at which Mneme earns its keep is the moment recall actually works. Not when it returns a confident-sounding paraphrase. When the answer comes back with the right page from the right document and you can verify it in one click.

This post is about how search is built to give you that — a hybrid vector search pipeline with BM25 rescoring and provenance — and where it still falls short.

The two ways search usually fails

If you’ve used a vector-search-backed AI tool before, you’ve felt both of these.

Failure one: the blur. You ask “who’s Sarah?” and the system returns five chunks about generic “people” and “contacts” because the cosine similarity to Sarah was diluted across a lot of nearby concepts. The exact name didn’t dominate the ranking. You knew the answer was in there. The retrieval didn’t.

Failure two: the confident wrong page. You ask about a clause in a contract and the system returns a chunk that sounds relevant but is from the wrong document. Or it cites “page 3” without the document name. You can’t easily verify it. You re-read the whole contract anyway.

Both failures have the same root: a single retrieval mechanism (pure vector, or pure keyword) is doing more work than it should.

What hybrid vector search actually does

Mneme’s search is hybrid by default. The full pipeline lives in mcp_tools.py and embed_service.py, and it does roughly this:

1. De-bias the query. Strip self-references like “in my Mneme corpus,” “using Mneme,” etc. — they would otherwise nudge the embedding toward a meta-cluster that doesn’t help anyone.
2. Embed the query with Vertex AI’s text-embedding-005 (the same model that embedded your chunks, so the geometry matches).
3. Coarse retrieval via Firestore’s find_nearest() on the chunk vector index. Overfetch — pull, say, 30 candidates instead of the 8 returned at the end.
4. Filter the candidates by folder, tags, from/to dates, and kinds (artifacts, memories, chunks). Filters can’t combine directly with find_nearest, which is why Mneme overfetches first and filters the result set.
5. Rescore using a hybrid that combines BM25 (good for exact terms) with the original cosine ranks via Reciprocal Rank Fusion. This is the step that handles Sarah and late fee and 2026-08-12 — anything where the precise token matters.
6. Return the top results with provenance: source, path, page anchor, artifact ID. The assistant gets enough metadata to cite the answer.

What you experience is “I asked a question and the right paragraph came back with a page number I can click.” What’s happening is two retrieval mechanisms agreeing on which paragraphs to surface.

A worked example

The question that became the title of this post is one I (Mark) actually asked Claude Desktop one morning: “what did I write about that contractor last August?” The corpus had eight months of Obsidian notes ingested via the workflow from post #6, plus the contractor’s bid PDF and a few text-message screenshots ingested as images.

Claude called search with:

{
  "tool": "search",
  "arguments": {
    "query": "what did I write about that contractor",
    "from": "2025-08-01",
    "to": "2025-08-31",
    "limit": 6
  }
}

The date filter narrowed the candidate pool to last August. Vector retrieval surfaced about 30 chunks; the rescorer pushed two notes to the top — a meeting summary from the day the contractor came to walk the property, and a note from the night I drafted my response to the bid. The PDF chunks were ranked lower because the question was about what I wrote, not what was bid.

The answer came back as: “Two relevant notes from August 2025: a walkthrough summary on Aug 12 (Home/Renovation/2025-08-12-walkthrough.md) noting concerns about timeline and bathroom subfloor, and a bid-response draft on Aug 22 (Home/Renovation/2025-08-22-bid-response.md) where you flagged three line items as overpriced. Want me to pull the specifics?”

That’s the moment recall earns its keep. The exact notes, the exact paths, retrievable in one follow-up question.

Filters that change everything

Most of search‘s practical power comes from filters, not from cleverer ranking.

• folder — narrow to a section of your corpus. “What did I decide about pricing?” in folder="Work/Foundry" is a different question than the same query unscoped.
• tags — array filter, useful when your folder structure is broad but your tags are narrow. tags=["lease"] cuts across Home/, Legal/, and Personal/.
• from / to — ISO date strings on content_date. Useful for “what was I working on the week of the launch?” or “what’s the most recent lab result?”
• kinds — restrict to artifacts, memories, or chunks. “Search just my memories” is a different question than “search every chunk” — both are useful at different times.
• limit — default 8. Bump to 20 or 30 when you’re doing a synthesis question. The reranker handles bigger pools without losing precision.

A pattern that’s worth internalizing: the better you get at filters, the better the AI gets at answers. Most “the AI couldn’t find it” experiences are actually “the assistant didn’t think to scope the search.” You can teach yourself to mention scope in the query — “in my Foundry folder,” “from last quarter,” “just memories” — and watch the answer quality jump.

Provenance as a feature

Every chunk Mneme returns carries enough metadata to cite the source: the artifact ID, the source name, the path, and a page or image anchor. The assistant gets to construct a citation like “per Legal/Vendors/foundry-msa.pdf (p.4)” without inventing anything.

This is not a small thing. It is the difference between trusting the answer and trusting the system. “AI says X” is a leap of faith; “AI quotes page 4 of the document and you can verify it” is operational.

If you want to go further, get_artifact returns the full markdown extract for any artifact ID. The chunk anchored to p.4 is one query away from the surrounding context if you want to read more. Full tool reference: docs/mcp-tools.md.

Where search still falls short

Honesty wins here.

Negation is hard. “Show me the notes that don’t mention pricing” is not a vector-search-friendly query. There’s no great answer for negation today; you’ll have to filter manually or list and skim.

Recency bias is your job. Mneme doesn’t decay older content. If a fact in your corpus was true in 2024 and is false now, the search will happily surface the 2024 chunk. This is why save_memory for current facts matters — memories tend to phrase things as “as of April 2026,” and the recency cue helps the assistant choose the right answer.

Long, vague queries underperform short, specific ones. “Tell me everything I should know about the Foundry deal” returns a soup; “what was the last status update on the IP redline?” returns a paragraph. The assistant can ask better questions on your behalf if you teach it to.

Hybrid still isn’t structured. If your real question is “sum the receipts in Tax/2026/“, search isn’t the right tool. list_folder plus arithmetic is. Structured tooling may come later; for now, know what search is for and what it isn’t.

What to try

The best test of recall is to ask a question you already know the answer to, on a corpus you’ve been building for a few weeks. Did the right document come back? Did it carry the page number? Did the assistant cite it accurately?

When the answer is yes — and it will be, more often than you expect — the next step is to lean on it. Stop re-explaining. Start asking.

That’s the engineering deep-dive on recall. The consumer-facing posts cover the same surface area without the implementation depth: portable memory, the four ingest paths, the trust contract. Read the connector docs for the full tool reference, or start your vault and put any one of the recall paths to use today.

The Muses remembered everything. Now your AI can too.