Retrieval and reranking
🏠The everyday picture
A recruiter first shortlists 20 CVs quickly by keywords, then reads those 20 carefully to choose the best five for interview. Fast first, careful second.
📘What it really is
- Two-stage retrieval is common:
- recall stage: hybrid search (Module 85) fetches a generous set of candidates quickly (for example, top 20 to 50);
- rerank stage: a reranker model scores each candidate against the question more carefully, and you keep the best few (for example, top 5).
-
Rerankers (offered by providers like Cohere and Voyage AI, or as open models) read the question and passage together, so they judge relevance better than comparing two separate embeddings. They cost a little time and money per request.
-
Filters before ranking:
- permissions: only chunks the user is allowed to see (by role, department or visibility), applied in the database query itself;
- status: only current, approved versions;
-
optional: department or document type when the question clearly concerns one.
-
How many passages to give the model: enough to include the answer, few enough to keep cost down and avoid distraction (Module 82). Often 3 to 8. Test it.
-
Query improvements:
- for follow-up questions ("and for kurtas?"), rewrite the question into a standalone one using the conversation, before searching;
-
keep the user's original wording too, as exact terms matter.
-
When nothing relevant is found (all scores low), say so rather than passing weak passages and hoping.
🖱️Try it
Change your retrieval to fetch the top 20 by hybrid search (with a permission filter by the user's role), rerank them (with a
reranker API or, as a simple stand-in, the model scoring relevance from 0 to 3), and keep the top 5. Measure hit rate at 5 before
and after reranking. Test that a staff user never retrieves a chunk marked managers-only.
✅Quick check
Why rerank after the first search?
Show the answer
A reranker judges relevance more carefully, so the best passages end up at the top.
Where should permission filtering happen?
Show the answer
In the retrieval query itself, so restricted chunks are never fetched for users who may not see them.