Subscribe for more posts like this →

Deep-dive 6 — Retrieval as a Distributed Read Path

Stackshala

Stackshala

— 3 min read
Share

Consequence from the field guide: measure the retriever independently of the generator; track recall/precision of retrieval on its own; and make "I don't have that information" a first-class, explicitly-rewarded behavior — because the default over a gap is a confident lie.

How it's done

A production retrieval pipeline is a three-stage read path:

  1. Chunk the corpus — structure-aware splitting (headings, function boundaries) with modest overlap by default; child-parent (embed small precise chunks, expand to their parent for context at read time); semantic or "agentic" chunking when prose lacks clean structure and you've measured a gap.
  2. Index and retrieve, hybrid. Dense vector search (semantic recall) plus sparse BM25 (lexical precision for names, IDs, codes). Retrieve top-k from each path.
  3. Fuse and rerank. Combine the two rankings with Reciprocal Rank Fusion (RRF, k≈60) — a rank-based algorithm that sidesteps the score-incompatibility that breaks naive weighted averaging. Then pass the shortlist (top ~100) to a cross-encoder reranker that scores each (query, chunk) pair jointly, and keep only the top 5–10 for the model.

Query transformation (rewriting/expansion) sits in front; JIT retrieval (page content in on demand) is the read-path analog of demand paging.

Tools available

  • Vector DBs with native hybrid: Weaviate, Qdrant (sparse-vector support), Elasticsearch / OpenSearch, Pinecone. If native hybrid isn't available, retrieve top-k per path and fuse outside the DB.
  • Rerankers: Cohere Rerank 3.5, Voyage rerank-2.5 (instruction-following, larger context — strong fresh default, though benchmarks are vendor-run), BGE-reranker-v2-m3 (open source, single-GPU), MS-MARCO cross-encoders (free baseline).
  • Learned sparse: SPLADE-v3.
  • Eval / frameworks: RAGAS (separates retrieval from generation quality), LlamaIndex, Haystack. Metrics: nDCG@k, MRR, recall@k.

Best practices

Hybrid by default — pure dense retrieval consistently misses exact-lexical queries (names, IDs, error codes), and it fails silently. Use RRF, not naive score-weighting. Rerank aggressively: retrieve wide, rerank, keep few — this is what actually defeats lost-in-the-middle, by ensuring the few chunks you hand the model are the right ones. Treat chunking as a first-class design decision and measure it; bad boundaries make a fact un-retrievable no matter how good everything downstream is. Reward abstention. And keep RAG even in the million-token-window era: long context complements retrieval (you can hand it more chunks) but doesn't replace it, for cost, latency, and signal-to-noise reasons — garbage in still produces confidently-worded garbage out.

Failure points teams ignore — and what each costs

  • Dense-only retrieval. Consequence: queries with exact lexical signals fail silently — the user asks for invoice #4471 and gets semantically-similar-but-wrong results, with no error.
  • Bad chunking. Consequence: the answer exists in the corpus but is un-retrievable because it was split across chunk boundaries — the silent killer of RAG quality, and invisible unless you measure retrieval directly.
  • No reranking. Consequence: the right chunk is retrieved but buried mid-list, and lost-in-the-middle means the model ignores it.
  • Staleness / replication lag. Index drifts from the source of truth. Consequence: the agent answers from outdated documents — a correctness bug that looks like a hallucination.
  • End-to-end-only evaluation. Consequence: when the system is wrong you can't tell whether retrieval or generation failed, so you can't fix the right thing.
  • Confabulation over gaps. A missing or wrong read doesn't throw; the model fills the gap with a plausible hallucination. Consequence: silent failure presented as a confident answer — the worst failure mode for user trust.
  • Naive score-weighted fusion. Consequence: dense and sparse scores live on incompatible scales; weighting them directly produces unstable rankings that look fine in the demo and drift in production.

How to evaluate and mitigate

Evaluate the stages separately. Retriever alone: recall@k, nDCG@k, MRR against a labeled query set. Generator alone: faithfulness / groundedness given gold context (RAGAS), which isolates hallucination from retrieval miss. Track freshness lag against the source of truth. And measure abstention accuracy — when the context genuinely lacks the answer, does the system say so?

Mitigate: hybrid + RRF + cross-encoder reranking; chunk experiments judged on retrieval metrics, not vibes; a freshness/reindex pipeline; separated-stage evaluation; explicit grounding with citations and a rewarded "I don't know."

Practical vs. still research

Practical / mature: hybrid retrieval, RRF, cross-encoder reranking, structure-aware and child-parent chunking, learned sparse (SPLADE), query rewriting, and stage-separated evaluation. This is well-supported, well-understood engineering — and it's your search-infrastructure wheelhouse with one new signal (embeddings).

Emerging: semantic and "agentic" chunking, graph-RAG at scale.

Still research: retrieval that reliably knows when it's missing something (the abstention problem is far from solved), and fully-automated chunk-strategy optimization. Until those land, the human decisions — chunking scheme, what "grounded" means, when to abstain — are where a Principal earns their keep.

Read more