Cross-Encoder Reranking for RAG: When and How

A typical RAG pipeline retrieves documents by embedding similarity, but cosine distance between independent query and document embeddings often misses subtle interactions like negation or synonymy. A cross-encoder reranks the retrieved set by jointly attending to the query and each document, producing a relevance score that captures those finer-grained signals — at the cost of an additional forward pass per document.

What a cross-encoder does that a bi-encoder cannot

A bi-encoder (dense retriever) maps query and document to separate vectors; scoring is a simple dot product or cosine similarity. This is fast because embeddings can be precomputed and indexed, but the interaction between query and document is reduced to a single geometric distance. A cross-encoder, by contrast, concatenates the query and document into a single input and runs a full transformer forward pass. The self-attention layers can model token-level interactions — for example, distinguishing “dogs chase cats” from “cats chase dogs” — which a bi-encoder often collapses into similar vectors.

The cost is linear in the number of documents. For a retrieval of N candidates, you need N forward passes. That’s why cross-encoders are almost always used as a second stage: retrieve a broad set (say, top-100) with a bi-encoder, then rerank only those. The pipeline stays tractable because the expensive cross-encoder never sees the full corpus.

Integrating a cross-encoder into a RAG pipeline

The flow looks like this:

1. Embed the query and retrieve the top-N documents from a vector index.
2. For each document, form a (query, doc) pair and run it through the cross-encoder.
3. Collect the output scores (usually a logit or probability for relevance).
4. Sort documents by that score, take the top-M (e.g., 5) and feed them into the LLM context.

You can optionally normalize the scores — a softmax over the N outputs gives a relative probability distribution — and apply a threshold to discard documents below a certain confidence. This reduces noise in the LLM context.

Here’s a minimal implementation using sentence-transformers:

from sentence_transformers import CrossEncoder
 
# Load a pre-trained cross-encoder
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
 
def rerank(query, docs, top_k=5):
    pairs = [(query, doc) for doc in docs]
    scores = model.predict(pairs, show_progress_bar=False)
    # scores is a list of floats (logits)
    scored_docs = list(zip(docs, scores))
    scored_docs.sort(key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in scored_docs[:top_k]]

The predict method handles batching internally. If your documents are long (above 512 tokens), the model will truncate; consider chunking or using a model with a longer context window if needed.

Choosing a cross-encoder model: size, speed, accuracy

Cross-encoders come in sizes that trade latency for accuracy. The table below compares three common options trained on MS MARCO:

On GPU, the 6-layer model reranks 100 documents in about 50–100 ms. On CPU, expect 1–2 seconds. If your latency budget is under 200 ms total (including retrieval and LLM generation), you may need GPU inference or a smaller model.

Domain-specific fine-tuned models exist — for legal, biomedical, or financial text. If your data is niche, test a general model first. If it underperforms, fine-tune on your own labeled pairs. The Hugging Face cross-encoder model hub is a good starting point.

Batch inference and latency management

Cross-encoders benefit from batching because they process multiple query–document pairs simultaneously on GPU. For reranking, you typically batch all N documents for a single query together. That means batch size = N. If N is large (say >500), you may run out of GPU memory. Split into smaller batches, e.g., 64 or 128, and concatenate the results.

You can also use ONNX Runtime to cut latency, especially on CPU. Export the model to ONNX using optimum-intel or the transformers.onnx package, then run inference with onnxruntime. The speedup is typically 2–3x.

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer
 
model = ORTModelForSequenceClassification.from_pretrained("onnx/model-folder")
tokenizer = AutoTokenizer.from_pretrained("onnx/model-folder")
 
def rerank_onnx(query, docs):
    inputs = tokenizer(
        [(query, doc) for doc in docs],
        padding=True, truncation=True, return_tensors="pt"
    )
    outputs = model(**inputs)
    scores = outputs.logits[:, 1].tolist()  # assuming binary classification
    return scores

The ONNX export process is documented in the Optimum library. Quantization (INT8) can further reduce latency at a small accuracy cost.

When reranking hurts more than helps

Reranking adds a fixed overhead per query. In real-time systems — chatbots, search autocomplete — that extra 50–200 ms can degrade user experience more than the relevance gain improves it. If your initial retriever is already strong (e.g., fine-tuned E5 or ColBERT), the improvement from a cross-encoder may be marginal. Run an offline evaluation on a sample of your queries before committing.

For very small top-k (e.g., you only ever retrieve the top 5 documents), the reranking step adds latency without meaningful reordering. The bi-encoder already picked the most similar items; the cross-encoder rarely shuffles them enough to matter. Skip reranking when k ≤ 10.

Also consider the context window management of your LLM: if you are already feeding a large context, reranking may not be the bottleneck. On the other hand, if your LLM hallucinates due to irrelevant context, reranking can help — but only if the reranker effectively filters noise.

The choice of retriever also matters. If you use a retriever that already models query–document interactions (like ColBERT's late interaction), the marginal gain from a cross-encoder shrinks. For guidance on selecting a vector database that fits your access pattern, see Comparing vector databases by access pattern, not benchmarks.

Failure modes: false positives, score calibration, and ordering instability

Cross-encoders trained on MS MARCO (web search queries) may not generalize to your domain. A legal document about “discovery” might get a high score for a query about “discovery requests” but also for “scientific discovery.” Always test on your own data.

Scores are not calibrated across queries. A score of 2.0 for one query does not mean the same relevance level as 2.0 for another. Using a global threshold (e.g., score > 0.5) is unreliable. Instead, use relative ranking — take the top M after sorting — or normalize scores per query with softmax.

Small changes in query phrasing can produce different rankings (adversarial sensitivity). If your system is exposed to user-generated queries, consider ensembling the cross-encoder score with other signals like recency, authority, or a separate retrieval score (reciprocal rank fusion). The RAG chunking strategies article discusses how chunk quality affects this.

Measuring the impact of reranking

Offline, compute NDCG@k, MRR, and recall@k on a labeled dataset with and without reranking. NDCG@k requires graded relevance judgments (e.g., 0–3) to capture how well the reranker pushes highly relevant documents to the top. MRR is simpler: it measures the rank of the first relevant document. Improvements of 5–15% are typical, but the variance is high. Also check whether the reranked context actually improves the final LLM answer. Use an LLM-as-judge evaluation (see Evaluating LLM Output Without a Golden Dataset) to measure correctness and reduction in hallucinations.

Online A/B tests are the gold standard. Track task completion rate, click-through rate (if your system has a UI), or direct user feedback. Offline gains don’t always translate — sometimes the reranker introduces a subtle bias that worsens the LLM’s output. Also monitor the per-query cost: reranking adds inference cost beyond the initial retrieval. For a detailed breakdown of how to measure actual LLM cost beyond token counts, see Token counting is wrong: measure actual LLM cost. If the reranker increases cost without a measurable improvement in end-task metrics, drop it.

Key takeaways

• Cross-encoder reranking improves relevance by modeling query–document interactions that cosine similarity misses, at the cost of O(N) forward passes.
• Always use cross-encoders as a second stage after bi-encoder retrieval; never apply them to the full corpus.
• Model choice depends on your latency budget: MiniLM-L-6 is a solid default; TinyBERT for CPU-only; L-12 for highest accuracy on GPU.
• Batch all query–document pairs together and consider ONNX quantization for CPU deployments.
• Skip reranking when the initial retriever is already strong, when k ≤ 10, or in real-time systems where sub-200 ms latency is critical.
• Evaluate offline with ranking metrics and online with end-task metrics; cross-encoder gains don’t always improve the LLM output.