Skip to main content

Optimizing Retrieval Quality

Retrieval quality is the single most important determinant of RAG system performance. No amount of prompt engineering or LLM capability compensates for poor retrieval β€” if the right context never reaches the model, the answer will be wrong.

This guide codifies our team's standards for building high-quality retrieval pipelines on Azure AI Search. Every pattern here is tested, opinionated, and reflects what we've learned from running production RAG systems.

Why Retrieval Quality Matters​

A RAG system is only as good as its retrieval. The chain of failure looks like this:

Bad chunking β†’ Wrong context retrieved β†’ LLM hallucinates or refuses to answer β†’ User loses trust

Conversely, well-tuned retrieval produces:

  • Higher factual accuracy β€” the LLM grounds answers in relevant, precise context.
  • Lower latency β€” fewer irrelevant chunks means smaller context windows and faster LLM calls.
  • Reduced cost β€” better precision at lower k values means fewer embedding and LLM tokens consumed.
  • Better user experience β€” cited sources are actually relevant to the question asked.

Our goal is precision at the top-k: the first 3–5 retrieved chunks should contain the answer. Everything in this guide serves that goal.

Chunking Strategies​

Chunking is the foundation of retrieval quality. How you split documents determines what the embedding model sees, what the vector index stores, and ultimately what gets retrieved. A bad chunking strategy cannot be fixed by a better embedding model or a higher k value.

Fixed-Size vs. Semantic Chunking​

Fixed-size chunking splits text at regular intervals (e.g., every 512 tokens). It is simple and predictable but breaks semantic units β€” a chunk might start mid-sentence or split a paragraph in half.

Semantic chunking splits at natural boundaries β€” paragraph ends, section breaks, or points where the topic shifts. It preserves meaning but requires more sophisticated logic.

For our team, the default is recursive character text splitting with document-aware fallbacks. Here is why:

StrategyProsCons
Fixed-sizeSimple, fast, predictable token countsBreaks semantic boundaries, poor quality on structured docs
Recursive character splitRespects natural boundaries, configurable, our defaultSlightly more complex setup
Semantic (LLM-driven)Highest quality boundary detectionExpensive, slow, impractical for large-scale ingestion
Document-aware (section/paragraph)Best for structured content (Markdown, HTML)Requires format-specific parsing

Our Default: Recursive Character Text Splitting​

The RecursiveCharacterTextSplitter from LangChain is our default because it recursively tries multiple separators (paragraphs, newlines, sentences, spaces) until it finds a boundary that produces chunks within the target size.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=[
        "\n\n",       # Paragraph breaks
        "\n",         # Line breaks
        " . ",        # Sentences (space-period-space)
        " ",          # Words
        "",           # Characters (last resort)
    ],
)

chunks = splitter.split_documents(documents)

The separator order matters. The splitter tries each separator in order, so paragraphs are split first, then lines, then sentences, then words. This hierarchy preserves the most semantically meaningful boundaries.

Overlap Strategies​

Overlap prevents information from being lost at chunk boundaries. Without overlap, a key phrase split across two chunks might not appear in either chunk's embedding.

Recommended overlap by chunk size:

Chunk Size (tokens)Overlap (tokens)Overlap Ratio
25620–30~10%
51250–60~10–12%
1024100–150~10–15%
warning

Excessive overlap (>20%) creates redundant chunks that waste index storage and embedding costs without improving retrieval. Keep overlap between 10–15%.

Document-Aware Chunking for Structured Content​

When your source documents have structure (Markdown headings, HTML sections, PDF page breaks), leverage it. Document-aware chunking produces higher-quality results because chunks align with semantic units.

from langchain.text_splitter import MarkdownHeaderTextSplitter

headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]

markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on,
)

# Split by headings first
header_splits = markdown_splitter.split_text(markdown_content)

# Then apply recursive splitting within each section
recursive_splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
)

chunks = []
for section in header_splits:
    section_chunks = recursive_splitter.split_documents([section])
    # Propagate heading metadata to all sub-chunks
    for chunk in section_chunks:
        chunk.metadata.update(section.metadata)
    chunks.extend(section_chunks)

Chunk Size Recommendations by Document Type​

Different document types require different chunk sizes. Use these as starting points and tune based on retrieval metrics:

Document TypeChunk SizeOverlapRationale
Technical docs / API references256–512 tokens20–50Dense information, short answers needed
Policy / legal documents512–768 tokens50–75Context-heavy, need surrounding clauses
Blog posts / articles512–768 tokens50–75Narrative flow, moderate density
Research papers768–1024 tokens75–100Long arguments, need broader context
Chat transcripts / Q&A256–384 tokens20–30Short turns, self-contained meaning
Code documentation256–384 tokens20–30Code snippets are short and precise
tip

Start with 512 tokens / 50 overlap as your baseline. Adjust only if you measure a retrieval quality gap for a specific document type.

Azure AI Search Integrated Chunking​

Azure AI Search now supports integrated chunking and vectorization in the indexer pipeline. If your data lives in a supported source (Azure Blob Storage, Cosmos DB, OneLake), you can let Azure handle chunking:

from azure.search.documents.indexes.models import (
    ChunkingSkill,
    SplitSkill,
)

# Use Azure's built-in chunking in the skillset
chunking_skill = SplitSkill(
    text_split_mode="pages",
    maximum_page_length=800,
    page_overlap_length=100,
)

Supported split modes:

  • pages β€” splits by token count (configurable max length and overlap).
  • recursive β€” mirrors RecursiveCharacterTextSplitter behavior.
  • fixed β€” splits at exact token boundaries.
info

When using integrated chunking, the maximum_page_length and page_overlap_length values are in tokens, not characters. A maximum_page_length of 800 with page_overlap_length of 100 roughly corresponds to our 512/50 recommendation when accounting for tokenization differences.

Reranking​

Reranking is the process of re-scoring initially retrieved results using a more sophisticated model. It sits between retrieval and context assembly, and it is the most effective single lever for improving retrieval quality.

What Is Reranking and When to Use It​

The initial retrieval step (vector or hybrid search) uses approximate similarity scoring β€” fast but imprecise. Reranking takes the top-N candidates and re-scores them using a model that jointly considers the query and each document, producing a more accurate relevance ranking.

Use reranking when:

  • Your RAG application serves end users (not internal tools).
  • You retrieve more than 5 candidates and need to narrow to the best 3–5.
  • Your domain has nuanced relevance criteria (e.g., legal, medical, financial).

Skip reranking when:

  • Your initial retrieval is already highly precise (small, well-curated corpus).
  • Latency budgets are extremely tight (< 200ms end-to-end).
  • You are in early prototyping and need to validate chunking and indexing first.

Azure AI Search Semantic Ranker​

Azure AI Search's built-in semantic ranker is the simplest reranking option. It is a deep learning model fine-tuned for query-document relevance, running entirely within the search service.

To enable it, configure a semantic configuration on your index:

from azure.search.documents.indexes.models import (
    SemanticSearch,
    SemanticConfiguration,
    SemanticPrioritizedFields,
    SemanticField,
)

semantic_config = SemanticSearch(
    configurations=[
        SemanticConfiguration(
            name="rag-semantic-config",
            prioritized_fields=SemanticPrioritizedFields(
                title_field=SemanticField(name="title"),
                content_fields=[SemanticField(name="content")],
                keywords_fields=[SemanticField(name="category")],
            ),
        )
    ]
)

# Attach to your index definition
index.semantic_settings = semantic_config

Then use it at query time:

from azure.search.documents.models import QueryType

results = search_client.search(
    search_text="How does hybrid search work?",
    vector_queries=[
        VectorQuery(
            kind="vector",
            fields=["contentVector"],
            k_nearest_neighbors=50,  # Microsoft recommends k=50 for semantic ranker
        )
    ],
    query_type=QueryType.SEMANTIC,
    semantic_configuration_name="rag-semantic-config",
    top=5,
)
tip

Microsoft recommends setting k_nearest_neighbors=50 when using semantic ranker. The ranker needs enough candidates to work with β€” too few and it cannot meaningfully re-score.

Cost note: Semantic ranking requires the Basic tier or higher. It is included in the tier price with no per-query charge.

Cross-Encoder Rerankers​

For maximum quality, use a dedicated cross-encoder reranker. These models score each (query, document) pair independently and produce more accurate rankings than the built-in semantic ranker.

import cohere

co = cohere.Client(api_key=os.environ["COHERE_API_KEY"])

# Initial retrieval
initial_results = search_client.search(
    search_text=query,
    vector_queries=[VectorQuery(kind="vector", fields=["contentVector"], k=50)],
    top=50,
)

# Rerank with Cohere
documents_to_rerank = [
    hit["content"] for hit in initial_results
]

reranked = co.rerank(
    model="rerank-english-v3.0",
    query=query,
    documents=documents_to_rerank,
    top_n=5,
)

# Extract top results with rerank scores
final_results = [
    {"content": documents_to_rerank(r.result_index), "score": r.relevance_score}
    for r in reranked.results
]

Performance vs. Quality Tradeoffs​

Reranking MethodAdded LatencyQuality GainCost
None (raw vector)0 msBaselineFree
Azure Semantic Ranker50–150 msModerate–HighIncluded in tier
Cohere Rerank100–300 msHigh~$0.0001 per document
BGE Cross-Encoder (local)50–200 msHighYour compute cost
BGE Cross-Encoder (GPU)20–80 msHighGPU infrastructure
tip

For most production RAG systems, Azure Semantic Ranker gives the best quality-to-latency ratio. Add a cross-encoder reranker only if you measure a meaningful quality gap on your evaluation set.

Metadata Filtering​

Metadata filtering narrows the search space before vector similarity scoring. It is the difference between searching your entire knowledge base and searching only the relevant subset.

Designing Metadata Schemas​

Every document chunk should carry structured metadata that enables filtering. Design your schema before ingestion:

# Recommended metadata fields for every chunk
CHUNK_METADATA_SCHEMA = {
    "id": "string",           # Unique chunk identifier
    "source_document_id": "string",   # Parent document ID
    "source_file_name": "string",    # Original filename
    "document_type": "string",       # "policy", "technical-doc", "blog", etc.
    "department": "string",          # "engineering", "legal", "hr", etc.
    "language": "string",           # "en", "de", etc.
    "created_at": "datetime",       # Document creation date
    "updated_at": "datetime",       # Last modification date
    "version": "string",            # Document version
    "section_heading": "string",    # Heading context from document-aware chunking
}
warning

Every metadata field you want to filter on must be marked filterable=True in the index schema. Non-filterable fields cannot be used in OData filter expressions.

Common Metadata Patterns​

Filter to a specific document category:

# Only retrieve from technical documentation
results = vector_store.similarity_search(
    "how to configure vector search",
    k=5,
    filter="document_type eq 'technical-doc'",
)

Query-Time Filtering vs. Index-Time Organization​

You have two architectural choices for how to handle content segmentation:

ApproachWhen to UseTradeoff
Query-time filtering (one index, filter at query time)Fewer than 5 content domains, overlapping metadataSimpler, single index to manage
Index-time organization (multiple indexes, one per domain)Distinct domains with different schemas, strict isolationMore indexes to maintain, but cleaner separation

Our recommendation: Use a single index with query-time filtering unless you have genuinely different schemas or strict data isolation requirements. Multiple indexes multiply operational complexity.

Filter Timing: Pre-Filter vs. Post-Filter​

Azure AI Search supports two filter application modes for vector queries:

  • Pre-filter β€” the filter is applied before vector search, narrowing the candidate set. Faster, but may exclude relevant results if the filter is too restrictive.
  • Post-filter β€” vector search runs on the full index, then results are filtered. More inclusive, but slower.
from azure.search.documents.models import VectorFilterMode

# Pre-filter: filter first, then search (faster, more restrictive)
results = search_client.search(
    search_text=query,
    vector_queries=[VectorQuery(kind="vector", fields=["contentVector"], k=50)],
    filter="department eq 'engineering'",
    vector_filter_mode=VectorFilterMode.PRE_FILTER,
)

# Post-filter: search first, then filter (more inclusive)
results = search_client.search(
    search_text=query,
    vector_queries=[VectorQuery(kind="vector", fields=["contentVector"], k=50)],
    filter="department eq 'engineering'",
    vector_filter_mode=VectorFilterMode.POST_FILTER,
)
tip

Use pre-filter for large indexes with narrow filter scopes. Use post-filter when you want maximum recall and your filter is broad.

Hybrid Search Tuning​

Hybrid search runs vector and keyword queries in parallel, then merges results using Reciprocal Rank Fusion (RRF). It is our default search mode for production RAG systems because it consistently outperforms pure vector or pure keyword search.

How RRF Works​

RRF combines ranked lists from different search modalities without requiring score normalization. For each document, it computes:

RRF score = Ξ£ (1 / (k + rank_i))

Where k is the RRF constant (default 61 in Azure AI Search) and rank_i is the document's rank in each modality's result list. Documents that rank highly in both vector and keyword results receive the highest combined score.

Weighting Vector vs. Keyword Signals​

You control the relative importance of vector versus keyword results through the weight parameter on vector queries:

# Vector-heavy: weight=2 means vector signal counts double
results = search_client.search(
    search_text="Azure AI Search vector search",
    vector_queries=[
        VectorQuery(
            kind="vector",
            fields=["contentVector"],
            k_nearest_neighbors=50,
            weight=2.0,  # Vector signal weighted 2x
        )
    ],
    top=5,
)

# Keyword-heavy: weight=0.5 means keyword signal dominates
results = search_client.search(
    search_text="HNSW algorithm configuration",
    vector_queries=[
        VectorQuery(
            kind="vector",
            fields=["contentVector"],
            k_nearest_neighbors=50,
            weight=0.5,  # Keyword signal weighted 2x
        )
    ],
    top=5,
)

When to Lean Vector-Heavy vs. Keyword-Heavy​

Query TypeRecommended WeightWhy
Conceptual questions ("how does X work?")Vector-heavy (1.5–2.0)Semantic understanding matters more than exact terms
Proper nouns, product codes, IDsKeyword-heavy (0.3–0.5)Exact matching is critical
Mixed queries (concept + specific terms)Balanced (0.8–1.2)Both signals contribute
Technical documentation lookupKeyword-heavy (0.5–0.8)Precise term matching often wins
General knowledge / FAQVector-heavy (1.5–2.0)Paraphrasing is common in questions

Semantic Configuration and Its Impact​

The semantic configuration tells Azure AI Search which fields to prioritize for semantic ranking. A well-tuned configuration can improve relevance by 10–30%:

from azure.search.documents.indexes.models import (
    SemanticSearch,
    SemanticConfiguration,
    SemanticPrioritizedFields,
    SemanticField,
)

semantic_config = SemanticSearch(
    configurations=[
        SemanticConfiguration(
            name="rag-semantic-config",
            prioritized_fields=SemanticPrioritizedFields(
                title_field=SemanticField(name="title"),
                content_fields=[
                    SemanticField(name="content"),
                    SemanticField(name="summary"),
                ],
                keywords_fields=[
                    SemanticField(name="tags"),
                    SemanticField(name="category"),
                ],
            ),
        )
    ]
)

Field roles in semantic configuration:

RolePurposeField Example
title_fieldPrimary title/heading for the documenttitle, heading
content_fieldsMain text content for relevance scoringcontent, body, summary
keywords_fieldsCategorical metadata for intent matchingtags, category, topic

Testing and Tuning Hybrid Weights​

Do not guess at weights. Test empirically on a representative query set:

import json
from dataclasses import dataclass, asdict

@dataclass
class HybridWeightTest:
    weight: float
    mean_reciprocal_rank: float
    precision_at_5: float
    avg_latency_ms: float

def evaluate_hybrid_weights(
    queries: list[str],
    ground_truth: dict[str, list[str]],  # query -> expected doc IDs
    search_client,
    weights: list[float] = None,
) -> list[HybridWeightTest]:
    """Test different vector weights and measure retrieval quality."""
    if weights is None:
        weights = [0.3, 0.5, 0.8, 1.0, 1.5, 2.0]

    results = []
    for weight in weights:
        mrr_scores = []
        precision_scores = []
        latencies = []

        for query in queries:
            import time
            start = time.perf_counter()

            search_results = search_client.search(
                search_text=query,
                vector_queries=[
                    VectorQuery(
                        kind="vector",
                        fields=["contentVector"],
                        k_nearest_neighbors=50,
                        weight=weight,
                    )
                ],
                top=5,
            )

            latency = (time.perf_counter() - start) * 1000
            latencies.append(latency)

            retrieved_ids = [doc["id"] for doc in search_results]
            expected_ids = ground_truth.get(query, [])

            # Mean Reciprocal Rank
            mrr = 0.0
            for rank, doc_id in enumerate(retrieved_ids, 1):
                if doc_id in expected_ids:
                    mrr = 1.0 / rank
                    break
            mrr_scores.append(mrr)

            # Precision@5
            hits = sum(1 for doc_id in retrieved_ids if doc_id in expected_ids)
            precision_scores.append(hits / len(retrieved_ids))

        results.append(HybridWeightTest(
            weight=weight,
            mean_reciprocal_rank=sum(mrr_scores) / len(mrr_scores),
            precision_at_5=sum(precision_scores) / len(precision_scores),
            avg_latency_ms=sum(latencies) / len(latencies),
        ))

    return results
tip

Run the weight evaluation monthly or whenever you add significant new content. Retrieval quality drifts as your corpus evolves.

Index Design Best Practices​

Field Type Selection​

Every field in your index needs a deliberate role. Over-indexing (marking every field as searchable) wastes resources and can degrade query performance.

Field AttributeWhen to UseWhen NOT to Use
searchable=TrueFields you want full-text search on (content, title)IDs, timestamps, numeric metadata
filterable=TrueFields used in OData filters (department, date, type)Large text fields, content bodies
facetable=TrueFields for grouping/aggregation (tags, categories)High-cardinality fields (UUIDs, URLs)
sortable=TrueFields for ordering (timestamps, scores)Text content fields
retrievable=TrueFields you want in query resultsInternal fields never returned

Recommended index schema for RAG:

from azure.search.documents.indexes.models import (
    SearchIndex,
    SearchField,
    SearchFieldDataType,
    SimpleVectorIndexer,
    HnswAlgorithmConfiguration,
    VectorSearch,
    VectorSearchProfile,
    VectorSearchAlgorithmKind,
    VectorSearchAlgorithmMetric,
    SemanticSearch,
    SemanticConfiguration,
    SemanticPrioritizedFields,
    SemanticField,
)

index = SearchIndex(
    name="rag-production-index",
    fields=[
        # Key field β€” unique identifier
        SearchField(
            name="id",
            type=SearchFieldDataType.String,
            key=True,
            retrievable=True,
        ),
        # Content β€” searchable for keyword search, retrievable for results
        SearchField(
            name="content",
            type=SearchFieldDataType.String,
            searchable=True,
            retrievable=True,
        ),
        # Title β€” searchable and used in semantic config
        SearchField(
            name="title",
            type=SearchFieldDataType.String,
            searchable=True,
            retrievable=True,
        ),
        # Vector field β€” stores embeddings
        SearchField(
            name="contentVector",
            type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
            searchable=True,
            vector_search_dimensions=512,
            vector_search_profile_name="default-profile",
        ),
        # Metadata β€” filterable but NOT searchable
        SearchField(
            name="document_type",
            type=SearchFieldDataType.String,
            filterable=True,
            facetable=True,
            retrievable=True,
        ),
        SearchField(
            name="department",
            type=SearchFieldDataType.String,
            filterable=True,
            facetable=True,
            retrievable=True,
        ),
        SearchField(
            name="source_file_name",
            type=SearchFieldDataType.String,
            filterable=True,
            retrievable=True,
        ),
        SearchField(
            name="created_at",
            type=SearchFieldDataType.DateTimeOffset,
            filterable=True,
            sortable=True,
            retrievable=True,
        ),
        SearchField(
            name="updated_at",
            type=SearchFieldDataType.DateTimeOffset,
            filterable=True,
            sortable=True,
            retrievable=True,
        ),
        SearchField(
            name="language",
            type=SearchFieldDataType.String,
            filterable=True,
            facetable=True,
            retrievable=True,
        ),
        SearchField(
            name="section_heading",
            type=SearchFieldDataType.String,
            filterable=True,
            retrievable=True,
        ),
    ],
    vector_search=VectorSearch(
        profiles=[
            VectorSearchProfile(
                name="default-profile",
                algorithm_configuration_name="default-algorithm",
            )
        ],
        algorithms=[
            HnswAlgorithmConfiguration(
                name="default-algorithm",
                kind=VectorSearchAlgorithmKind.HNSW,
                parameters={
                    "m": 4,
                    "efConstruction": 400,
                    "efSearch": 500,
                    "metric": VectorSearchAlgorithmMetric.COSINE,
                },
            )
        ],
    ),
    semantic_settings=SemanticSearch(
        configurations=[
            SemanticConfiguration(
                name="default",
                prioritized_fields=SemanticPrioritizedFields(
                    title_field=SemanticField(name="title"),
                    content_fields=[SemanticField(name="content")],
                ),
            )
        ]
    ),
)

HNSW Parameters​

The HNSW algorithm configuration controls the tradeoff between index quality, build speed, and search speed:

ParameterRecommendedEffect
m4 (default)Max connections per layer. Higher = better recall, more memory.
efConstruction400Build-time exploration. Higher = better index quality, slower build.
efSearch500Query-time exploration. Higher = better recall, slower queries.
metricCOSINEDistance metric. Use cosine for normalized embeddings.
tip

For most RAG workloads, the defaults (m=4, efConstruction=400, efSearch=500) are well-tuned. Only increase efSearch if you measure recall degradation.

When to Use Multiple Indexes​

Use multiple indexes when:

  • Content domains have fundamentally different schemas.
  • You need strict data isolation (e.g., tenant separation in multi-tenant apps).
  • Different content types require different embedding dimensions or models.

Use a single index when:

  • Content shares a common schema.
  • You want unified search across domains.
  • Operational simplicity is a priority.

Partitioning and Replication​

For production scale:

ScalePartitionsReplicasNotes
Development (< 100K docs)11Basic tier sufficient
Production (100K–10M docs)1–21–2Standard tier, scale replicas for read throughput
High-scale (10M+ docs)2–42+Standard 2/3, monitor query latency
  • Partitions scale write throughput and index size.
  • Replicas scale read throughput and provide redundancy.

Index Lifecycle Management​

# Check index status
index_stats = index_client.get_index_stats("rag-production-index")
print(f"Document count: {index_stats.document_count}")
print(f"Storage (MB): {index_stats.storage_size / 1024 / 1024}")

# Soft delete pattern β€” mark documents as deleted instead of removing
def soft_delete_documents(search_client, doc_ids: list[str]):
    """Mark documents as deleted by updating a status field."""
    updates = [
        {"id": doc_id, "@search.action": "merge", "status": "deleted"}
        for doc_id in doc_ids
    ]
    search_client.merge_or_upload_documents(updates)

# Filter out soft-deleted documents at query time
results = search_client.search(
    search_text=query,
    filter="status ne 'deleted'",
    # ... rest of query
)
warning

Avoid frequent index deletion and recreation. It causes downtime and requires full re-indexing. Use soft deletes and incremental updates instead.

Cost Optimization​

Understanding Azure AI Search Costs​

Azure AI Search uses a tiered pricing model. Vector search itself is free across all tiers β€” you pay for the service capacity, not per-query.

TierMonthly Cost (approx.)Max DocumentsVector SearchSemantic Ranker
Free$010,000NoNo
Basic~$501,000,000YesYes
Standard~$15010,000,000YesYes
Standard 2~$30010,000,000YesYes
Standard 3~$450100,000,000YesYes

The real costs are in the embedding model calls (Azure OpenAI), not the search service:

Cost CenterWhat You Pay ForOptimization Lever
Azure AI SearchTier price (fixed)Right-size the tier
Azure OpenAI EmbeddingsPer-token (ingestion + queries)Reduce chunk count, cache queries
Azure OpenAI Chat (LLM)Per-token (context + response)Better retrieval = smaller context
Cross-encoder rerankerPer-document (if using Cohere/etc.)Lower initial k, use Azure ranker

Caching Strategies​

Reduce embedding costs by caching query embeddings:

import hashlib
import json
import os
from pathlib import Path

CACHE_DIR = Path("query_embedding_cache")
CACHE_DIR.mkdir(exist_ok=True)

def cache_key(text: str) -> str:
    """Generate a deterministic cache key from query text."""
    return hashlib.sha256(text.encode()).hexdigest()

def get_cached_embedding(query: str) -> list[float] | None:
    """Check cache before calling the embedding API."""
    key = cache_key(query)
    cache_file = CACHE_DIR / f"{key}.json"
    if cache_file.exists():
        return json.loads(cache_file.read_text())
    return None

def embed_query_with_cache(query: str, embedding_model) -> list[float]:
    """Embed a query, using cache when available."""
    cached = get_cached_embedding(query)
    if cached:
        return cached

    # Cache miss β€” generate embedding
    response = embedding_model.embed_query(query)
    cache_file = CACHE_DIR / f"{cache_key(query)}.json"
    cache_file.write_text(json.dumps(response))
    return response
tip

Query embedding caching is most effective for FAQ-style applications where users ask similar questions. For open-domain QA, cache hit rates are lower.

Right-Sizing the Pricing Tier​

Start with Basic for development and small production workloads (< 1M documents). Upgrade when you hit document limits or need cognitive skillsets.

Monitor your usage:

from azure.search.documents.indexes import SearchIndexClient
from azure.core.credentials import AzureKeyCredential

index_client = SearchIndexClient(
    endpoint=os.environ["AZURE_SEARCH_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["AZURE_SEARCH_API_KEY"]),
)

stats = index_client.get_index_stats("rag-production-index")
print(f"Documents: {stats.document_count:,}")
print(f"Storage: {stats.storage_size / (1024**2):.1f} MB")

# Alert if approaching tier limits
BASIC_LIMIT = 1_000_000
if stats.document_count > BASIC_LIMIT * 0.8:
    print("WARNING: Approaching Basic tier document limit. Consider upgrading.")

Monitoring and Alerting on Costs​

Set up Azure Monitor alerts for your search service:

# Example: Azure CLI to create a cost alert
# az monitor metrics alert create \
#   --name "SearchDocumentCountAlert" \
#   --resource-group rg-ai-engineering \
#   --scopes "/subscriptions/{id}/resourceGroups/rg-ai-engineering/providers/Microsoft.Search/searchServices/ai-eng-search-prod" \
#   --condition "GreaterThan DocumentCount 800000" \
#   --action-group "/subscriptions/{id}/resourceGroups/rg-ai-engineering/providers/Microsoft.Insights/actionGroups/alert-group"

Monitoring Retrieval Quality​

You cannot improve what you do not measure. Track retrieval quality continuously.

Key Metrics​

MetricDefinitionTarget
Recall@KFraction of relevant documents found in top-K results> 0.8 at K=10
Precision@KFraction of top-K results that are relevant> 0.7 at K=5
MRR (Mean Reciprocal Rank)Average of 1/rank for first relevant result> 0.6
NDCG@KNormalized Discounted Cumulative Gain> 0.7
Retrieval Latency (p95)95th percentile query time< 500 ms

Evaluation Framework​

from dataclasses import dataclass
from typing import Callable

@dataclass
class RetrievalMetrics:
    recall_at_k: float
    precision_at_k: float
    mrr: float
    ndcg_at_k: float
    avg_latency_ms: float

def compute_retrieval_metrics(
    queries: list[str],
    ground_truth: dict[str, set[str]],
    retriever: Callable[[str], list[dict]],
    k: int = 5,
) -> RetrievalMetrics:
    """Compute retrieval quality metrics against a labeled test set."""
    recalls = []
    precisions = []
    reciprocal_ranks = []
    ndcgs = []
    latencies = []

    import time

    for query in queries:
        relevant_ids = ground_truth.get(query, set())
        if not relevant_ids:
            continue

        start = time.perf_counter()
        results = retriever(query)
        latency = (time.perf_counter() - start) * 1000
        latencies.append(latency)

        retrieved_ids = [doc["id"] for doc in results[:k]]

        # Recall@K
        hits = len(set(retrieved_ids) & relevant_ids)
        recalls.append(hits / len(relevant_ids) if relevant_ids else 0)

        # Precision@K
        precisions.append(hits / k if k > 0 else 0)

        # MRR
        rr = 0.0
        for rank, doc_id in enumerate(retrieved_ids, 1):
            if doc_id in relevant_ids:
                rr = 1.0 / rank
                break
        reciprocal_ranks.append(rr)

        # NDCG@K (simplified binary relevance)
        dcg = sum(
            (1.0 / (2**i - 1)) if doc_id in relevant_ids else 0
            for i, doc_id in enumerate(retrieved_ids, 1)
        )
        ideal_hits = min(len(relevant_ids), k)
        idcg = sum(1.0 / (2**i - 1) for i in range(1, ideal_hits + 1))
        ndcgs.append(dcg / idcg if idcg > 0 else 0)

    return RetrievalMetrics(
        recall_at_k=sum(recalls) / len(recalls) if recalls else 0,
        precision_at_k=sum(precisions) / len(precisions) if precisions else 0,
        mrr=sum(reciprocal_ranks) / len(reciprocal_ranks) if reciprocal_ranks else 0,
        ndcg_at_k=sum(ndcgs) / len(ndcgs) if ndcgs else 0,
        avg_latency_ms=sum(latencies) / len(latencies) if latencies else 0,
    )

Using Langfuse to Monitor Retrieval Quality​

Langfuse provides observability for LLM applications, including retrieval tracking. Instrument your RAG chain:

from langfuse import Langfuse

langfuse = Langfuse()

def trace_retrieval(
    trace_name: str,
    query: str,
    retrieved_docs: list[dict],
    metrics: RetrievalMetrics | None = None,
):
    """Log retrieval trace to Langfuse."""
    trace = langfuse.trace(
        name=trace_name,
        input={"query": query},
        output={
            "doc_count": len(retrieved_docs),
            "doc_ids": [doc["id"] for doc in retrieved_docs[:10]],
        },
    )

    # Attach retrieval metrics as scores
    if metrics:
        trace.score(name="recall_at_5", value=metrics.recall_at_k)
        trace.score(name="precision_at_5", value=metrics.precision_at_k)
        trace.score(name="mrr", value=metrics.mrr)
        trace.score(name="retrieval_latency_ms", value=metrics.avg_latency_ms)

A/B Testing Retrieval Configurations​

Test retrieval changes before deploying:

import random

class RetrievalABTest:
    """A/B test different retrieval configurations."""

    def __init__(self, experiment_name: str, variants: dict[str, dict]):
        self.experiment_name = experiment_name
        self.variants = variants  # {"control": {...}, "treatment": {...}}
        self.results = {"control": [], "treatment": []}

    def get_variant(self) -> str:
        return random.choice(list(self.variants.keys()))

    def record_result(self, variant: str, metrics: RetrievalMetrics):
        self.results[variant].append({
            "recall": metrics.recall_at_k,
            "precision": metrics.precision_at_k,
            "mrr": metrics.mrr,
            "latency_ms": metrics.avg_latency_ms,
        })

    def compare(self) -> dict:
        """Compare variant performance."""
        summary = {}
        for variant, results in self.results.items():
            if not results:
                continue
            summary[variant] = {
                "avg_recall": sum(r["recall"] for r in results) / len(results),
                "avg_precision": sum(r["precision"] for r in results) / len(results),
                "avg_mrr": sum(r["mrr"] for r in results) / len(results),
                "avg_latency_ms": sum(r["latency_ms"] for r in results) / len(results),
                "sample_size": len(results),
            }
        return summary
tip

Run A/B tests on a representative sample of production queries for at least one week before making changes permanent. Retrieval quality varies by query type and content distribution.

Quick Reference: Our Team Defaults​

SettingOur Default
Chunk size512 tokens
Chunk overlap50 tokens
Text splitterRecursiveCharacterTextSplitter
Embedding modeltext-embedding-3-small, 512 dimensions
Distance metricCosine
HNSW efSearch500
Hybrid searchAlways (vector weight 1.0, tune per domain)
Semantic rankerEnabled (k=50)
Cross-encoder rerankerOnly if Azure ranker is insufficient
Index tierBasic (dev), Standard (prod)
Retrieval k5 (final), 50 (for reranking)
Filter modePre-filter (large indexes), Post-filter (small indexes)
MonitoringLangfuse + monthly retrieval evaluation