Skip to main content

Embeddings + Vector Search on Azure

Azure AI Search is the managed search service our team uses as the vector database backend for retrieval-augmented generation (RAG) applications. This guide walks you through the core concepts, resource setup, and your first end-to-end RAG query — from document ingestion to LLM-powered answers.

If you understand basic LangChain concepts and need to set up vector search on Azure, this is the place to start.

Azure AI Search (formerly Azure Cognitive Search) is a fully managed cloud search service that combines traditional keyword search with vector (embedding-based) search, semantic ranking, and rich data enrichment pipelines. It runs on Azure infrastructure and integrates natively with Azure OpenAI, Azure Machine Learning, and managed identity.

Why We Chose It Over Alternatives​

We evaluated Pinecone, Weaviate, Qdrant, and PostgreSQL with pgvector before settling on Azure AI Search. Here is why:

CriterionAzure AI SearchPinecone / WeaviatePostgreSQL + pgvector
Unified keyword + vectorNative hybrid searchVector-only (keyword requires add-on)Requires custom application logic
Semantic rankingBuilt-in, tier-dependentNot availableNot available
Azure ecosystemNative managed identity, VNet integrationExternal service, extra networkingRequires self-management on Azure
Data enrichmentSkillsets for OCR, translation, entity extractionManual pipeline neededManual pipeline needed
Operational overheadFully managedManaged but separate cloudSelf-managed or Azure Database for PostgreSQL
Cost modelPredictable per-hour tier pricingPer-operation + storageServer sizing + storage

The deciding factor was hybrid search with semantic ranking — the ability to combine keyword and vector signals in a single query with re-ranking, all managed by Azure. For RAG workloads, this consistently produces higher-quality results than pure vector search.

Key Capabilities​

  • Vector search — similarity search against high-dimensional embeddings using cosine distance, dot product, or Euclidean distance.
  • Hybrid search — combines keyword (full-text) and vector search in a single query, with configurable weighting.
  • Semantic ranking — re-ranks hybrid results using a semantic ranker model trained for relevance.
  • Filtering — OData filter expressions on non-searchable fields for metadata-based narrowing.
  • Scoring profiles — custom weighting and boosting rules for different field types.
  • Indexers — automated data ingestion from Azure Blob Storage, SQL Database, Cosmos DB, and more.

Core Concepts​

Before writing code, understand the building blocks that every Azure AI Search interaction is built on.

Indexes and Fields​

An index is the fundamental data structure in Azure AI Search. It defines the schema — what fields exist, their types, and how they are searchable. Each field has a role:

Field AttributePurposeExample
@searchableFull-text keyword searchDocument content, titles
@searchable + vector profileVector (embedding) searchEmbedded document chunks
@filterableOData filteringDocument source, category, date
@facetableFaceted navigation / groupingTags, authors
@retrievableReturned in search resultsAll fields you want in responses
@sortableOrdering resultsTimestamps, scores

A typical RAG index has three kinds of fields:

  1. A key field — unique identifier (string, required).
  2. A vector field — stores embeddings, referenced by a vector search profile.
  3. Searchable text fields — for keyword search (document content, titles).
  4. Metadata fields — filterable, retrievable fields for document metadata.

Embeddings and Embedding Models​

An embedding is a dense vector representation of text — a list of floating-point numbers that capture semantic meaning. Similar texts produce similar vectors, enabling similarity search.

Azure OpenAI provides two families of embedding models:

ModelDimensionsMax Input (tokens)Best For
text-embedding-3-small512 (default) or 15368,191General-purpose, cost-effective
text-embedding-3-large3072 (default) or 1536, 2568,191Highest quality, complex semantics
text-embedding-ada-0021,536 (fixed)8,191Legacy compatibility
tip

Use text-embedding-3-small with 512 dimensions for most RAG workloads. The quality-to-cost ratio is excellent, and the smaller dimensionality means faster indexing, lower storage, and quicker search. Only move to text-embedding-3-large if you measure a meaningful recall gap.

Understanding when to use each search mode is critical:

  • Vector search — finds semantically similar content. Strong for paraphrased queries and conceptual matching. Weak at exact term matching and boolean logic.
  • Keyword search — finds exact or stemmed term matches. Strong for proper nouns, product codes, and boolean queries. Blind to semantic similarity.
  • Hybrid search — combines both signals in a single query with reciprocal rank fusion. Best overall quality for RAG because it covers both semantic and lexical matching.

Semantic Ranking​

Semantic ranking is an Azure AI Search feature that re-ranks hybrid search results using a deep learning model fine-tuned for query-document relevance. It understands query intent and document context beyond what vector similarity or keyword matching alone can capture.

Semantic ranking requires the Basic tier or higher and must be explicitly enabled on the index.

Creating an Azure AI Search Resource​

You can create an Azure AI Search service from the Azure Portal or via the CLI.

  1. Navigate to the Azure Portal and select Create a resource.

  2. Search for Azure AI Search and select it.

  3. Fill in the creation form:

    • Subscription — your team's subscription.
    • Resource group — use the shared AI engineering resource group.
    • Name — a unique service name (e.g., ai-eng-search-prod).
    • Region — match your Azure OpenAI region (e.g., swedencentral).
    • Pricing tier — select based on feature requirements (see below).

  4. Review and create.

    info

    Service names must be globally unique across Azure. If your desired name is taken, append a suffix like -eu or -de.

Pricing Tiers and Feature Availability​

Choose your tier based on the features you need:

FeatureFreeBasicStandardStandard 2Standard 3
Indexes1100100100100
Documents10K1M10M10M100M
Vector searchNoYesYesYesYes
Hybrid searchNoYesYesYesYes
Semantic rankingNoYesYesYesYes
Cognitive skillsNoLimitedYesYesYes
Managed identityYesYesYesYesYes
Private endpointNoYesYesYesYes
tip

Start with the Basic tier for development. It supports vector search, hybrid search, and semantic ranking. Upgrade to Standard when you need cognitive skillsets or higher document limits.

Setting Up Authentication​

Azure AI Search supports two authentication methods:

Every search service has an admin key (full access) and one or more query keys (read-only). Use query keys in application code:

# List admin keys
az search admin-key show \
  --service-name ai-eng-search-prod \
  --resource-group rg-ai-engineering

# Create a query key with limited scope
az search query-key create \
  --service-name ai-eng-search-prod \
  --resource-group rg-ai-engineering \
  --name rag-query-key

Set the keys as environment variables:

export AZURE_SEARCH_API_KEY="your-query-key-here"
export AZURE_SEARCH_ENDPOINT="https://ai-eng-search-prod.search.windows.net"

Creating Your First Index with Vector Fields​

An index with vector search capability requires a vector search configuration that defines the algorithmic configuration (HNSW or exhaustive k-NN) and the vector field that references it.

  1. In the Azure Portal, open your Azure AI Search service.
  2. Navigate to Indexes and select + Add index.
  3. Define fields:
    • id (Key, Retrievable, sortable)
    • content (Searchable, Retrievable)
    • embedding (Vector, 1536 dimensions, referencing a vector profile)
    • source (Filterable, Facetable, Retrievable)
  4. Under Vector search, add a profile using hnsw algorithm and cosine distance.
  5. Save the index.

Embedding Models​

Available Models in Azure OpenAI​

Embedding models must be deployed to your Azure OpenAI resource before use. The deployment name is what you reference in code, not the model name itself.

  1. Open your Azure OpenAI resource in the portal.
  2. Go to Deployments → Create new deployment.
  3. Select the model (text-embedding-3-small, text-embedding-3-large, or text-embedding-ada-002).
  4. Give it a deployment name (e.g., embeddings-small).
  5. Deploy.

Dimension Sizes and Model Selection​

512 dimensions (recommended default):

from openai import AzureOpenAI

client = AzureOpenAI(
    api_version="2024-10-01-preview",
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
)

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Azure AI Search supports hybrid vector and keyword search.",
    dimensions=512,
)

embedding = response.data[0].embedding
print(f"Dimensionality: {len(embedding)}")  # 512

Cost Comparison​

Approximate pricing per 1 million tokens (Azure OpenAI, pay-as-you-go):

ModelPrice (per 1M tokens)Dimensions (default)Quality Tier
text-embedding-3-small$0.020512 (configurable)Good
text-embedding-3-large$0.1303072 (configurable)Best
text-embedding-ada-002$0.1001536 (fixed)Good
info

text-embedding-3-small at 512 dimensions gives you the best cost-to-quality ratio for RAG. At roughly 6.5x cheaper than the large model, it handles the embedding workload for ingestion pipelines and query embedding without meaningful recall degradation on typical document corpora.

Best Practices for Embedding Generation​

  • Chunk before embedding. Split documents into 256–1024 token chunks with 50–100 token overlap. Embedding whole documents wastes tokens and dilutes retrieval precision.
  • Use the same model for ingestion and query. Mixing embedding models produces incompatible vector spaces.
  • Set consistent dimensions. If you use text-embedding-3-small with 512 dimensions for ingestion, use 512 for queries too.
  • Batch embeddings. The API accepts up to 16,384 input items per call. Batch to reduce round trips and cost.
  • Add prefix instructions for task-specific embeddings. Prepend a task description to each text (e.g., "passage: " for document chunks, "query: " for search queries) to improve retrieval alignment.
# Batch embedding example
texts = [
    "Azure AI Search supports hybrid search combining vector and keyword signals.",
    "The semantic ranker re-orders results based on query intent.",
    "Vector search uses cosine similarity to find semantically related content.",
]

response = client.embeddings.create(
    model="text-embedding-3-small",
    input=["passage: " + t for t in texts],
    dimensions=512,
)

embeddings = [item.embedding for item in response.data]
print(f"Generated {len(embeddings)} embeddings of dimension {len(embeddings[0])}")

Your First RAG Query​

This section walks through a complete RAG pipeline: install dependencies, create an index, ingest documents, and run queries using vector search, keyword search, and hybrid search.

Step 1 — Install Dependencies​

# Create the project
uv init rag-demo
cd rag-demo

# Install LangChain integrations
uv add langchain-azure-ai-search langchain-openai

# Sync the environment
uv sync

Step 2 — Set Environment Variables​

# Azure AI Search
export AZURE_SEARCH_ENDPOINT="https://ai-eng-search-prod.search.windows.net"
export AZURE_SEARCH_API_KEY="your-query-key"

# Azure OpenAI
export AZURE_OPENAI_API_KEY="your-aoai-key"
export AZURE_OPENAI_ENDPOINT="https://your-aoai.openai.azure.com"
export AZURE_OPENAI_API_VERSION="2024-10-01-preview"

Step 3 — Create Index and Ingest Documents​

We use LangChain's AzureAISearch vector store to create the index and load documents in one step:

import os
from langchain_openai import AzureOpenAIEmbeddings
from langchain_azure_ai_search import AzureAISearch
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.docstore.document import Document

# --- Embedding model ---
embeddings = AzureOpenAIEmbeddings(
    model="text-embedding-3-small",
    azure_deployment="embeddings-small",
)

# --- Sample documents ---
raw_documents = [
    Document(
        page_content="Azure AI Search is a fully managed cloud search service "
                     "that supports vector search, keyword search, and hybrid search.",
        metadata={"source": "azure-docs", "topic": "overview"},
    ),
    Document(
        page_content="Hybrid search combines the precision of keyword matching "
                     "with the semantic understanding of vector search using "
                     "reciprocal rank fusion.",
        metadata={"source": "azure-docs", "topic": "hybrid-search"},
    ),
    Document(
        page_content="The semantic ranker uses a deep learning model to re-rank "
                     "search results based on query-document relevance beyond "
                     "simple similarity scores.",
        metadata={"source": "azure-docs", "topic": "semantic-ranking"},
    ),
    Document(
        page_content="HNSW (Hierarchical Navigable Small World) is the default "
                     "vector search algorithm in Azure AI Search, offering "
                     "fast approximate nearest neighbor search.",
        metadata={"source": "azure-docs", "topic": "vector-algorithms"},
    ),
]

# --- Split documents into chunks ---
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
)
documents = splitter.split_documents(raw_documents)

# --- Create vector store (creates index if it does not exist) ---
vector_store = AzureAISearch(
    embedding=embeddings,
    index_name="rag-demo-index",
    azure_search_endpoint=os.environ["AZURE_SEARCH_ENDPOINT"],
    azure_search_key=os.environ["AZURE_SEARCH_API_KEY"],
)

# --- Ingest documents ---
vector_store.add_documents(documents)
print(f"Ingested {len(documents)} document chunks.")

Step 4 — Build a Simple RAG Chain​

Now connect the vector store to an LLM for question answering:

from langchain_openai import AzureChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

# --- LLM ---
llm = AzureChatOpenAI(
    model="gpt-4o",
    azure_deployment="gpt-4o",
)

# --- Context retrieval ---
def retrieve_context(query: str) -> str:
    """Retrieve relevant document chunks from the vector store."""
    results = vector_store.similarity_search(query, k=3)
    return "\n\n".join(doc.page_content for doc in results)

# --- RAG prompt ---
rag_prompt = ChatPromptTemplate.from_messages([
    (
        "system",
        "You are a helpful assistant. Answer the user's question using "
        "only the provided context. If the context does not contain "
        "enough information, say so honestly.",
    ),
    ("human", "Context:\n{context}\n\nQuestion: {question}"),
])

# --- RAG chain ---
rag_chain = (
    RunnablePassthrough.assign(context=retrieve_context)
    | rag_prompt
    | llm
    | StrOutputParser()
)

# --- Run a query ---
answer = rag_chain.invoke({"question": "What is hybrid search and how does it work?"})
print(answer)

Step 5 — Try Different Search Modes​

Azure AI Search supports multiple search strategies. Here is how to use each one:

Pure semantic similarity search:

# Vector search — finds semantically similar content
results = vector_store.similarity_search(
    "How do I combine keyword and vector search?",
    k=3,
)

for doc in results:
    print(f"[{doc.metadata.get('topic')}] {doc.page_content[:100]}...")

Connecting with LangChain​

The langchain-azure-ai-search package provides a first-class integration between LangChain and Azure AI Search.

The AzureAISearch Vector Store Class​

AzureAISearch implements LangChain's VectorStore interface, giving you a consistent API regardless of your underlying vector database:

from langchain_azure_ai_search import AzureAISearch
from langchain_openai import AzureOpenAIEmbeddings

embeddings = AzureOpenAIEmbeddings(
    model="text-embedding-3-small",
    azure_deployment="embeddings-small",
)

vector_store = AzureAISearch(
    embedding=embeddings,
    index_name="my-rag-index",
    azure_search_endpoint=os.environ["AZURE_SEARCH_ENDPOINT"],
    azure_search_key=os.environ["AZURE_SEARCH_API_KEY"],
)

Key initialization parameters:

ParameterDescription
embeddingLangChain embedding model instance
index_nameName of the Azure AI Search index
azure_search_endpointSearch service URL
azure_search_keyAPI key (or use managed identity via azure_credential)
fieldsCustom field mapping for content, metadata, and embeddings
vector_search_configurationOverride default HNSW settings

LangChain provides two retrieval strategies:

query = "vector search algorithms"

# Similarity search — pure cosine similarity
similar_docs = vector_store.similarity_search(query, k=5)

# MMR (Maximal Marginal Relevance) — balances relevance with diversity
diverse_docs = vector_store.max_marginal_relevance_search(
    query,
    k=5,
    lambda_mult=0.5,  # 0 = max diversity, 1 = max similarity
)
  • Similarity search returns the top-k most similar vectors. Simple and fast, but can return redundant results.
  • MMR search trades some relevance for diversity by penalizing vectors that are too similar to already-selected results. Use lambda_mult to tune the balance.

Metadata Filtering​

Narrow search results by applying filters on metadata fields:

# Filter to documents from a specific source
filtered_docs = vector_store.similarity_search(
    "how does hybrid search work?",
    k=3,
    filter="source eq 'azure-docs'",
)

# Filter by multiple conditions
filtered_docs = vector_store.similarity_search(
    "vector search configuration",
    k=3,
    filter="source eq 'azure-docs' and topic eq 'vector-algorithms'",
)

# Filter by date range
filtered_docs = vector_store.similarity_search(
    "recent updates",
    k=3,
    filter="created_at ge 2025-01-01T00:00:00Z",
)

Filters use OData syntax. Supported operators include eq, ne, gt, ge, lt, le, and, or, and not.

Asynchronous Operations​

For high-throughput ingestion or concurrent queries, use async methods:

import asyncio

async def ingest_and_query():
    # Async document ingestion
    await vector_store.aadd_documents(documents)

    # Async similarity search
    results = await vector_store.asimilarity_search(
        "semantic ranking in Azure AI Search",
        k=3,
    )

    for doc in results:
        print(doc.page_content[:100])

asyncio.run(ingest_and_query())

Using as a Retriever in LCEL Chains​

The cleanest integration pattern wraps the vector store as a retriever and plugs it into an LCEL chain:

from langchain_core.runnables import RunnablePassthrough
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

# Create a retriever from the vector store
retriever = vector_store.as_retriever(
    search_type="similarity",  # or "mmr"
    search_kwargs={"k": 5, "filter": "source eq 'internal-wiki'"},
)

# Format retrieved documents into context
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# RAG prompt
prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer based on the following context:\n{context}"),
    ("human", "{question}"),
])

# Full RAG chain
chain = (
    {"question": lambda x: x["question"]}
    | RunnablePassthrough.assign(context=retriever | format_docs)
    | prompt
    | llm
    | StrOutputParser()
)

# Invoke
answer = chain.invoke({"question": "What are the differences between HNSW and exhaustive k-NN?"})
print(answer)

Next Steps​

You now have a working Azure AI Search index, ingested documents, and a RAG chain that answers questions. Here is where to go next:

tip

Before moving to production, read our Best Practices guide. It covers chunking strategies, embedding model selection trade-offs, and monitoring patterns specific to our Azure setup.