Setting Up Golden Datasets & Evaluations

You've built a LangGraph agent. It loops, calls tools, and produces answers. But can you trust it? Can you measure whether it got better after your last prompt change? Can you catch regressions before they hit production?

This guide walks you through deploying Langfuse — our team's observability platform — on Azure, instrumenting your agents, creating golden datasets, and running evaluations. Every agent on our team ships with observability and a golden dataset. No exceptions.

By the end of this article, you will have:

• A Langfuse instance deployed on Azure (Container Apps or AKS).
• Your LangGraph agent instrumented with automatic and manual tracing.
• A golden dataset of curated examples for your agent's domain.
• An evaluation pipeline that runs against your golden dataset on every agent change.

---

What Is Langfuse

Langfuse is an open-source AI engineering platform that gives teams end-to-end visibility into their LLM applications. It captures traces from every agent invocation, surfaces latency and cost metrics, and provides a framework for evaluating agent quality against curated datasets.

Why Observability Is Mandatory

Building LLM applications without observability is like building a web API without logs. You cannot debug what you cannot see. Specifically, Langfuse gives us:

Capability	Why It Matters
Tracing	See every LLM call, tool invocation, and retrieval inside an agent run.
Cost tracking	Know exactly how much each agent invocation costs, broken down by model and token usage.
Latency profiling	Identify slow tool calls, redundant LLM invocations, and inefficient retrieval.
Golden datasets	Maintain a curated set of input/output pairs that define "good" agent behavior.
Evaluations	Quantitatively measure agent quality against golden datasets and detect regressions.
Session views	Reconstruct full multi-turn conversations for debugging and review.

Without these capabilities, you are flying blind. Prompt changes, model swaps, and tool modifications all silently affect agent behavior. Langfuse makes those changes visible and measurable.

What You Will Learn

This article is structured as a progressive tutorial. Each section builds on the previous one:

1. Deploying Langfuse on Azure — Get the platform running.
2. Instrumenting Your Agent — Connect Langfuse to your code.
3. Understanding Traces — Read the Langfuse UI.
4. Setting Up Golden Datasets — Create your evaluation baseline.
5. Running Your First Evaluation — Measure agent quality.

---

Deploying Langfuse on Azure

Our team self-hosts Langfuse on Azure. This section covers two deployment options: Azure Container Apps (recommended for most teams) and Azure Kubernetes Service (for larger teams with higher throughput needs).

Architecture Overview

A self-hosted Langfuse deployment consists of the following services:

┌─────────────────────────────────────────────────────┐
│                  Azure Container Apps                │
│                                                     │
│  ┌──────────────┐    ┌──────────────┐               │
│  │ langfuse-web │    │langfuse-      │               │
│  │ (Next.js UI) │    │worker         │               │
│  │ Port 3000    │    │ Port 3030     │               │
│  └──────┬───────┘    └──────┬────────┘               │
│         │                   │                        │
└─────────┼───────────────────┼────────────────────────┘
          │                   │
    ┌─────▼────────┐   ┌──────▼───────┐
    │  PostgreSQL   │   │    Redis     │
    │  (v17)       │   │  (v7)        │
    │  Metadata +   │   │  Queue +     │
    │  Auth        │   │  Cache       │
    └──────────────┘   └──────────────┘
          │
    ┌─────▼────────┐   ┌──────────────┐
    │  ClickHouse   │   │    MinIO     │
    │  (Analytics)  │   │  (Object     │
    │              │   │   Storage)   │
    └──────────────┘   └──────────────┘

Langfuse v3 uses ClickHouse for analytics and trace storage, PostgreSQL for metadata and authentication, Redis for task queues and caching, and MinIO (S3-compatible) for media and event storage.

Option A — Azure Container Apps (Recommended)

Azure Container Apps is our preferred deployment method. It provides serverless container hosting with auto-scaling, managed environments, and integrated logging.

Step 1 — Prerequisites

Ensure you have the following installed and configured:

# Azure CLI with Container Apps extension
az --version
az extension add --name containerapp --upgrade

# Docker (for local testing)
docker --version
docker compose version

# Logged in to Azure
az login
az account show --query "{name: name, id: id}"

Step 2 — Create the Infrastructure

RESOURCE_GROUP="rg-ai-observability"
LOCATION="swedencentral"
ENVIRONMENT_NAME="cae-langfuse"

# Create resource group
az group create \
  --name "$RESOURCE_GROUP" \
  --location "$LOCATION"

# Create Container Apps environment
az containerapp env create \
  --name "$ENVIRONMENT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --location "$LOCATION"

Step 3 — Generate Secrets

Langfuse requires several secrets. Generate them securely:

# NEXTAUTH_SECRET — validates login session cookies (256+ bits of entropy)
NEXTAUTH_SECRET=$(openssl rand -base64 32)

# SALT — salts hashed API keys (256+ bits of entropy)
SALT=$(openssl rand -base64 32)

# ENCRYPTION_KEY — encrypts sensitive data (256 bits, hex format)
ENCRYPTION_KEY=$(openssl rand -hex 32)

# Database password
POSTGRES_PASSWORD=$(openssl rand -base64 24)

# Redis password
REDIS_PASSWORD=$(openssl rand -base64 24)

# ClickHouse password
CLICKHOUSE_PASSWORD=$(openssl rand -base64 24)

echo "NEXTAUTH_SECRET=$NEXTAUTH_SECRET"
echo "SALT=$SALT"
echo "ENCRYPTION_KEY=$ENCRYPTION_KEY"
echo "POSTGRES_PASSWORD=$POSTGRES_PASSWORD"
echo "REDIS_PASSWORD=$REDIS_PASSWORD"
echo "CLICKHOUSE_PASSWORD=$CLICKHOUSE_PASSWORD"

Secret Management

Store these secrets in Azure Key Vault, not in environment variables or configuration files. Container Apps can reference Key Vault secrets directly.

KEY_VAULT_NAME="kv-ai-observability"

az keyvault create \
  --name "$KEY_VAULT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --location "$LOCATION" \
  --sku standard

# Store each secret
az keyvault secret set --vault-name "$KEY_VAULT_NAME" --name "langfuse-nextauth-secret" --value "$NEXTAUTH_SECRET"
az keyvault secret set --vault-name "$KEY_VAULT_NAME" --name "langfuse-salt" --value "$SALT"
az keyvault secret set --vault-name "$KEY_VAULT_NAME" --name "langfuse-encryption-key" --value "$ENCRYPTION_KEY"
az keyvault secret set --vault-name "$KEY_VAULT_NAME" --name "langfuse-postgres-password" --value "$POSTGRES_PASSWORD"
az keyvault secret set --vault-name "$KEY_VAULT_NAME" --name "langfuse-redis-password" --value "$REDIS_PASSWORD"
az keyvault secret set --vault-name "$KEY_VAULT_NAME" --name "langfuse-clickhouse-password" --value "$CLICKHOUSE_PASSWORD"

Step 4 — Deploy With Docker Compose on Azure

For Container Apps, we use a modified Docker Compose file that adapts the official Langfuse stack for Azure. Create the following file:

docker-compose.yml

# docker-compose.yml — Langfuse on Azure Container Apps
# Based on: https://github.com/langfuse/langfuse/blob/main/docker-compose.yml

services:
  langfuse-web:
    image: docker.io/langfuse/langfuse:3
    restart: always
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      clickhouse:
        condition: service_healthy
      minio:
        condition: service_healthy
    ports:
      - "3000:3000"
    environment:
      NEXTAUTH_URL: ${NEXTAUTH_URL:-https://langfuse.yourdomain.com}
      NEXTAUTH_SECRET: ${NEXTAUTH_SECRET}
      DATABASE_URL: postgresql://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-langfuse}
      SALT: ${SALT}
      ENCRYPTION_KEY: ${ENCRYPTION_KEY}
      HOSTNAME: 0.0.0.0
      TELEMETRY_ENABLED: "false"
      LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: "false"

      # ClickHouse
      CLICKHOUSE_MIGRATION_URL: clickhouse://clickhouse:${CLICKHOUSE_PASSWORD}@clickhouse:9000
      CLICKHOUSE_URL: http://clickhouse:8123
      CLICKHOUSE_USER: clickhouse
      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
      CLICKHOUSE_CLUSTER_ENABLED: "false"

      # Redis
      REDIS_HOST: redis
      REDIS_PORT: "6379"
      REDIS_AUTH: ${REDIS_PASSWORD}

      # S3-compatible storage (MinIO)
      LANGFUSE_S3_EVENT_UPLOAD_BUCKET: langfuse
      LANGFUSE_S3_EVENT_UPLOAD_REGION: auto
      LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${MINIO_ROOT_USER:-minio}
      LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${MINIO_ROOT_PASSWORD:-miniosecret}
      LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: http://minio:9000
      LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: "true"
      LANGFUSE_S3_EVENT_UPLOAD_PREFIX: events/

      LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: langfuse
      LANGFUSE_S3_MEDIA_UPLOAD_REGION: auto
      LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: ${MINIO_ROOT_USER:-minio}
      LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: ${MINIO_ROOT_PASSWORD:-miniosecret}
      LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: http://minio:9000
      LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: "true"
      LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: media/

      # Headless initialization (creates first org, project, and user)
      LANGFUSE_INIT_ORG_NAME: ${LANGFUSE_INIT_ORG_NAME:-AI Engineering}
      LANGFUSE_INIT_PROJECT_NAME: ${LANGFUSE_INIT_PROJECT_NAME:-Production}
      LANGFUSE_INIT_USER_EMAIL: ${LANGFUSE_INIT_USER_EMAIL:-admin@yourdomain.com}
      LANGFUSE_INIT_USER_NAME: ${LANGFUSE_INIT_USER_NAME:-Admin}
      LANGFUSE_INIT_USER_PASSWORD: ${LANGFUSE_INIT_USER_PASSWORD}

  langfuse-worker:
    image: docker.io/langfuse/langfuse-worker:3
    restart: always
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      clickhouse:
        condition: service_healthy
      minio:
        condition: service_healthy
    environment:
      NEXTAUTH_URL: ${NEXTAUTH_URL:-https://langfuse.yourdomain.com}
      DATABASE_URL: postgresql://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-langfuse}
      SALT: ${SALT}
      ENCRYPTION_KEY: ${ENCRYPTION_KEY}
      TELEMETRY_ENABLED: "false"

      # ClickHouse
      CLICKHOUSE_MIGRATION_URL: clickhouse://clickhouse:${CLICKHOUSE_PASSWORD}@clickhouse:9000
      CLICKHOUSE_URL: http://clickhouse:8123
      CLICKHOUSE_USER: clickhouse
      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
      CLICKHOUSE_CLUSTER_ENABLED: "false"

      # Redis
      REDIS_HOST: redis
      REDIS_PORT: "6379"
      REDIS_AUTH: ${REDIS_PASSWORD}

      # S3-compatible storage
      LANGFUSE_S3_EVENT_UPLOAD_BUCKET: langfuse
      LANGFUSE_S3_EVENT_UPLOAD_REGION: auto
      LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${MINIO_ROOT_USER:-minio}
      LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${MINIO_ROOT_PASSWORD:-miniosecret}
      LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: http://minio:9000
      LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: "true"
      LANGFUSE_S3_EVENT_UPLOAD_PREFIX: events/

      LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: langfuse
      LANGFUSE_S3_MEDIA_UPLOAD_REGION: auto
      LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: ${MINIO_ROOT_USER:-minio}
      LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: ${MINIO_ROOT_PASSWORD:-miniosecret}
      LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: http://minio:9000
      LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: "true"
      LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: media/

  postgres:
    image: docker.io/postgres:17
    restart: always
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-postgres}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB:-langfuse}
      TZ: UTC
      PGTZ: UTC
    volumes:
      - langfuse_postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-postgres}"]
      interval: 5s
      timeout: 5s
      retries: 10
      start_period: 10s

  clickhouse:
    image: docker.io/clickhouse/clickhouse-server:latest
    restart: always
    user: "101:101"
    environment:
      CLICKHOUSE_DB: default
      CLICKHOUSE_USER: clickhouse
      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
    volumes:
      - langfuse_clickhouse_data:/var/lib/clickhouse
      - langfuse_clickhouse_logs:/var/log/clickhouse-server
    healthcheck:
      test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1
      interval: 5s
      timeout: 5s
      retries: 10
      start_period: 5s

  redis:
    image: docker.io/redis:7
    restart: always
    command: >
      --requirepass ${REDIS_PASSWORD}
      --maxmemory-policy noeviction
    volumes:
      - langfuse_redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
      interval: 5s
      timeout: 5s
      retries: 10

  minio:
    image: cgr.dev/chainguard/minio:latest
    restart: always
    entrypoint: sh
    command: -c 'mkdir -p /data/langfuse && minio server --address ":9000" /data'
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minio}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-miniosecret}
    volumes:
      - langfuse_minio_data:/data
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 1s
      timeout: 5s
      retries: 5
      start_period: 1s

volumes:
  langfuse_postgres_data:
    driver: local
  langfuse_clickhouse_data:
    driver: local
  langfuse_clickhouse_logs:
    driver: local
  langfuse_redis_data:
    driver: local
  langfuse_minio_data:
    driver: local

.env

# .env — Langfuse Azure Deployment
# NEVER commit this file to version control.

# === Authentication ===
NEXTAUTH_URL=https://langfuse.yourdomain.com
NEXTAUTH_SECRET=<generated-by-openssl>
SALT=<generated-by-openssl>
ENCRYPTION_KEY=<generated-by-openssl-hex>

# === PostgreSQL ===
POSTGRES_USER=postgres
POSTGRES_PASSWORD=<generated-by-openssl>
POSTGRES_DB=langfuse

# === ClickHouse ===
CLICKHOUSE_PASSWORD=<generated-by-openssl>

# === Redis ===
REDIS_PASSWORD=<generated-by-openssl>

# === MinIO ===
MINIO_ROOT_USER=minio
MINIO_ROOT_PASSWORD=<generated-by-openssl>

# === Headless Initialization ===
LANGFUSE_INIT_ORG_NAME=AI Engineering
LANGFUSE_INIT_PROJECT_NAME=Production
LANGFUSE_INIT_USER_EMAIL=admin@yourdomain.com
LANGFUSE_INIT_USER_NAME=Admin
LANGFUSE_INIT_USER_PASSWORD=<strong-password>

Step 5 — Start the Stack

# Start all services in the background
docker compose up -d

# Watch the logs
docker compose logs -f

# Wait until you see "Ready" from the langfuse-web container
# This typically takes 2-3 minutes on first startup

Verify the deployment:

# Check all containers are healthy
docker compose ps

# Expected output: all services should show "healthy" status
# langfuse-web, langfuse-worker, postgres, clickhouse, redis, minio

Step 6 — Access the Langfuse UI

Open http://<your-azure-vm-ip>:3000 or https://langfuse.yourdomain.com (if you configured a custom domain) in your browser. Log in with the credentials from LANGFUSE_INIT_USER_EMAIL and LANGFUSE_INIT_USER_PASSWORD.

Custom Domain and TLS on Azure

For production, front Langfuse with an Azure Application Gateway or Azure Front Door to provide TLS termination and custom domain routing. Do not expose port 3000 directly to the internet.

# Example: Create a managed certificate and bind it to your Container App
az containerapp ingress cert create \
  --name "$APP_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --cert-file /path/to/cert.pem \
  --key-file /path/to/key.pem \
  --domain-name langfuse.yourdomain.com

Alternatively, use Azure Key Vault certificates with Container Apps for automatic TLS management.

Option B — Azure Kubernetes Service (AKS)

For larger teams with higher throughput needs, deploy Langfuse on AKS using the official Helm chart.

Step 1 — Install the Helm Chart

# Add the Langfuse Helm repository
helm repo add langfuse https://langfuse.github.io/helm-charts
helm repo update

# Create a namespace
kubectl create namespace langfuse

# Install with custom values
helm install langfuse langfuse/langfuse \
  --namespace langfuse \
  --values values-azure.yaml

Step 2 — Azure-Specific Helm Values

Create values-azure.yaml:

# values-azure.yaml
postgres:
  enabled: true
  auth:
    password: "${POSTGRES_PASSWORD}"
  primary:
    persistence:
      storageClass: premium-ssd
      size: 50Gi

clickhouse:
  enabled: true
  auth:
    password: "${CLICKHOUSE_PASSWORD}"
  persistence:
    storageClass: premium-ssd
    size: 100Gi

redis:
  enabled: true
  auth:
    password: "${REDIS_PASSWORD}"

minio:
  enabled: true
  auth:
    password: "${MINIO_ROOT_PASSWORD}"
  persistence:
    storageClass: premium-ssd
    size: 50Gi

langfuse:
  web:
    replicas: 2
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "1"
        memory: "2Gi"
    ingress:
      enabled: true
      className: azure-application-gateway
      hosts:
        - host: langfuse.yourdomain.com
          paths:
            - /
      tls:
        - secretName: langfuse-tls
          hosts:
            - langfuse.yourdomain.com
  worker:
    replicas: 2
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "1"
        memory: "2Gi"

Resource Sizing Recommendations

Tier	CPU	Memory	Storage	Use Case
Development	4 cores	8 GiB	50 GiB	Single developer, local testing
Small team	8 cores	16 GiB	100 GiB	2-5 engineers, moderate trace volume
Production	16 cores	32 GiB	200 GiB	Full team, all agents instrumented
High throughput	32+ cores	64+ GiB	500+ GiB	Multiple agents, high trace volume

Cost Estimation

For a small team deployment on Azure:

Service	SKU	Monthly Cost (est.)
Azure Container Apps Environment	Standard	~$7
Container Apps (langfuse-web + worker)	2 vCPU, 4 GiB each	~$40
Azure Database for PostgreSQL	Burstable B2s	~$20
Azure Cache for Redis	Basic C1	~$25
Azure Blob Storage (events + media)	Hot	~$5-15
Total		~$97-107/mo

A ClickHouse managed alternative (Azure Synapse or self-hosted on a VM) adds ~$50-100/mo but is recommended for production deployments with more than a few thousand traces per day.

Production Checklist

Before considering your Langfuse deployment production-ready:

• All secrets stored in Azure Key Vault (not environment variables).
• Custom domain configured with TLS (no plain HTTP).
• Database backups configured (PostgreSQL and ClickHouse).
• Resource limits set to prevent runaway memory usage.
• Health checks configured for Container Apps readiness probes.
• Log Analytics workspace connected for centralized logging.
• Network rules restrict inbound traffic to your team's IPs.
• Headless initialization credentials rotated or disabled after first login.

---

Instrumenting Your Agent

With Langfuse running, the next step is connecting it to your LangGraph agent. This section covers both automatic instrumentation (zero-code) and manual instrumentation (fine-grained control).

Step 1 — Install the Langfuse SDK

# Using uv (our team's package manager)
uv add langfuse

# Verify installation
uv run python -c "import langfuse; print(langfuse.__version__)"

The langfuse package is the official Python SDK. It handles trace collection, batching, retries, and automatic LangChain/LangGraph integration.

Step 2 — Configure API Keys

Langfuse generates API keys per project. Find them in the Langfuse UI under Project Settings > API Keys.

.env

# .env
LANGFUSE_PUBLIC_KEY=pk-lf-your-public-key
LANGFUSE_SECRET_KEY=sk-lf-your-secret-key
LANGFUSE_HOST=https://langfuse.yourdomain.com

Python config

# src/observability.py
import os

import langfuse

langfuse_client = langfuse.Langfuse(
    public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
    secret_key=os.environ["LANGFUSE_SECRET_KEY"],
    host=os.environ.get("LANGFUSE_HOST", "https://cloud.langfuse.com"),
)

Secret Management for SDK Keys

The LANGFUSE_SECRET_KEY should be stored as a Container Apps secret or in Azure Key Vault. Never hardcode it in your agent repository.

az containerapp secret set \
  --name "research-agent" \
  --resource-group "rg-ai-observability" \
  --secrets \
    langfuse-secret-key="sk-lf-..."

Step 3 — Automatic Instrumentation with LangChain / LangGraph

Langfuse integrates with LangChain and LangGraph out of the box. When the SDK is initialized, it automatically traces every LLM call, tool invocation, and retrieval:

# src/agent.py
import os
import langfuse
from langfuse.langchain import LangfuseCallbackHandler

# Initialize Langfuse — this enables automatic tracing
langfuse_client = langfuse.Langfuse()

# Create the callback handler for LangChain/LangGraph
langfuse_handler = LangfuseCallbackHandler()

# Your LangGraph agent
from langgraph.graph import StateGraph, START, END
from langchain.chat_models import init_chat_model
from langchain.tools import tool

@tool
def search_knowledge_base(query: str) -> str:
    """Search the knowledge base for relevant documents."""
    return f"Results for: {query}"

model = init_chat_model("gpt-4o")
model_with_tools = model.bind_tools([search_knowledge_base])

# Build the graph (same as your existing agent code)
# ...

# Invoke with Langfuse tracing
result = agent.invoke(
    {"messages": [{"role": "user", "content": "What is LangGraph?"}]},
    config={
        "callbacks": [langfuse_handler],
        "configurable": {"thread_id": "test-session-1"},
    },
)

With automatic instrumentation, every LLM call, tool execution, and graph step is captured as a Langfuse span. You get traces without writing any additional code.

Step 4 — Manual Instrumentation for Custom Components

For components that Langfuse does not automatically detect (custom retrievers, post-processing logic, external API calls), use the @observe decorator or the context manager API:

# src/observability.py
import langfuse
from langfuse.decorators import observe

langfuse_client = langfuse.Langfuse()


@observe(name="custom_rag_retrieval")
def retrieve_context(query: str, top_k: int = 5) -> list[str]:
    """Custom RAG retrieval with manual Langfuse tracing."""
    # The @observe decorator automatically creates a span
    # You can add metadata, tags, and level information
    langfuse.update_current_observation(
        metadata={"top_k": top_k, "strategy": "hybrid"},
        tags=["rag", "retrieval"],
    )

    # Your retrieval logic here
    results = perform_hybrid_search(query, top_k)
    return results


@observe(name="response_post_processor")
def post_process_response(raw_response: str) -> str:
    """Post-process LLM output with tracing."""
    langfuse.update_current_observation(
        input={"raw_length": len(raw_response)},
    )

    processed = clean_and_format(raw_response)

    langfuse.update_current_observation(
        output={"processed_length": len(processed)},
    )
    return processed

Step 5 — Tracing Individual LLM Calls, Tool Calls, and Retrievals

For fine-grained control, create spans manually:

# src/tracing.py
import langfuse

langfuse_client = langfuse.Langfuse()


def trace_agent_run(user_message: str, thread_id: str) -> dict:
    """Run the agent with full manual tracing."""
    # Create a trace for the entire agent run
    trace = langfuse_client.trace(
        name="research-agent-run",
        input={"message": user_message, "thread_id": thread_id},
        session_id=thread_id,
        tags=["production", "research-agent"],
    )

    # Create a generation span for the LLM call
    generation = langfuse_client.generation(
        trace_id=trace.id,
        name="gpt-4o-reasoning",
        model="gpt-4o",
        input=[{"role": "user", "content": user_message}],
    )

    try:
        # Your agent logic here
        result = run_agent_logic(user_message)

        # Update the generation with output and usage
        generation.end(
            output=result["response"],
            usage={
                "input": result.get("input_tokens", 0),
                "output": result.get("output_tokens", 0),
                "total": result.get("total_tokens", 0),
            },
        )

        # Update the trace with final output
        trace.update(
            output={"step_count": result.get("step_count", 0)},
            level="default",
        )

        return result

    except Exception as e:
        # Mark the trace as errored
        generation.end(level="ERROR", output={"error": str(e)})
        trace.update(level="ERROR", output={"error": str(e)})
        raise

Complete Instrumented Agent Example

Here is a full example showing an instrumented LangGraph agent:

# src/agent.py
"""Research agent with full Langfuse observability."""

import os

import langfuse
from langfuse.langchain import LangfuseCallbackHandler
from langfuse.decorators import observe

from langchain.chat_models import init_chat_model
from langchain.tools import tool
from langchain.messages import HumanMessage, SystemMessage, ToolMessage
from langgraph.graph import StateGraph, START, END
from typing_extensions import TypedDict, Annotated
import operator


# --- Langfuse setup ---
langfuse_client = langfuse.Langfuse()
langfuse_handler = LangfuseCallbackHandler()


# --- Agent state ---
class AgentState(TypedDict):
    messages: Annotated[list, operator.add]
    query: str
    step_count: int


# --- Tools ---
@tool
def search_knowledge_base(query: str) -> str:
    """Search the knowledge base for relevant documents."""
    # This tool call is automatically traced by Langfuse
    results = perform_search(query)
    return "\n".join(results)


# --- Nodes ---
def call_llm(state: AgentState) -> dict:
    """Call the LLM with current conversation state."""
    system = SystemMessage(
        content="You are a research assistant. Use tools to find accurate information."
    )
    model = init_chat_model("gpt-4o").bind_tools([search_knowledge_base])
    response = model.invoke([system] + state["messages"])
    return {
        "messages": [response],
        "step_count": state.get("step_count", 0) + 1,
    }


def execute_tools(state: AgentState) -> dict:
    """Execute tool calls from the LLM response."""
    results = []
    last_message = state["messages"][-1]

    for tool_call in last_message.tool_calls:
        tool_fn = search_knowledge_base
        observation = tool_fn.invoke(tool_call["args"])
        results.append(ToolMessage(
            content=observation,
            tool_call_id=tool_call["id"],
        ))

    return {"messages": results}


def should_continue(state: AgentState) -> str:
    """Route based on whether the LLM requested tool calls."""
    if state["messages"][-1].tool_calls:
        return "execute_tools"
    return END


# --- Build graph ---
graph = StateGraph(AgentState)
graph.add_node("call_llm", call_llm)
graph.add_node("execute_tools", execute_tools)
graph.add_edge(START, "call_llm")
graph.add_conditional_edges("call_llm", should_continue, ["execute_tools", END])
graph.add_edge("execute_tools", "call_llm")
agent = graph.compile()


# --- Entry point with tracing ---
@observe(name="research-agent")
def run_research_agent(user_message: str, thread_id: str) -> dict:
    """Run the research agent with full Langfuse tracing."""
    result = agent.invoke(
        {
            "messages": [HumanMessage(content=user_message)],
            "query": user_message,
            "step_count": 0,
        },
        config={
            "callbacks": [langfuse_handler],
            "configurable": {"thread_id": thread_id},
        },
    )

    # Log metadata to the trace
    langfuse.update_current_trace(
        metadata={
            "thread_id": thread_id,
            "step_count": result.get("step_count", 0),
        },
    )

    return result

---

Understanding Traces

Once your agent is instrumented, every invocation produces a trace in Langfuse. This section explains how to read and navigate the Langfuse UI.

Trace, Span, and Observation

Langfuse models observability data as a hierarchy:

Concept	Description	Example
Trace	Top-level container for a single agent invocation. Groups all related activity.	One user question → one trace.
Span	A named unit of work within a trace. Spans are nested and timed.	An LLM call, a tool execution, a retrieval step.
Observation	Generic term for any traced unit (spans, generations, events).	All spans are observations; not all observations are spans.

The hierarchy looks like this:

Trace: "research-agent-run"
├─ Generation: "gpt-4o" (LLM call)
│  ├─ Input: 1,200 tokens
│  └─ Output: 450 tokens
├─ Span: "custom_rag_retrieval"
│  ├─ Span: "vector_search" (200ms)
│  └─ Span: "keyword_search" (150ms)
├─ Generation: "gpt-4o" (second LLM call after tool results)
│  ├─ Input: 2,400 tokens
│  └─ Output: 380 tokens
└─ Span: "response_post_processor" (50ms)

Navigating the Langfuse Dashboard

After deployment, access the Langfuse dashboard at https://langfuse.yourdomain.com. The main views are:

Traces list — The default view shows all traces, sorted by recency. Each row displays:

• Trace name (e.g., research-agent-run)
• Duration (total wall-clock time)
• Total tokens and cost
• Level (default, warning, error)
• Tags and session ID

Trace detail — Click any trace to see the full span tree. The tree view shows:

• Each span's duration, input/output tokens, and latency breakdown.
• Color-coded levels (green = success, yellow = warning, red = error).
• Expandable input/output payloads for each observation.

Sessions — Group traces by session_id to view full multi-turn conversations. This is essential for debugging conversation-based agents where context spans multiple traces.

Filtering and Searching

The traces list supports powerful filtering:

# Filter by trace name
name: "research-agent-run"

# Filter by tags
tags: "production"

# Filter by duration (slower than 10 seconds)
duration: ">10s"

# Filter by cost (more than $0.01)
cost: ">0.01"

# Filter by error level
level: "ERROR"

# Combine filters
name: "research-agent-run" AND tags: "production" AND level: "ERROR"

Understanding Latency and Cost Metrics

Each trace and span reports:

Metric	Source	Use
Duration	Wall-clock time from span start to end.	Identify slow tools, network bottlenecks, and inefficient graph paths.
Input tokens	Counted from LLM API responses.	Track context window usage and detect prompt bloat.
Output tokens	Counted from LLM API responses.	Measure response verbosity and optimize prompts.
Total cost	Computed from token counts and model pricing.	Budget tracking and cost optimization across agents.
Time to first token (TTFT)	Measured from LLM streaming responses.	User experience metric for interactive agents.

Session Views for Conversation Agents

If your agent supports multi-turn conversations, always set session_id on traces. This groups related traces in the Langfuse UI, making it easy to follow a full conversation thread.

trace = langfuse_client.trace(
    name="chat-agent",
    session_id=user_session_id,  # Groups traces by conversation
    input={"message": user_message},
)

---

Setting Up Golden Datasets

Mandatory — Every Agent Needs a Golden Dataset

This is not optional. Every agent on our team must have a golden dataset before it ships to production. The golden dataset is your team's shared definition of "good" agent behavior, and it is the foundation for all evaluations.

Without a golden dataset, you have no way to detect regressions. A prompt change that seems fine in manual testing might silently degrade quality on edge cases. The golden dataset catches those regressions automatically.

What Is a Golden Dataset

A golden dataset is a curated collection of input/output pairs that represent the expected behavior of your agent. Each entry contains:

• Input — The user query or prompt.
• Expected output — The ideal response your agent should produce.
• Metadata — Optional context (difficulty level, topic category, etc.).

Golden datasets serve as the ground truth for evaluations. When you change your agent's prompt, swap models, or modify tools, you run the golden dataset through the new configuration and compare results against the expected outputs.

Creating Your First Dataset in the UI

The fastest way to create a dataset is through the Langfuse UI:

1. Navigate to Datasets in the left sidebar.
2. Click Create Dataset and give it a name (e.g., research-agent-golden-set).
3. Add examples one at a time:
   • Input: The user query (e.g., "What causes seasonal allergies?").
   • Expected output: The ideal response (e.g., "Seasonal allergies are caused by...").
   • Metadata: Optional tags like {"category": "health", "difficulty": "easy"}.
4. Save the dataset.

This approach works well for small datasets (5-20 examples). For larger or programmatic workflows, use the API.

Creating Datasets Programmatically

For our team's workflow, we create and maintain golden datasets as code. This ensures they are version-controlled, reviewed in pull requests, and reproducible.

# scripts/create_golden_dataset.py
"""Create and populate the golden dataset for the research agent."""

import os
import langfuse

# Connect to Langfuse
client = langfuse.Langfuse(
    public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
    secret_key=os.environ["LANGFUSE_SECRET_KEY"],
    host=os.environ.get("LANGFUSE_HOST", "https://cloud.langfuse.com"),
)

# Define the golden dataset examples
GOLDEN_EXAMPLES = [
    {
        "input": "What is the capital of France?",
        "expected_output": "The capital of France is Paris.",
        "metadata": {"category": "geography", "difficulty": "easy"},
    },
    {
        "input": "Explain how photosynthesis works in simple terms.",
        "expected_output": (
            "Photosynthesis is how plants make their own food using sunlight. "
            "Plants take in carbon dioxide from the air and water from the soil. "
            "Using energy from sunlight, they convert these into glucose (sugar) "
            "and release oxygen as a byproduct."
        ),
        "metadata": {"category": "science", "difficulty": "medium"},
    },
    {
        "input": "What are the main differences between mitosis and meiosis?",
        "expected_output": (
            "Mitosis produces two identical daughter cells and is used for growth "
            "and repair. Meiosis produces four genetically diverse cells and is "
            "used for sexual reproduction. Key differences include the number of "
            "divisions (one vs. two), the number of daughter cells (two vs. four), "
            "and whether genetic recombination occurs."
        ),
        "metadata": {"category": "biology", "difficulty": "hard"},
    },
    {
        "input": "How does the internet route data between computers?",
        "expected_output": (
            "The internet routes data using a system of routers and routing protocols. "
            "When you send data, it is broken into packets that travel independently "
            "through the network. Each router examines the packet's destination address "
            "and forwards it along the best available path. Protocols like BGP "
            "(Border Gateway Protocol) coordinate routing across different networks."
        ),
        "metadata": {"category": "technology", "difficulty": "hard"},
    },
    {
        "input": "What year did World War II end?",
        "expected_output": "World War II ended in 1945.",
        "metadata": {"category": "history", "difficulty": "easy"},
    },
]

# Create dataset items
dataset_name = "research-agent-golden-set"

for i, example in enumerate(GOLDEN_EXAMPLES):
    client.create_dataset_item(
        dataset_name=dataset_name,
        input=example["input"],
        expected_output=example["expected_output"],
        metadata=example["metadata"],
        id=f"golden-{i:03d}",
    )
    print(f"Added example {i + 1}/{len(GOLDEN_EXAMPLES)}")

print(f"Golden dataset '{dataset_name}' created with {len(GOLDEN_EXAMPLES)} examples.")

Run the script:

uv run scripts/create_golden_dataset.py

Dataset Versioning and Management

Golden datasets evolve as your agent's capabilities grow. Manage changes with these practices:

Pin dataset versions for evaluations:

# Get the current dataset snapshot
dataset = client.get_dataset("research-agent-golden-set")
dataset_version = dataset.created_at  # Use timestamp as version marker

print(f"Dataset version: {dataset_version}")
print(f"Total items: {len(dataset.items)}")

Add new examples without disrupting existing evaluations:

# Adding a new example does not invalidate existing dataset items
client.create_dataset_item(
    dataset_name="research-agent-golden-set",
    input="What is quantum entanglement?",
    expected_output="Quantum entanglement is a phenomenon where...",
    metadata={"category": "physics", "difficulty": "hard"},
    id="golden-005",
)

Deactivate examples that are no longer relevant:

# Mark an example as inactive instead of deleting it
client.update_dataset_item(
    id="golden-001",
    status="INACTIVE",
)

Golden Dataset Guidelines

• Start small: 10-20 well-chosen examples are better than 100 mediocre ones.
• Cover edge cases: Include ambiguous queries, multi-part questions, and domain-specific terminology.
• Balance difficulty: Mix easy, medium, and hard examples.
• Review regularly: Schedule monthly reviews to add, update, or retire examples.
• Store as code: Keep your golden dataset definitions in your repository, not just in the Langfuse UI.

---

Running Your First Evaluation

With a golden dataset in place, you can now evaluate your agent's quality quantitatively. This section covers setting up and running evaluations in Langfuse.

What Is an Evaluation

An evaluation in Langfuse is a structured comparison between your agent's actual outputs and the expected outputs from your golden dataset. It produces:

• Per-example scores — How well the agent performed on each individual example.
• Aggregate metrics — Overall quality score, pass rate, and trend data.
• Regression detection — Automatic alerts when quality drops below a threshold.

Setting Up an Eval Run

Langfuse provides a high-level run_experiment API for running evaluations against golden datasets:

# scripts/run_evaluation.py
"""Run evaluation against the golden dataset."""

import os
import langfuse
from langfuse import Evaluation, RegressionError

# Connect to Langfuse
client = langfuse.Langfuse(
    public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
    secret_key=os.environ["LANGFUSE_SECRET_KEY"],
    host=os.environ.get("LANGFUSE_HOST", "https://cloud.langfuse.com"),
)

# Define the evaluation task — this function runs your agent on each example
def evaluate_example(item: dict) -> dict:
    """Run the agent on a single golden dataset example."""
    from src.agent import run_research_agent

    # Run the agent with the golden dataset input
    result = run_research_agent(
        user_message=item["input"],
        thread_id=f"eval-{item.get('id', 'unknown')}",
    )

    # Return the agent's output for comparison
    return {
        "output": result["messages"][-1]["content"],
        "step_count": result.get("step_count", 0),
    }

# Run the experiment
experiment_result = client.run_experiment(
    name="research-agent-eval",
    dataset_name="research-agent-golden-set",
    task=evaluate_example,
    metadata={
        "agent_version": "1.0.0",
        "model": "gpt-4o",
        "prompt_version": "v3",
    },
)

print(f"Experiment ID: {experiment_result}")
print("Evaluation complete. Check Langfuse UI for results.")

Built-In Evaluation Metrics

Langfuse provides several built-in evaluation methods:

LLM-as-Judge — Use an LLM to score the agent's output against the expected output:

from langfuse import Evaluation

# Define an LLM-as-judge evaluator
def llm_judge_evaluator(item: dict, output: str) -> float:
    """Score agent output using an LLM judge."""
    from langchain.chat_models import init_chat_model

    judge = init_chat_model("gpt-4o")

    response = judge.invoke([
        {
            "role": "system",
            "content": (
                "You are an impartial judge. Score the following response "
                "on a scale of 0.0 to 1.0 based on accuracy, completeness, "
                "and relevance. Return only a number."
            ),
        },
        {
            "role": "user",
            "content": (
                f"Question: {item['input']}\n"
                f"Expected answer: {item['expected_output']}\n"
                f"Agent answer: {output}\n\n"
                "Score (0.0-1.0):"
            ),
        },
    ])

    # Parse the score from the LLM response
    score = float(response.content.strip())
    return score

# Register the evaluator with Langfuse
client.score(
    name="llm-judge-accuracy",
    value=llm_judge_evaluator,
    type="LLM_SCORE",
)

Regex Matching — Check if the agent's output matches a pattern:

# Simple regex-based evaluation
import re

def regex_evaluator(item: dict, output: str) -> dict:
    """Check if key terms from the expected output appear in the agent's response."""
    expected = item["expected_output"].lower()
    actual = output.lower()

    # Extract key phrases from expected output
    key_phrases = re.findall(r'\b[a-z]{4,}\b', expected)[:10]
    matches = sum(1 for phrase in key_phrases if phrase in actual)

    score = matches / len(key_phrases) if key_phrases else 0.0

    return {
        "score": score,
        "matches": matches,
        "total_phrases": len(key_phrases),
    }

Similarity Scoring — Measure semantic similarity between expected and actual output:

from sentence_transformers import SentenceTransformer

# Load a pre-trained model for semantic similarity
similarity_model = SentenceTransformer("all-MiniLM-L6-v2")

def similarity_evaluator(item: dict, output: str) -> float:
    """Compute cosine similarity between expected and actual output."""
    expected_embedding = similarity_model.encode([item["expected_output"]])
    actual_embedding = similarity_model.encode([output])

    # Cosine similarity
    similarity = float(
        np.dot(expected_embedding, actual_embedding.T)
        / (
            np.linalg.norm(expected_embedding)
            * np.linalg.norm(actual_embedding)
        )
    )
    return similarity

Custom Evaluation Functions

For domain-specific quality checks, write custom evaluators:

from langfuse.batch_evaluation import CompositeEvaluatorFunction

def factuality_check(item: dict, output: str) -> float:
    """Check if the agent's response contains factual errors."""
    # Your factuality checking logic here
    # Return a score between 0.0 and 1.0
    return 0.95  # Example score


def tone_check(item: dict, output: str) -> float:
    """Check if the agent's tone is appropriate."""
    # Check for overly casual language, inappropriate content, etc.
    return 0.88  # Example score


# Combine multiple evaluators into a composite score
composite_evaluator = CompositeEvaluatorFunction(
    evaluators=[factuality_check, tone_check],
    aggregation="mean",  # or "weighted_mean", "min", "max"
)

Reading Evaluation Results

After running an evaluation, navigate to the Experiments tab in the Langfuse UI. You will see:

• Overall score — Aggregate quality metric across all examples.
• Per-example breakdown — Individual scores for each golden dataset item.
• Trend chart — Quality over time as you run successive evaluations.
• Regression alerts — Automatic flags when scores drop below configured thresholds.

# Programmatically check for regressions
from langfuse import RegressionError

try:
    result = client.run_experiment(
        name="research-agent-eval",
        dataset_name="research-agent-golden-set",
        task=evaluate_example,
        metadata={"agent_version": "1.1.0"},
    )

    # Check if the experiment score meets the threshold
    experiment = client.get_experiment(result)
    if experiment.avg_score < 0.8:
        raise RegressionError(
            result=experiment,
            metric="avg_score",
            value=experiment.avg_score,
            threshold=0.8,
        )

    print(f"Evaluation passed. Average score: {experiment.avg_score:.2f}")

except RegressionError as e:
    print(f"REGRESSION DETECTED: {e.metric} = {e.value} (threshold: {e.threshold})")
    # Block deployment, notify the team, etc.

Integrating Evaluations into Your CI/CD Pipeline

Run evaluations automatically on every pull request:

# .github/workflows/evaluate.yml
name: Evaluate Agent

on:
  pull_request:
    paths:
      - "src/agent.py"
      - "src/prompts/**"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install uv
        run: curl -LsSf https://astral.sh/uv/install.sh | sh

      - name: Install dependencies
        run: uv sync

      - name: Run evaluation
        env:
          LANGFUSE_PUBLIC_KEY: ${{ secrets.LANGFUSE_PUBLIC_KEY }}
          LANGFUSE_SECRET_KEY: ${{ secrets.LANGFUSE_SECRET_KEY }}
          LANGFUSE_HOST: ${{ secrets.LANGFUSE_HOST }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: uv run scripts/run_evaluation.py

      - name: Check regression threshold
        run: |
          # Parse evaluation results and fail if below threshold
          uv run scripts/check_regression.py --threshold 0.8

---

Next Steps

You now have Langfuse deployed, your agent instrumented, a golden dataset created, and evaluations running. Here is where to go next:

• Langfuse Best Practices — Our team's standards for production-grade observability, including trace retention policies, cost optimization, evaluation scheduling, and incident response patterns.
• LangGraph Best Practices — Production patterns for agent design, error handling, and testing.
• Azure Container Apps Best Practices — Revision management, scaling strategies, and security hardening.
• Langfuse Official Documentation — Comprehensive reference for the Langfuse API, SDK, and self-hosted configuration.
• Langfuse Python SDK Reference — Source code, examples, and migration guides for the Python SDK.

Start Evaluating Early

Do not wait until your agent is "ready." Set up a minimal golden dataset with 5-10 examples immediately. Add more examples as you discover edge cases in production. The earlier you start evaluating, the more historical data you accumulate for trend analysis and regression detection.