Deploying a language model to production is only half the battle. The harder part is knowing whether it’s actually working. Without systematic evaluation, you ship models that degrade silently, RAG pipelines that hallucinate, and agentic systems that drift from their intended behavior. This guide walks you through building evaluation frameworks that catch these problems early and keep your LLM applications reliable at scale.
Why LLM Evaluation Matters
Traditional software testing gives you deterministic outputs. Feed input X, expect output Y. Language models don’t work that way. The same prompt can produce different outputs on different days, different models, or even different inference runs. Evaluation is how you measure whether those variations stay within acceptable bounds.
Without it, you’re flying blind. A RAG pipeline might start returning irrelevant context. A fine-tuned model might forget its instructions. An agent might loop endlessly. You won’t know until a user reports it or your monitoring catches something. Evaluation frameworks let you detect these issues before they reach production.
Core Evaluation Metrics
Start by choosing metrics that match your use case. Not all metrics work for all applications.
String-Based Metrics
BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) are fast, deterministic, and work well for generation tasks where exact phrasing matters. BLEU compares n-gram overlap between generated and reference text. ROUGE measures recall of overlapping units. Both are useful for summarization, translation, and structured output tasks.
The trade-off: they penalize semantically correct answers that use different wording. If your reference says “the capital of France” and the model says “Paris”, BLEU scores that as a miss even though it’s correct.
Use these for tasks where the expected output is relatively fixed, like code generation or form filling.
Semantic Similarity Metrics
Embedding-based similarity captures meaning beyond exact word matches. You encode both the generated output and reference answer into a vector space, then compute cosine similarity. This handles paraphrasing and semantic equivalence much better than string metrics.
Libraries like Sentence Transformers make this straightforward:
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer('all-MiniLM-L6-v2')
generated = "Paris is the capital of France"
reference = "The capital city of France is Paris"
generated_embedding = model.encode(generated, convert_to_tensor=True)
reference_embedding = model.encode(reference, convert_to_tensor=True)
similarity = util.pytorch_cos_sim(generated_embedding, reference_embedding)
print(f"Similarity: {similarity.item():.3f}")
This approach works well for QA, retrieval tasks, and any scenario where multiple correct answers exist. The downside: it requires more compute than string metrics and depends on the quality of your embedding model.
Task-Specific Metrics
For specialized tasks, build custom metrics. Evaluating a RAG pipeline? Check whether the retrieved context actually appears in the final answer. Evaluating an agent? Count successful task completions. Evaluating a classifier? Use precision, recall, and F1 score.
The key is making metrics measurable and reproducible. Avoid vague criteria like “is the response helpful?” Instead, define concrete checks: Does the response cite sources? Is the response under 500 tokens? Does it follow the requested format?
Building an Evaluation Harness
A basic evaluation harness loads test cases, runs your model against them, computes metrics, and reports results. Here’s a minimal structure:
from dataclasses import dataclass
from typing import Callable, Any
import json
@dataclass
class TestCase:
input_text: str
expected_output: str
metadata: dict = None
@dataclass
class EvaluationResult:
test_case: TestCase
generated_output: str
metrics: dict
passed: bool
class EvaluationHarness:
def __init__(self, model_func: Callable, metrics: dict[str, Callable]):
self.model_func = model_func
self.metrics = metrics
def evaluate(self, test_cases: list[TestCase]) -> list[EvaluationResult]:
results = []
for test_case in test_cases:
generated = self.model_func(test_case.input_text)
metric_scores = {
name: metric_fn(generated, test_case.expected_output)
for name, metric_fn in self.metrics.items()
}
passed = all(score >= 0.8 for score in metric_scores.values())
results.append(EvaluationResult(
test_case=test_case,
generated_output=generated,
metrics=metric_scores,
passed=passed
))
return results
def report(self, results: list[EvaluationResult]) -> dict:
total = len(results)
passed = sum(1 for r in results if r.passed)
avg_metrics = {}
for metric_name in self.metrics.keys():
scores = [r.metrics[metric_name] for r in results]
avg_metrics[metric_name] = sum(scores) / len(scores)
return {
'total_tests': total,
'passed': passed,
'pass_rate': passed / total if total > 0 else 0,
'average_metrics': avg_metrics,
'failed_cases': [r for r in results if not r.passed]
}
# Usage
def dummy_model(text: str) -> str:
return "Generated response"
test_cases = [
TestCase("What is 2+2?", "4"),
TestCase("What is the capital of France?", "Paris"),
]
harness = EvaluationHarness(
model_func=dummy_model,
metrics={
'semantic_similarity': lambda gen, ref: 0.95, # placeholder
}
)
results = harness.evaluate(test_cases)
report = harness.report(results)
print(json.dumps(report, indent=2, default=str))
This gives you a foundation. Extend it to handle batching, parallel evaluation, timeout handling, and cost tracking as your needs grow.
Using Evaluation Frameworks
Rather than building everything from scratch, consider frameworks that handle the infrastructure for you.
Ragas
Ragas focuses on RAG pipeline evaluation. It provides metrics specifically for retrieval-augmented generation: context relevance, answer relevance, faithfulness (whether the answer is grounded in the context), and semantic similarity. It integrates with LangChain and works with any LLM backend.
from ragas import evaluate
from ragas.metrics import (
context_relevance,
faithfulness,
answer_relevance,
answer_correctness
)
# Your RAG results
rag_results = {
'question': ["What is machine learning?"],
'answer': ["Machine learning is..."],
'contexts': [["Context chunk 1", "Context chunk 2"]],
'ground_truth': ["ML is a subset of AI..."]
}
scores = evaluate(
rag_results,
metrics=[context_relevance, faithfulness, answer_relevance, answer_correctness]
)
print(scores)
Ragas handles the LLM calls needed for semantic evaluation and abstracts away the complexity. The trade-off is cost (each evaluation calls an LLM) and latency.
DeepEval
DeepEval is a framework for building custom evaluation logic. It provides decorators and assertions that make it easy to define test cases and run them at scale. It integrates with pytest and CI/CD tools.
from deepeval import assert_test
from deepeval.metrics import SimilarityMetric, FactualCorrectness
def test_chatbot_response():
response = chatbot.query("What is Python?")
assert_test(
actual_output=response,
expected_output="Python is a programming language",
metric=SimilarityMetric()
)
def test_factual_accuracy():
response = llm.generate("Name the first president of the US")
assert_test(
actual_output=response,
metric=FactualCorrectness()
)
DeepEval shines when you want to treat evaluation like unit testing. Write tests, run them as part of your workflow, and get pass/fail results.
Braintrust
Braintrust provides a managed platform for evaluation with experiment tracking, comparison, and collaborative review. It’s useful if you want to track evaluation runs over time and compare model versions.
import braintrust
project = braintrust.init_project(name="llm-evals")
with project.experiment(name="model-v2") as exp:
for test_case in test_cases:
output = model(test_case.input_text)
exp.log(
input=test_case.input_text,
output=output,
expected=test_case.expected_output,
scores={'similarity': compute_similarity(output, test_case.expected_output)}
)
The benefit: centralized tracking and easy comparison between model versions. The cost: vendor lock-in and additional infrastructure.
Evaluating RAG Pipelines
RAG evaluation is harder than single-model evaluation because multiple components can fail. The retriever might return irrelevant context. The reranker might rank correctly but poorly. The LLM might hallucinate despite good context. Your evaluation needs to isolate these failures.
Structure your evaluation in layers:
- Retrieval layer: Does the retriever return relevant documents? (Measure with precision, recall, or MRR)
- Reranking layer: Does the reranker improve ranking quality? (Compare ranking before and after)
- Generation layer: Does the LLM produce faithful, relevant answers? (Use faithfulness and semantic similarity metrics)
- End-to-end layer: Does the full pipeline produce correct answers? (Measure with answer correctness or human rating)
This breakdown lets you identify which component is causing failures. If retrieval is fine but generation is poor, you know to focus on prompt engineering or model selection.
Evaluating Agentic Workflows
Agents are harder to evaluate because they take variable-length paths to reach answers. A simple metric won’t work. Instead, measure outcomes:
- Did the agent reach a correct final answer?
- How many steps did it take?
- Did it get stuck in loops?
- Did it use the right tools in the right order?
Build evaluation around these dimensions. For example:
class AgentEvaluator:
def evaluate(self, agent_trace: dict) -> dict:
final_answer = agent_trace['final_answer']
steps = agent_trace['steps']
tool_calls = [s for s in steps if s['type'] == 'tool_call']
return {
'answer_correct': final_answer == expected_answer,
'steps_taken': len(steps),
'tools_used': [t['tool_name'] for t in tool_calls],
'had_loops': self._detect_loops(steps),
'efficiency_score': self._compute_efficiency(steps),
}
def _detect_loops(self, steps: list) -> bool:
tool_sequence = [s['tool_name'] for s in steps if s['type'] == 'tool_call']
# Simple loop detection: same tool called 3+ times in a row
for i in range(len(tool_sequence) - 2):
if tool_sequence[i] == tool_sequence[i+1] == tool_sequence[i+2]:
return True
return False
def _compute_efficiency(self, steps: list) -> float:
# Fewer steps is more efficient
return 1.0 / (1.0 + len(steps))
The key: define what success looks like for your agent before you build it. Then measure against those criteria consistently.
CI/CD Integration and Quality Gates
Evaluation is only useful if it’s automated and enforced. Integrate evaluation into your CI/CD pipeline so every model change is tested before it reaches production.
A typical workflow looks like this:
- Developer commits a model change or prompt update
- CI pipeline triggers evaluation on a test set
- Metrics are computed and compared against baseline
- If metrics degrade beyond a threshold, the build fails
- If metrics pass, the model advances to staging or production
Here’s a GitHub Actions example:
name: LLM Evaluation
on: [push, pull_request]
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Install dependencies
run: |
pip install -r requirements.txt
- name: Run evaluations
run: python scripts/evaluate.py
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- name: Check metrics
run: python scripts/check_thresholds.py
- name: Upload results
uses: actions/upload-artifact@v3
with:
name: evaluation-results
path: results/
And the threshold check script:
import json
with open('results/metrics.json') as f:
current = json.load(f)
with open('results/baseline.json') as f:
baseline = json.load(f)
thresholds = {
'accuracy': {'min': 0.85, 'max_drop': 0.05},
'latency_ms': {'max': 2000},
'cost_per_call': {'max': 0.01}
}
failed = False
for metric, rules in thresholds.items():
current_val = current.get(metric)
baseline_val = baseline.get(metric)
if 'min' in rules and current_val < rules['min']:
print(f"FAIL: {metric} below minimum ({current_val} < {rules['min']})")
failed = True
if 'max_drop' in rules and baseline_val:
drop = (baseline_val - current_val) / baseline_val
if drop > rules['max_drop']:
print(f"FAIL: {metric} degraded by {drop:.1%}")
failed = True
if 'max' in rules and current_val > rules['max']:
print(f"FAIL: {metric} exceeds maximum ({current_val} > {rules['max']})")
failed = True
if failed:
exit(1)
print("All checks passed")
This ensures that every model change is validated before deployment. Set thresholds based on your business requirements. For a customer-facing application, be strict. For experimental features, you can be more lenient.
Handling Scale and Cost
As your test set grows, evaluation becomes expensive. A few strategies to manage this:
Cache results: Store evaluation outputs so you don’t recompute metrics for the same inputs and model versions. This is especially important for LLM-based metrics that call external APIs.
Sample intelligently: Don’t evaluate every test case every time. Sample a representative subset, or use stratified sampling to ensure coverage across different input types.
Use cheaper metrics first: Run fast string-based metrics first. Only compute expensive semantic or LLM-based metrics if the fast metrics pass. This is a simple but effective optimization.
Batch evaluation: Group test cases and process them in parallel. Libraries like Ray or Dask make this straightforward.
import ray
@ray.remote
def evaluate_batch(test_cases: list[TestCase], model_func) -> list[EvaluationResult]:
results = []
for test_case in test_cases:
output = model_func(test_case.input_text)
# compute metrics
results.append(result)
return results
# Split test cases into batches
batch_size = 100
batches = [test_cases[i:i+batch_size] for i in range(0, len(test_cases), batch_size)]
# Evaluate in parallel
futures = [evaluate_batch.remote(batch, model_func) for batch in batches]
results = ray.get(futures)
# Flatten results
all_results = [r for batch in results for r in batch]
Monitor your evaluation costs and adjust your strategy as you scale. What works for 100 test cases might not work for 10,000.
Putting It Together
A production LLM evaluation system typically includes:
- A test set of representative cases covering different scenarios and edge cases
- A set of metrics chosen for your specific task (string-based, semantic, and task-specific)
- An evaluation harness that runs tests consistently and reports results
- Integration with your model serving infrastructure so you can evaluate the exact model in production
- CI/CD automation that runs evaluation on every change and enforces quality gates
- Monitoring and alerting so you catch degradation after deployment
- Cost and latency tracking so you understand the trade-offs of your evaluation strategy
Start simple. Pick one metric, evaluate one test set, and integrate it into your CI/CD. Then expand from there. Add more metrics as you understand your failure modes. Add monitoring as you deploy to production. The key is to start measuring early and make evaluation a core part of your development process, not an afterthought.
Evaluation won’t make bad models good, but it will make sure you know when something breaks. That’s the foundation of reliable AI systems.
What’s the difference between BLEU and semantic similarity metrics?
BLEU measures exact n-gram overlap between generated and reference text. It’s fast and deterministic but penalizes semantically correct answers that use different wording. Semantic similarity uses embeddings to capture meaning, so it handles paraphrasing better, but it requires more compute and depends on your embedding model’s quality. Use BLEU for tasks with fixed expected output like code generation. Use semantic similarity for open-ended tasks like QA where multiple correct answers exist.
How do I evaluate a RAG pipeline if multiple components can fail?
Layer your evaluation by component: measure retriever quality (precision, recall), reranker quality (ranking improvements), generation quality (faithfulness, relevance), and end-to-end correctness. This breakdown isolates which component is causing failures. If retrieval is fine but generation is poor, focus on prompt engineering or model selection. If retrieval is poor, focus on your embedding model or vector database.
Can I use the same evaluation framework for all LLM tasks?
Not effectively. Different tasks need different metrics. Summarization benefits from ROUGE. Translation from BLEU. RAG from context relevance and faithfulness. Agents from success rate and step count. Start with task-specific metrics that match your business goals. Generic metrics like semantic similarity can complement task-specific ones, but shouldn’t replace them.
How do I handle the cost of LLM-based evaluation metrics?
Use a tiered approach: run fast string-based metrics first, then semantic similarity, then expensive LLM-based metrics only if needed. Cache results so you don’t recompute for the same inputs. Sample intelligently instead of evaluating every test case. Batch evaluation to process multiple cases in parallel. Monitor your evaluation costs and adjust your strategy as you scale.
What’s a reasonable pass rate threshold for CI/CD quality gates?
It depends on your task and risk tolerance. For customer-facing applications, aim for 95%+ pass rate. For internal tools or experimental features, 85%+ might be acceptable. Set thresholds based on your business requirements and error impact. Also set thresholds for metric degradation: if accuracy drops more than 5% compared to the previous version, fail the build. Review and adjust thresholds as you learn what’s realistic for your models.
