Building an AI agent that works in a notebook is straightforward. Shipping one that performs reliably at scale is a different challenge entirely. The gap between the two lies in evaluation: how you measure what works, benchmark changes, and continuously improve.
I discovered this the hard way. A prompt change that seemed minor in testing reduced our agent’s average latency by 40% and improved accuracy by 15% when we finally measured it properly. But we only found it because we’d built an evaluation framework first. Without systematic measurement, that improvement would have remained invisible.
This is the quality engineering layer of AI that separates hobby projects from production systems. Let’s walk through how to build it.
Why Standard Testing Breaks for LLM Agents
Traditional software testing assumes deterministic behavior. Given input X, you get output Y every time. LLMs don’t work that way. The same prompt produces slightly different responses each time, even at zero temperature. This variability makes it impossible to test agents like you’d test a REST API.
You also can’t just count correct answers. An agent might produce the right final result through a chain of reasoning, or it might get lucky. You need to measure what actually matters: accuracy on your specific use case, latency at the 95th percentile, cost per inference, hallucination rate, and whether the agent follows your guardrails.
This requires three things: clear success metrics, a representative benchmark dataset, and a framework to run evaluations repeatedly and compare results.
Defining Success Metrics That Matter
Start by being explicit about what good looks like for your agent. Here are the metrics that tend to matter in production:
- Accuracy: Does the agent produce correct outputs for your use case? This is domain-specific. For a customer support agent, accuracy might mean resolving the customer’s issue. For a data analysis agent, it might mean returning the correct answer to a question.
- Latency: How fast does the agent respond? Track both mean and percentiles (p50, p95, p99). Agents that think too long frustrate users and cost more to run.
- Cost: How many tokens does each agent call consume? With large models, this adds up quickly. Measure tokens per successful interaction, not just per call.
- Hallucination rate: How often does the agent invent facts or make up tool outputs? Even if the final answer seems correct, hallucinations erode trust.
- Tool accuracy: If your agent calls external tools, does it call them correctly? Does it parse the results accurately?
- Safety: Does the agent refuse harmful requests? Does it stay within your guardrails?
You won’t measure all of these on day one. Start with one or two that directly impact your business. For most agents, accuracy and latency are the place to begin.
Building a Benchmark Dataset
An evaluation is only as good as the data you test against. You need a dataset of representative inputs that covers the real scenarios your agent will face.
Start small. Collect 20 to 50 realistic examples that span your use cases. If your agent handles customer support, include simple requests, complex multi-step issues, edge cases, and requests where the agent should refuse. Annotate each example with the correct output or desired behavior.
Here’s what a simple benchmark might look like for a document Q&A agent:
[
{
"id": "qa_001",
"query": "What is the refund policy?",
"document": "...",
"expected_answer": "Refunds are available within 30 days of purchase.",
"category": "policy_lookup"
},
{
"id": "qa_002",
"query": "Can you tell me a joke?",
"document": "...",
"expected_behavior": "refuse",
"category": "out_of_scope"
}
]
Version control this dataset. As you find edge cases or failures in production, add them to your benchmark. Over time, it becomes a regression test suite that prevents regressions as you iterate.
Running Evaluations Programmatically
Once you have a metric and a dataset, you need a harness to run your agent against the dataset and score the results. This is where evaluation frameworks come in.
For some metrics, you can score automatically. For others, you need a human or an LLM judge. Here’s a simple example using an LLM to evaluate accuracy:
from langchain.chat_models import ChatOpenAI
from langchain.prompts import PromptTemplate
def evaluate_accuracy(agent_output, expected_output, query):
"""Use an LLM to judge if agent output is accurate."""
evaluator = ChatOpenAI(model="gpt-4")
prompt = PromptTemplate.from_template(
"""You are an evaluator. Judge if the agent's response correctly answers the user's query.
Query: {query}
Expected: {expected}
Agent output: {actual}
Respond with CORRECT or INCORRECT, then explain briefly."""
)
result = evaluator.predict(
query=query,
expected=expected_output,
actual=agent_output
)
return "CORRECT" in result.upper()
def run_benchmark(agent, dataset):
"""Run agent against benchmark and collect metrics."""
results = []
for example in dataset:
output = agent.run(example["query"])
is_correct = evaluate_accuracy(
output,
example["expected_answer"],
example["query"]
)
results.append({
"example_id": example["id"],
"correct": is_correct,
"output": output
})
accuracy = sum(r["correct"] for r in results) / len(results)
return {"accuracy": accuracy, "results": results}
This is a minimal example. In production, you’d also measure latency, token usage, and cost. Tools like LangSmith and Ragas provide more sophisticated evaluation out of the box, but the principle is the same: run your agent, collect outputs, score them, and aggregate metrics.
A/B Testing Prompt Variations
Once you have a baseline, you can test prompt changes systematically. Create two versions of your prompt, run both against your benchmark, and compare the results.
Here’s a practical example. Suppose your agent’s accuracy is 82%. You hypothesize that adding a few examples to the system prompt will help. You’d create two variants:
Version A (baseline):
You are a helpful customer support agent. Answer customer questions about our products.
Version B (with examples):
You are a helpful customer support agent. Answer customer questions about our products.
Examples of good responses:
- Q: Can I return my order? A: Yes, we offer 30-day returns. Visit our returns portal to start.
- Q: What's your shipping time? A: Standard shipping takes 5-7 business days. Express shipping is 2-3 days.
Always be specific and provide actionable information.
Run both against your benchmark dataset. If Version B consistently scores higher across your test set, deploy it. If the difference is within noise, keep investigating.
Track these experiments. After a few iterations, you’ll develop intuition for what works: more specific instructions, examples of correct behavior, constraints on output format, or clearer role definitions.
Integrating Evaluation into Your Pipeline
Evaluation shouldn’t be a manual step. Integrate it into your CI/CD pipeline so every prompt change is automatically tested before it reaches production.
A simple workflow looks like this:
- Developer commits a prompt change.
- CI pipeline checks out the code, loads the benchmark dataset, and runs the evaluation harness.
- Results are compared against the baseline. If accuracy drops more than 2%, the build fails.
- Developer sees the failure, investigates, and either fixes the prompt or updates the benchmark if the change is intentional.
- On merge, the new baseline is recorded for future comparisons.
In a GitHub Actions workflow, this might look like:
name: Agent Evaluation
on: [pull_request, push]
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: pip install -r requirements.txt
- name: Run evaluation
run: python scripts/evaluate.py --dataset benchmarks/dataset.json --output results.json
- name: Compare against baseline
run: python scripts/compare_results.py results.json benchmarks/baseline.json --threshold 0.02
- name: Upload results
uses: actions/upload-artifact@v3
with:
name: evaluation-results
path: results.json
This ensures that every change is measured and compared. Over time, you build confidence that your agent is improving, not regressing.
Tools and Frameworks
Several tools can accelerate this work. Start with a custom harness if your needs are simple. As your evaluation requirements grow, consider adopting a framework to reduce boilerplate and standardize how you measure quality.
LangSmith: LangChain’s platform for debugging and evaluating chains and agents. It captures execution traces, makes it easy to run custom evaluators, and stores results for comparison. Use this if you’re already in the LangChain ecosystem or need a managed platform.
Ragas: An open-source framework specifically for evaluating RAG pipelines. It includes metrics like context relevance, answer relevance, and hallucination detection. Pick this if you’re building retrieval-augmented systems.
DeepEval: Another open-source framework for LLM evaluation with pre-built metrics and a simple API. It’s lightweight and good for getting started quickly.
Custom harnesses: For domain-specific metrics or when you need full control, write your own evaluation code. Keep it simple, version it with your agent code, and document what each metric measures.
Continuous Improvement Patterns
Evaluation is only useful if it feeds back into development. Here are patterns that work in practice:
Weekly prompt review: Each week, run your agent against your benchmark. Look for categories where accuracy has dropped. Investigate and iterate on the prompt.
Production failures as test cases: When your agent fails in production, add that example to your benchmark. This ensures you don’t regress on the same issue twice.
Cost tracking: Monitor token usage alongside accuracy. A prompt that’s 2% more accurate but costs 30% more might not be worth it. Make these tradeoffs explicit.
Multi-model evaluation: Test your prompt against different models (GPT-4, Claude, Llama). A prompt that works great on GPT-4 might not work on Claude. Build robustness.
Seasonal regression testing: Before major deployments or during high-traffic periods, run your full benchmark suite. Catch regressions early.
Getting Started
You don’t need a perfect evaluation system on day one. Start here:
- Define one or two key metrics for your agent (accuracy and latency are a good start).
- Collect 20 to 30 representative examples and annotate them with expected outputs.
- Write a simple evaluation script that runs your agent against this dataset and scores the results.
- Run it once. Record the baseline.
- Make a small prompt change and run it again. Compare.
- If the results improve and make sense, commit the change. If not, revert and try something else.
After a few iterations, you’ll have a working evaluation loop. Integrate it into CI/CD. Expand your metrics as you learn what matters most for your use case.
This is how you move from a prototype to a production system: measure what works, iterate systematically, and build confidence that your agent is getting better, not worse.
What’s the difference between evaluation and testing for LLM agents?
Traditional testing checks for deterministic behavior: given input X, you always get output Y. LLMs produce variable outputs, so you can’t test them that way. Evaluation instead measures probabilistic properties: across many runs, does the agent produce correct answers a high percentage of the time? Does it stay within acceptable latency? Evaluation is about measuring quality across distributions, not checking for exact matches.
How many examples do I need in a benchmark dataset?
Start with 20 to 50 examples that cover your main use cases and edge cases. This is enough to catch major regressions and trends. As your system matures and you add more metrics, grow your dataset to 100 to 500 examples. The key is that examples should be representative of real production scenarios, not random. A small, well-curated dataset is more useful than a large one that doesn’t reflect actual usage.
Can I use an LLM to evaluate other LLMs?
Yes, and it’s often practical. An LLM can judge whether an agent’s response is accurate, on-topic, or follows guidelines. However, LLM judges have their own biases and can be inconsistent. Use them for subjective qualities (tone, helpfulness) and pair them with automated metrics for objective ones (correctness, latency, cost). Always validate LLM judge results on a sample of cases with human review to ensure they’re reliable.
How often should I run evaluations?
Run evaluations every time you change a prompt or agent behavior, before merging code. Automate this in CI/CD so it happens without manual intervention. Additionally, run your full benchmark suite weekly or before major deployments. In production, continuously monitor your agent’s performance on live traffic and add failing cases to your benchmark to prevent regression.
What’s a good accuracy target for a production agent?
It depends on your use case. A customer support agent might need 90% accuracy to avoid frustrating users. A data analysis agent might need 95% or higher because incorrect data is worse than no answer. A content moderator might need 99% to avoid letting harmful content through. Define your target based on business impact, then work backward to identify what prompt changes, training data, or models will get you there.
