LLM Evals: How to Test and Measure AI Features in Production

Traditional software has unit tests because a function given the same input returns the same output, and you can assert on it. LLM features break that assumption. Outputs vary from run to run, "correct" is often a judgement call, and a prompt tweak that fixes one case quietly breaks five others. Teams that ship reliable AI features handle this with evals: systematic, repeatable measurement of quality. This guide covers how to build them, which metrics to use for RAG and agents, how to tell real improvements from noise, and how to wire everything into CI.

Why Every AI Feature Needs Evals

Consider a support assistant for an Indian e-commerce company. You change the system prompt to make answers shorter. Did quality improve? Without evals, someone tries five questions, the answers look fine, and it ships. A week later, refund-policy answers have started dropping the condition "within 7 days of delivery", because shorter answers trimmed it. Nobody tested refund questions.

Evals turn that into a measurable decision: 300 representative questions, scored on correctness, faithfulness to policy and length, compared before and after. They also let you answer questions that come up constantly in AI teams:

• Can we switch to a cheaper or faster model without losing quality?
• Does adding a reranker improve answers enough to justify 150 ms extra latency?
• Did the new model version from our provider change behaviour?
• Which kinds of questions fail most, so we know what to fix next?

The mindset shift is evaluation-driven development: write the eval cases for a behaviour before you tune the prompt for it, the way test-driven development writes the test first.

Types of Evals: A Map

Building a Golden Dataset

The dataset matters more than any metric. A beautiful scoring pipeline over unrepresentative cases produces confident, wrong conclusions.

Where cases should come from

1. Real user inputs. Sample from production logs (with PII removed and consent and policy respected). Before launch, collect inputs from internal dogfooding or support tickets.
2. Known failures. Every bug report or bad answer becomes a test case. This is the most valuable source over time.
3. Edge cases you design. Ambiguous questions, out-of-scope requests, Hinglish or mixed-language input, very long inputs, adversarial and injection attempts.
4. Synthetic expansion, carefully. Use an LLM to generate variations of real cases, then review them. Purely synthetic sets tend to be cleaner and easier than real traffic.

Stratify by slices

Tag every case with the slices that matter: intent (refund, delivery, account), language, difficulty, customer tier, whether the answer exists in the knowledge base. Then report scores per slice. An overall score of 88% can hide a 60% score on refund questions, which may be the slice that costs money.

What a case looks like

{"id": "refund-017", "slice": ["refund", "english"], "input": "I got a damaged phone 10 days ago, can I return it?", "reference": "Damaged items can be reported within 7 days of delivery; after that, a warranty claim with the brand applies.", "must_include": ["7 days"], "must_not_include": ["full refund guaranteed"], "critical": true}
{"id": "oos-003", "slice": ["out_of_scope"], "input": "Which stock should I buy today?", "reference": null, "expected_behavior": "politely decline and redirect to shopping help", "critical": false}

Write labelling guidelines alongside the data: what counts as correct, how strict to be about partial answers, how to treat answers that are correct but cite the wrong source. Two people labelling the same 30 cases and comparing disagreements is the fastest way to sharpen those guidelines. Version the dataset in Git like code.

Code-Based Checks: Use Them First

Many quality criteria can be checked deterministically, cheaply and without flakiness. Always exhaust these before reaching for an LLM judge.

• Format: valid JSON, matches a schema, required fields present (see structured outputs).
• Exact or normalised match for classification labels, extracted IDs, amounts and dates.
• Must include / must not include keywords or regexes (policy phrases, banned claims, competitor names).
• Length and language constraints.
• Citations that point to sources actually retrieved.
• Executable checks for code generation: does it compile, do tests pass.
• Tool-call checks for agents: correct tool, valid arguments.

Here is a minimal, dependency-free eval harness. The system under test is any function from input to output; scorers are plain functions; the report includes per-slice pass rates with confidence intervals.

import json, math, re
from collections import defaultdict

def wilson(passed, n, z=1.96):
    """95% Wilson score interval for a pass rate."""
    if n == 0:
        return (0.0, 0.0)
    p = passed / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return (centre - half, centre + half)

def check_case(case, output):
    failures = []
    for s in case.get("must_include", []):
        if s.lower() not in output.lower():
            failures.append(f"missing '{s}'")
    for s in case.get("must_not_include", []):
        if s.lower() in output.lower():
            failures.append(f"forbidden '{s}'")
    if len(output.split()) > case.get("max_words", 120):
        failures.append("too long")
    return failures

def run_suite(cases, system):
    by_slice = defaultdict(lambda: [0, 0])
    critical_failures = []
    for case in cases:
        out = system(case["input"])
        fails = check_case(case, out)
        for s in case["slice"] + ["ALL"]:
            by_slice[s][0] += not fails
            by_slice[s][1] += 1
        if fails and case.get("critical"):
            critical_failures.append((case["id"], fails))
    for s, (p, n) in sorted(by_slice.items()):
        lo, hi = wilson(p, n)
        print(f"{s:14s} {p:3d}/{n:<3d} = {p/n:6.1%}  95% CI [{lo:.1%}, {hi:.1%}]")
    return by_slice, critical_failures

# Toy system under test, replace with your real pipeline
def toy_system(q):
    if "return" in q or "refund" in q:
        return "Damaged items must be reported within 7 days of delivery."
    return "I can help with orders, deliveries and returns."

cases = [
    {"id": "r1", "slice": ["refund"], "input": "Can I return a damaged phone?",
     "must_include": ["7 days"], "critical": True},
    {"id": "r2", "slice": ["refund"], "input": "refund for late delivery?",
     "must_include": ["7 days"], "must_not_include": ["guaranteed"]},
    {"id": "o1", "slice": ["out_of_scope"], "input": "Which stock should I buy?",
     "must_not_include": ["buy"]},
]
_, crit = run_suite(cases, toy_system)
print("critical failures:", crit)
print("200 cases at 85%:", tuple(round(x, 3) for x in wilson(170, 200)))

Running it prints a per-slice table and, for the last line, an interval of roughly (0.794, 0.893). That interval is the most important number on this page: it tells you that a single run on 200 cases cannot distinguish 85% from 88%.

LLM-as-Judge: How to Make It Trustworthy

Some criteria cannot be checked with code: is the answer faithful to the policy, is it helpful, is the tone appropriate, does it actually answer the question? Here you use a model as the grader. Done carelessly, it produces numbers that look precise and mean little. Done well, it scales human judgement cheaply.

Rules for good judge prompts

1. One criterion per judge. Separate judges for faithfulness, relevance and tone. A single "rate quality 1–10" judge mixes everything and is hard to act on.
2. Binary or three-level scales. "Pass / fail" or "fully / partially / not supported" are far more consistent than 1–10 scores.
3. Write the rubric explicitly, with examples of each label, including borderline ones.
4. Ask for a short reasoning before the verdict and return structured output, so you can audit why.
5. Give the judge what it needs: the question, the answer, the retrieved context or reference answer. A judge without the source cannot assess faithfulness.
6. Use a capable model at low temperature, and pin its version so scores stay comparable over time.

You are grading whether an answer is supported by the provided context.

Context:
{context}

Question: {question}
Answer: {answer}

Rules:
- "supported": every factual claim in the answer is stated in or directly
  implied by the context.
- "partially_supported": at least one claim is supported and at least one
  claim is not in the context.
- "unsupported": the main claim is not in the context or contradicts it.
- Ignore style and length. Do not use outside knowledge.

Return JSON: {"reasoning": "<two sentences>", "label": "supported" |
"partially_supported" | "unsupported"}

Known judge biases

Calibrate against humans

Before trusting a judge, have humans label 100 or so outputs using the same rubric, then measure agreement between judge and humans. Report raw agreement and Cohen's kappa, which corrects for agreement by chance. If agreement is poor, the rubric is usually the problem, not the model. Re-check calibration whenever you change the judge model or prompt.

Metrics for RAG: Retrieval and Generation Separately

A RAG answer can fail because retrieval did not find the right passage, or because generation ignored or distorted it. One end-to-end score cannot tell you which, so measure both stages. (New to RAG? Read the RAG deep dive first.)

Retrieval metrics

Label, for each question, which chunks or documents are relevant. Then:

• Recall@k: fraction of relevant chunks that appear in the top k. The most important retrieval metric for RAG, because the generator cannot use what it never sees.
• Precision@k: fraction of the top k that are relevant. Low precision wastes context and can distract the model.
• MRR (mean reciprocal rank): average of 1/rank of the first relevant result. Rewards putting the right chunk near the top.
• nDCG@k: handles graded relevance and position together.

Worked example for one question with two relevant chunks, A and B, and retrieved top-5 list [X, A, Y, Z, B]: recall@5 = 2/2 = 1.0; precision@5 = 2/5 = 0.4; reciprocal rank = 1/2 (A is at rank 2); recall@3 = 1/2 = 0.5. If your generator only receives the top 3, you have a retrieval problem even though recall@5 looks perfect.

Generation metrics

• Faithfulness (groundedness): are the answer's claims supported by the retrieved context? Measured with a judge like the one above, often claim by claim. This is your hallucination metric.
• Answer relevance: does it address the question asked?
• Correctness: does it match the reference answer? Needs a gold answer.
• Abstention quality: when the answer is not in the knowledge base, does the system say so instead of guessing? Include unanswerable questions in the golden set specifically for this.
• Citation accuracy: do cited sources actually contain the cited claims?

Libraries such as Ragas implement many of these as reusable LLM-based metrics. They are a good starting point, but read their prompts and calibrate them on your data like any other judge.

Evaluating Agents and Multi-Step Systems

Agents add trajectories: sequences of model calls and tool calls. Evaluate at three levels:

• Outcome: did the task succeed? Check final state where possible (was the ticket created with the right fields, did the tests pass) rather than judging the final message.
• Trajectory: were the right tools called with valid arguments, in a sensible order, without unnecessary steps or loops? Did it ask for approval before risky actions?
• Efficiency: number of steps, tokens, cost and wall-clock time per task.

Because agents are stochastic, run each task several times. Two useful summaries: pass@k (succeeds at least once in k tries, relevant when a human picks the best attempt) and pass^k (succeeds in all k tries, relevant for reliability when the agent acts unattended). An agent at 80% per-run success has a pass^3 of about 0.8³ ≈ 51% if runs are independent, which is a sobering number for an unattended workflow. Also include adversarial tasks containing injected instructions; see prompt injection defences.

Is the Difference Real? Sample Sizes and Noise

The most common eval mistake is declaring victory on a difference smaller than the noise. Two sources of noise matter: sampling noise (your dataset is a sample of possible inputs) and generation noise (the same input gives different outputs).

Those come from the standard error of a proportion, √(p(1−p)/n). At n = 200 and p = 0.85: √(0.85 × 0.15 / 200) ≈ 0.025, and 1.96 × 0.025 ≈ 0.049. To reliably detect a 3-point improvement from 85% to 88% with two independent samples, the usual sample-size formula, n ≈ (1.96 + 0.84)² × (0.85·0.15 + 0.88·0.12) / 0.03², gives roughly 2,000 cases per version.

Practical ways to get trustworthy comparisons without 2,000 labelled cases:

• Paired comparison. Run both versions on the same cases and look only at cases where they disagree (McNemar's test). Shared easy and impossible cases cancel out, which needs far fewer cases.
• Repeat generations (three to five samples per case) and average, to reduce generation noise.
• Focus on slices. A targeted fix should show a large change on its slice and no regression elsewhere.
• Look at the diffs. Read the cases that flipped. Ten flips you understand beat a p-value you do not.

Running Evals in CI

Evals deliver value when they run automatically on every change to prompts, model configuration, retrieval settings or tool definitions. Treat prompts as code: in version control, reviewed, and tested.

A tiered setup

# tests/test_support_bot_evals.py
import json
import pytest
from evals.harness import run_suite          # the harness shown above
from app.bot import answer                    # your real pipeline

CASES = [json.loads(l) for l in open("evals/golden_smoke.jsonl", encoding="utf-8")]
THRESHOLDS = {"ALL": 0.85, "refund": 0.90}

@pytest.fixture(scope="module")
def results():
    return run_suite(CASES, answer)

def test_no_critical_failures(results):
    _, critical = results
    assert not critical, f"critical cases failed: {critical}"

@pytest.mark.parametrize("slice_name,minimum", THRESHOLDS.items())
def test_slice_thresholds(results, slice_name, minimum):
    by_slice, _ = results
    passed, n = by_slice[slice_name]
    assert passed / n >= minimum, f"{slice_name}: {passed}/{n} below {minimum:.0%}"

name: llm-evals
on:
  pull_request:
    paths: ["app/prompts/**", "app/bot/**", "evals/**"]
jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: pytest tests/test_support_bot_evals.py -q
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}

Keep costs and flakiness down by caching model responses keyed by (prompt template version, model, input), using code checks in the PR tier, and setting thresholds with margin below your current score so normal noise does not block merges. Tools such as promptfoo, DeepEval and Inspect provide this structure out of the box if you prefer not to write your own harness.

Online Evals: Measuring in Production

Offline evals tell you a change is safe to ship; production tells you whether it works. Instrument from day one (our production AI guide covers tracing):

• Explicit feedback: thumbs up or down, "was this helpful", with an optional reason. Low volume but high signal.
• Implicit signals: user rephrased the question, escalated to a human agent, copied the answer, abandoned the session. Often more abundant than explicit feedback.
• Sampled judge scoring: run your faithfulness and safety judges on a random sample of production traces daily, and alert on drops.
• Business metrics: deflection rate for support bots, resolution time, conversion. The ultimate measure, but slow and confounded.
• A/B tests for significant changes, with guardrail metrics such as escalation rate and complaint rate.

Close the loop: every week, review low-scored and negatively rated traces, add the instructive ones to the golden set, and fix the underlying issue. That flywheel is how eval sets become representative.

Common Eval Mistakes

• Vibes-based testing: trying five favourite prompts after each change.
• Only synthetic data, which is cleaner and easier than real traffic.
• One overall score that hides slice regressions.
• Uncalibrated judges treated as ground truth.
• 1–10 scales that drift and resist interpretation.
• Ignoring noise: shipping a "2-point improvement" measured on 80 cases.
• Testing the prompt in isolation when failures come from retrieval or tools.
• Letting the eval set go stale while the product and users change.
• Overfitting to the eval set: tuning prompts until the specific cases pass. Keep a held-out set you look at rarely.

How Evals Come Up in AI Engineering Interviews

Evaluation has become one of the main things interviewers use to separate engineers who have shipped AI features from those who have built demos. Expect questions like:

• "How do you know your RAG system works?" Strong answer: golden set from real queries, retrieval recall@k and faithfulness measured separately, judge calibrated against human labels, slice reporting, CI gates, production sampling.
• "You switched to a cheaper model and the eval score dropped from 87% to 85%. Ship it?" Discuss confidence intervals, paired comparison, which slices moved, the cost saving, and reading the flipped cases.
• "How would you evaluate a summarisation feature with no reference summaries?" Reference-free judges for coverage of key points and faithfulness, pairwise comparisons, human calibration.
• "What are the limitations of LLM-as-judge?" Position, verbosity and self-preference biases; mitigations.

If you are building a portfolio project, an eval suite is one of the most convincing things you can add: a README table showing recall@5, faithfulness and correctness for two retrieval configurations, with confidence intervals, says more about your engineering judgement than another chatbot demo. For system design rounds, mention evaluation and monitoring explicitly when you design an LLM application.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.