How to Evaluate LLM Features with DeepEval Before Shipping

An LLM feature can pass every manual check on Friday and quietly get worse on Monday, because someone tweaked a prompt or the provider updated a model. Evals are the fix: automated test cases that score the feature's output on metrics like faithfulness, answer relevancy and hallucination, so quality becomes a number you can protect.

DeepEval is a popular way to do this in Python because it works like pytest: evals live next to the code, run in CI and fail the build when a change makes quality worse. This guide covers why evals matter, how to set DeepEval up, how to pick metrics and write test cases, and how to run them in CI so prompts get the same regression protection as the rest of your code.

Key takeaways

• Evals turn "it feels better" into a pass rate you can track across prompt and model changes.
• Generate outputs with your real pipeline, so the eval tests the system, not a hand-written answer.
• Pick a few metrics that match the failure you fear most: faithfulness to sources, relevancy to the question, and hallucination against known facts.
• Run a focused suite in CI on every change to prompts, retrieval or model settings, with thresholds that leave headroom for judge noise.
• Every production failure becomes a test case, so the same bug can't come back.

Why evals matter

LLM output is non-deterministic and prompts are fragile. You tweak one line of a system prompt to fix one bad case, and three other cases silently get worse. Without evals you find out from users. "It looked good in my manual testing" is not a quality bar; it is a sample size of five and a recency bias toward the cases you happened to try.

Evals turn "feels better" into a number. You build a small dataset of representative inputs with expected behaviour, you score each output against metrics, and you get a pass rate you can track across prompt and model changes. When you upgrade a model or refactor a prompt, the suite tells you whether quality held. This is the core of LLMOps: treat prompts and model configuration as code, and gate changes on measurable quality.

Setting up DeepEval

DeepEval is a Python package. Most of its metrics are themselves LLM-judged (a strong model scores your output against the input and context), so you set an API key for the judge model.

pip install deepeval
export OPENAI_API_KEY="..."   # used by the default judge model

The unit you test is the LLMTestCase. It holds the input, your system's actual output, and, depending on the metric, the retrieval context and an expected output.

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What is the refund window for digital products?",
    actual_output="Digital products can be refunded within 14 days of purchase.",
    retrieval_context=[
        "Refunds: Physical goods within 30 days. "
        "Digital products within 14 days of purchase, if unused."
    ],
)

actual_output is what your feature produced for that input; call your own pipeline to generate it. retrieval_context is the text your RAG step pulled in, which the faithfulness metric checks the output against. The hallucination metric uses a separate context field instead: the known-correct facts the answer must not contradict.

Picking metrics

DeepEval ships a set of metrics. Three cover most of what a RAG or Q&A feature needs.

Faithfulness checks that the output is supported by the retrieval context. It catches the most damaging failure: a confident answer that contradicts your own source data.

Answer relevancy checks that the output actually addresses the input question, rather than wandering or padding.

Hallucination checks the output for contradictions against context, the facts you know to be true. Faithfulness measures the answer against what your retriever returned; hallucination measures the generator against ground truth.

Each metric takes a threshold, the minimum score (0 to 1) to pass, and explains its verdict, which is useful when a case fails and you need to know why.

from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    HallucinationMetric,
)

faithfulness = FaithfulnessMetric(threshold=0.8)
relevancy = AnswerRelevancyMetric(threshold=0.7)
hallucination = HallucinationMetric(threshold=0.7)  # needs `context` on the test case

In current versions of DeepEval, all three scores are higher-is-better, and a case passes at or above its threshold. Older tutorials describe the hallucination score the other way round, so read each metric's current definition before trusting a number.

Writing test cases

Build a dataset from real and adversarial inputs: questions users actually ask, plus the edge cases that broke before. For each, generate the output with your real pipeline so the eval tests the system, not a hand-written answer.

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric

from my_app.rag import answer_question  # your pipeline under test

# (input question, the context your retriever should surface)
DATASET = [
    (
        "What is the refund window for digital products?",
        ["Digital products within 14 days of purchase, if unused."],
    ),
    (
        "Can I return a physical item after a month?",
        ["Physical goods can be refunded within 30 days of delivery."],
    ),
]


@pytest.mark.parametrize("question, context", DATASET)
def test_refund_answers(question, context):
    output = answer_question(question)

    test_case = LLMTestCase(
        input=question,
        actual_output=output,
        retrieval_context=context,
    )

    assert_test(
        test_case,
        [
            FaithfulnessMetric(threshold=0.8),
            AnswerRelevancyMetric(threshold=0.7),
        ],
    )

assert_test runs each metric and fails the test if any score misses its threshold. Because this is parametrized pytest, every dataset row is its own reported test, so a failure points at the exact input that regressed.

Running evals in CI

The whole point is automation. Run the suite with the DeepEval test runner, which gives you a metrics summary on top of the normal pass/fail.

deepeval test run tests/test_refund_answers.py

Then gate it in CI. Evals run on every pull request that touches prompts, retrieval, or model config, and a failed eval blocks the merge.

# .github/workflows/evals.yml
name: LLM evals
on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/rag/**"
      - "tests/test_*_answers.py"

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: deepeval test run tests/
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Two practical notes. LLM-judged metrics cost API calls and add latency, so keep the CI dataset focused: 20 to 50 well-chosen cases beat 500 redundant ones. And because judges are themselves non-deterministic, set thresholds with a little headroom so a borderline score does not flap red and green between runs. If flakiness is a problem, run the judge at low temperature and average a couple of samples.

Treat prompts like code

A prompt is a dependency your feature's behaviour rests on, so version it and protect it with regression tests. Keep prompts in files under source control, not inline strings buried in functions. When you change one, the eval suite is your regression check: it tells you whether the change you made to fix one case quietly broke five others.

When a real failure ships, do not just patch the prompt. Add the failing input to the dataset first, confirm it fails, then fix it and confirm it passes. That is the LLM version of a regression test, and it stops the same bug from coming back after the next prompt edit. Over time your dataset becomes a record of every way the feature has failed, which is exactly the suite you want guarding the next model upgrade.

FAQ

What are LLM evals?

Automated tests for features built on language models. Each test case pairs an input with the feature's real output, and metrics score that output for qualities such as faithfulness to sources, relevance to the question and absence of hallucinations. A pass rate across the dataset tells you whether a change helped or hurt.

Which DeepEval metrics should I start with?

For a RAG or question-answering feature, start with faithfulness (is the answer supported by retrieved text?) and answer relevancy (does it address the question?). Add hallucination checks against known facts when you have reliable ground truth to compare with.

How many test cases does an eval suite need?

Start with 20 to 50 well-chosen cases: common questions, known edge cases and every past failure. LLM-judged metrics cost API calls, so a focused suite in CI beats a huge one, and you can run a larger set nightly.

Are LLM-judged evals reliable?

Reliable enough to catch regressions when used carefully. Judges are themselves non-deterministic, so leave headroom in thresholds, run the judge at low temperature, and review failing cases rather than trusting a single score blindly.

Next steps

If you're shipping an LLM feature and want a safety net before the next prompt or model change, I can help you set up an eval suite and wire it into CI. Get in touch, or see my services.

Sources

• DeepEval documentation, Confident AI
• Faithfulness metric, DeepEval
• Hallucination metric, DeepEval
• Answer relevancy metric, DeepEval