LLM Evaluation in Python: Build an Eval Harness From Scratch

You change the system prompt behind an LLM feature, try a few examples, and think you’ve fixed the bug. The new outputs look better when you inspect them manually, so you deploy the change. All is well. That is, until you discover that a separate class of requests has regressed or that responses seem worse on average. This is the problem with making ad hoc prompt changes without systematic LLM evaluation.

Ad hoc changes and manual inspection are useful while you’re experimenting, but they don’t give you a repeatable regression signal. You also find it increasingly difficult to distinguish stochastic variation from real improvement. In this tutorial, you’ll build an LLM evaluation harness from scratch that catches potential output regressions and turns product expectations into verifiable tests.

By the end of this tutorial, you’ll understand that:

• LLM evaluation tests model outputs against explicit criteria, so you can tell whether a prompt or model change made things better or worse.
• Deterministic checks should handle anything code can verify, and an LLM judge should handle only what code can’t, like helpfulness or tone.
• LLM judges are trustworthy only after you calibrate them against human-labeled examples.
• Release gates block changes that make a previously passing case fail a blocker criterion, even if the average scores improve.

Along the way, you’ll build a framework-agnostic Python command-line program that compares two prompt versions on the same test cases and returns a PASS, FAIL, or REVIEW decision, with a matching exit code you can wire into CI. Here’s the final comparison you’ll reproduce:

Baseline: support_v1
Candidate: support_v3_guarded

Criterion                  Baseline  Candidate     Change
---------------------------------------------------------
policy_correctness             100%       100%      +0 pp
  verdicts (P/F/R)            4/0/0      4/0/0
required_next_step             100%       100%      +0 pp
  verdicts (P/F/R)            4/0/0      4/0/0
safety_and_privacy             100%       100%      +0 pp
  verdicts (P/F/R)            4/0/0      4/0/0
helpfulness                    4.00       4.00      +0.00
  verdicts (P/F/R)            4/0/0      4/0/0
professional_tone              4.00       4.00      +0.00
  verdicts (P/F/R)            4/0/0      4/0/0
concision                      3.00       4.00      +1.00
  verdicts (P/F/R)            4/0/0      4/0/0

New critical failures: 0
New blocker failures:  0
Carryover failures:    0
Fixed failures:        0
Cases needing review:  0

Release decision: PASS
Reason: The candidate introduced no new blocker failures.

The output compares two prompt versions across six evaluation dimensions, separates new failures from ones that already existed, and ends with a clear release decision.

Take the Quiz: Test your knowledge with our interactive “LLM Evaluation in Python: Build an Eval Harness From Scratch” quiz. You’ll receive a score upon completion to help you track your learning progress:

---

Interactive Quiz

LLM Evaluation in Python: Build an Eval Harness From Scratch

Test your understanding of LLM evaluation in Python, from rubrics and code checks to LLM judges, calibration, and release gates.

Project Overview

You’ll build an LLM evaluator for a customer-support bot. The bot receives a customer message and the relevant support policy, then generates a reply. Return eligibility has an authoritative answer, while qualities such as helpfulness and tone require semantic judgment.

Your harness will evaluate LLM output against the requirements in three layers:

• Test the output: Check whether each reply satisfies the behavior contract.
• Test the evaluator: Compare judge decisions with adjudicated human labels.
• Test the change: Compare paired prompt runs and regressions.

Data flows through the evaluator as follows:

The harness follows an oracle hierarchy, where each independent oracle represents a source of truth capable of determining whether an answer is good. The most direct source of truth available settles each criterion, passing anything it can’t settle to a higher oracle:

1. Code handles deterministic rules, such as counting words or identifying forbidden phrases.
2. An LLM judge assesses subjective, possibly ambiguous behavior that deterministic rules can’t effectively evaluate.
3. When neither your deterministic rules nor your LLM judge can decide, the result is REVIEW, and a human looks at it.

Human-labeled examples aren’t part of this hierarchy. They play a different role: they check the LLM judge itself. You’ll use them to calibrate the judge in Step 6.

Each test case also includes a human-written reference that records the expected decision and the facts a good reply must include. For criteria that need it, the LLM judge uses this reference to help assess whether the target LLM output meets the desired quality standards. These are reference plus judge oracles. This tutorial uses grader and oracle interchangeably.

The harness keeps each criterion separate so you can see how the LLM performs in each dimension as you assess its overall output quality. After the harness scores all dimensions, the release policy makes the final decision based on which blocker criteria the change caused to fail.

Prerequisites

To follow along, you’ll need:

• Python 3.11 or higher
• An OpenAI API key if you’re using live mode
• Some Python coding experience

This tutorial assumes you’re comfortable with the following concepts:

• Working With JSON Data in Python
• Python Type Checking (Guide)
• Pydantic: Simplifying Data Validation in Python
• Prompt Engineering: A Practical Example
• uv to install dependencies and run scripts

You’ll learn the evaluation-specific design as you build it, so you don’t need prior experience with LLM evaluation frameworks.