evaluation

Evaluation is the process of systematically measuring how well an AI system satisfies its intended objectives using well-defined datasets, metrics, and procedures.

In practice, evaluation typically begins offline with held-out or cross-validation splits and task-specific metrics such as accuracy, precision, recall, and F1. Then, it extends to production with online methods like A/B tests to assess real user impact, business outcomes, and safety.

For generative and language models, evaluation typically combines standardized task benchmarks, such as MMLU-Pro, GPQA Diamond, SWE-bench Verified, and LiveCodeBench. It pairs those with task-specific automatic metrics like BLEU, ROUGE, and perplexity for translation, summarization, and language modeling.

Evaluation also draws on human review and on LLM-as-a-judge grading, where a separate model scores each output, to assess properties like factuality, coherence, helpfulness, and toxicity.

For agentic systems, evaluation also scores the trajectory, meaning the sequence of plans, tool calls, and intermediate steps the agent took. Trajectory-level checks catch an agent that reaches the right result through faulty reasoning or a needlessly expensive path, which output-only scoring misses.

Related Resources

Course

Splitting Datasets With scikit-learn and train_test_split()

In this course, you'll learn why it's important to split your dataset in supervised machine learning and how to do that with train_test_split() from scikit-learn.

intermediate data-science machine-learning numpy

For additional information on related topics, take a look at the following resources:

• Split Your Dataset With scikit-learn's train_test_split() (Tutorial)
• Split Your Dataset With scikit-learn's train_test_split() (Quiz)