evaluation

Evaluation is the process of systematically measuring how well an AI system satisfies its intended objectives using well-defined datasets, metrics, and procedures.

In practice, evaluation typically begins offline with held-out or cross-validation splits and task-specific metrics such as accuracy, precision, recall, and F1. Then, it extends to production with online methods like A/B tests to assess real user impact, business outcomes, and safety.

For generative and language models, evaluation typically combines standardized task benchmarks, such as MMLU-Pro, GPQA Diamond, SWE-bench Verified, and LiveCodeBench, with task-specific automatic metrics like BLEU, ROUGE, and perplexity for translation, summarization, and language modeling. It also draws on human review and on LLM-as-a-judge grading, where a separate model scores each output, to assess properties like factuality, coherence, helpfulness, and toxicity.

Related Resources

Course

Splitting Datasets With scikit-learn and train_test_split()

In this course, you'll learn why it's important to split your dataset in supervised machine learning and how to do that with train_test_split() from scikit-learn.

intermediate data-science machine-learning numpy