hypothesis testing

Hypothesis testing is a formal statistical method for deciding whether sample data provides enough evidence to reject a default assumption about a larger population. It frames a question as two competing claims and measures how well the observed data fits the default one.

That default claim is the null hypothesis, usually a statement of no effect or no difference, such as a redesign making no difference to sign-ups. The alternative hypothesis is the claim a researcher suspects instead.

The test reduces the sample to a single test statistic and asks how surprising that value would be if the null hypothesis were true. That surprise is captured by a p-value, the probability of a result at least as extreme as the observed one, assuming the null hypothesis holds.

A p-value below a chosen threshold, the significance level (often 0.05), leads to rejecting the null hypothesis. Because the decision rests on probability, it can go wrong in two ways:

• A Type I error rejects a true null hypothesis, a false positive.
• A Type II error fails to reject a false null hypothesis, a false negative.

The interactive figure below shows how the two errors trade off as the decision threshold moves:

Named procedures such as the t-test, chi-squared test, and analysis of variance (ANOVA) apply this framework to different kinds of data. In software, hypothesis testing underlies A/B testing, performance benchmarking, and the statistical evaluation of machine learning models.

Related Resources

Tutorial

Python Statistics Fundamentals: How to Describe Your Data

Calculate descriptive Python statistics and visualize your data using NumPy, SciPy, pandas, Matplotlib, and the built-in statistics library.

intermediate data-science numpy

For additional information on related topics, take a look at the following resources:

• NumPy, SciPy, and pandas: Correlation With Python (Tutorial)
• Linear Regression in Python (Tutorial)
• Starting With Linear Regression in Python (Course)
• NumPy Tutorial: Your First Steps Into Data Science in Python (Tutorial)
• Python Statistics Fundamentals: How to Describe Your Data (Quiz)
• Linear Regression in Python (Quiz)
• NumPy Tutorial: Your First Steps Into Data Science in Python (Quiz)