linear regression

Linear regression is a machine learning model that predicts a continuous target value as a weighted sum of input features plus a constant intercept. With a single input feature, that sum is a straight line drawn through the data points. Fitting it means choosing the weights that minimize the sum of the squared differences between the predicted and the observed values. That fitting rule is called ordinary least squares (OLS).

Moving the slope and the intercept below redraws the line and resizes the square attached to each observation, where the side of a square is that point’s prediction error. The least-squares solution is the one setting where the total shaded area is as small as it can get.

Unlike most models trained by gradient descent, OLS has a closed-form solution. In scikit-learn, LinearRegression reaches it through a matrix decomposition borrowed from SciPy, at a cost that grows linearly in the number of samples and quadratically in the number of features. Sparse input routes to an iterative least-squares solver instead.

Each fitted coefficient reads as the expected change in the target for a one-unit change in its feature while the rest hold steady. That interpretability keeps the model in use where an explanation matters as much as accuracy. The reading breaks down when features are strongly correlated, because the least-squares estimate then swings on small changes in the data. Regularized variants add a penalty term to steady it:

• Ridge regression adds an L2 penalty that shrinks coefficients toward zero without reaching it.
• Lasso adds an L1 penalty that drives some coefficients exactly to zero, which doubles as feature selection.
• Elastic net mixes both penalties and tends to keep groups of correlated features together.

That same form turns up inside neural networks. A single unit with no activation function is a linear regression, and stacking such units under nonlinear activations is what lets deeper models fit curves that a straight line cannot.

The name comes from Francis Galton, who coined it in an 1886 study of human height, where the statures of children regressed toward the population average. Least squares itself dates to Legendre and Gauss in the early 1800s. In Python, statsmodels complements scikit-learn by reporting standard errors, p-values, and confidence intervals next to the coefficients.

Related Resources

Tutorial

Linear Regression in Python

Use Python to build a linear model for regression, fit data with scikit-learn, read R2, and make predictions in minutes.

intermediate data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Starting With Linear Regression in Python (Course)
• Stochastic Gradient Descent Algorithm With Python and NumPy (Tutorial)
• Logistic Regression in Python (Tutorial)
• Split Your Dataset With scikit-learn's train_test_split() (Tutorial)
• Splitting Datasets With scikit-learn and train_test_split() (Course)
• Linear Regression in Python (Quiz)
• Split Your Dataset With scikit-learn's train_test_split() (Quiz)