logistic regression

Logistic regression is a machine learning model that estimates the probability of a categorical outcome, such as whether an email is spam, from a weighted sum of input features. Despite the name, it performs classification rather than regression. The weighted sum passes through the logistic (sigmoid) curve, an S-shaped activation function that maps any real number onto a value between 0 and 1.

Equivalently, the model treats the log-odds of the outcome as a linear function of the features. Log-odds is the logarithm of the odds, which is the probability that the outcome happens divided by the probability that it doesn’t. Each coefficient then says how much the odds multiply for a one-unit change in its feature. That interpretability is much of the appeal in fields such as medicine and finance.

In the sweep below, dragging the weight and the bias reshapes the curve, slides the 0.5 decision boundary, and updates the odds multiplier for each extra hour of study.

Fitting the coefficients maximizes the likelihood of the observed labels. No closed-form solution exists, so training minimizes cross-entropy loss with gradient descent or a quasi-Newton solver. The multinomial variant swaps the sigmoid for a softmax over per-class scores to handle more than two classes.

Both forms sit inside modern deep learning. A single logistic unit is the smallest neural network, one neuron with no hidden layer, and the softmax output layer of a classifier is multinomial logistic regression stacked on learned features. The head of an LLM scores the whole vocabulary that way before the next token is sampled.

In Python, scikit-learn’s LogisticRegression is the standard implementation, and it applies L2 regularization by default. Its speed and interpretable coefficients make it a common baseline to beat before trying a larger model.

Related Resources

Tutorial

Logistic Regression in Python

In this step-by-step tutorial, you'll get started with logistic regression in Python. Classification is one of the most important areas of machine learning, and logistic regression is one of its basic methods. You'll learn how to create, evaluate, and apply a model to make predictions.

intermediate algorithms data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Practical Text Classification With Python and Keras (Tutorial)
• Split Your Dataset With scikit-learn's train_test_split() (Tutorial)
• Stochastic Gradient Descent Algorithm With Python and NumPy (Tutorial)
• Learn Text Classification With Python and Keras (Course)
• Splitting Datasets With scikit-learn and train_test_split() (Course)
• Split Your Dataset With scikit-learn's train_test_split() (Quiz)