naive Bayes

Naive Bayes is a family of probabilistic classifiers that apply Bayes’ theorem while assuming that every feature is independent of the others once the class label is known. Given an email to sort, for example, the model weighs how much each word shifts the odds and picks the likelier class, spam or not spam.

That independence assumption is the naive part, and it rarely holds in real data, yet the classifiers it produces stay accurate enough to remain a standard baseline in machine learning.

Training estimates the prior probability of each class along with the probability of each feature value within each class. Treating the features as independent lets every one of those conditional distributions be estimated separately as a one-dimensional problem, which keeps fitting fast and blunts the curse of dimensionality.

Prediction multiplies a class prior by the matching per-feature probabilities and returns the highest-scoring class. Smoothing, usually adding a small constant to every count, keeps an unseen feature-and-class pair from zeroing out that product.

Scoring a short message against a small labeled corpus makes that arithmetic concrete, with one factor per word and a switch for the smoothing term:

Variants differ mainly in the distribution assumed for the features:

• Multinomial naive Bayes models word counts or tf-idf weights, which makes it the usual choice for document classification.
• Bernoulli naive Bayes models the presence or absence of each term as a binary feature, which can work better on shorter documents.
• Gaussian naive Bayes assumes continuous features follow a normal distribution.
• Complement naive Bayes adapts the multinomial variant for imbalanced data, a frequent problem in text corpora.

Naive Bayes was the workhorse of spam filtering and other natural language processing tasks for decades, and it is now mostly a cheap baseline that heavier neural network models are expected to beat. Its main caveat is calibration, because the independence assumption skews the predicted probabilities even when the predicted class is right.

Related Resources

Tutorial

Sentiment Analysis: First Steps With Python's NLTK Library

In this tutorial, you'll learn how to work with Python's Natural Language Toolkit (NLTK) to process and analyze text. You'll also learn how to perform sentiment analysis with built-in as well as custom classifiers!

intermediate algorithms data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Practical Text Classification With Python and Keras (Tutorial)
• Natural Language Processing With Python's NLTK Package (Tutorial)
• Use Sentiment Analysis With Python to Classify Movie Reviews (Tutorial)
• Split Your Dataset With scikit-learn's train_test_split() (Tutorial)
• Learn Text Classification With Python and Keras (Course)
• Natural Language Processing With Python's NLTK Package (Quiz)
• Splitting Datasets With scikit-learn and train_test_split() (Course)
• Split Your Dataset With scikit-learn's train_test_split() (Quiz)