Naive Bayes Classifier

The naive Bayes classifier is a simple probabilistic classifier based on Bayes’ theorem. It is especially useful when we want a fast baseline for classification, when the feature dimension is large, or when the data are naturally modeled with simple class-conditional distributions.

Suppose the class label is $Y\in \{1,\dots ,K\}$ and the feature vector is $X=(X_1,\dots ,X_p)$. The classifier models the posterior probability of a class by

$$\mathbb{P}(Y=k\mid X=x)=\frac{\mathbb{P}(Y=k)f(x\mid Y=k)}{{\sum }_{c=1}^K\mathbb{P}(Y=c)f(x\mid Y=c)}.$$

The key simplifying assumption is that the features are conditionally independent given the class. Under that assumption,

$$f(x\mid Y=k)=\prod _{j=1}^{p}f(x_j\mid Y=k).$$

Thus,

$$\mathbb{P}(Y=k\mid X=x)\propto \mathbb{P}(Y=k)\prod _{j=1}^{p}f(x_j\mid Y=k).$$

Definition

The naive Bayes classifier predicts the class with largest posterior probability:

$$\hat{y}(x)=\arg \underset{k\in \{1,\dots ,K\}}{\max }\mathbb{P}(Y=k\mid X=x).$$

Interpretation

Naive Bayes is a generative classifier. It models how features are generated within each class, then uses Bayes’ theorem to turn those class-conditional models into posterior class probabilities.

Conditional Independence

The adjective “naive” refers to the assumption that once we know the class label, the features no longer provide additional information about one another.

Formally, the model assumes that for each class $k$,

$$X_1,\dots ,X_p\text{ are conditionally independent given }Y=k.$$

This means that if we know the class, the joint density or mass function factors into a product of one-dimensional terms:

$$f(x\mid Y=k)=\prod _{j=1}^{p}f(x_j\mid Y=k).$$

Intuition

Without the conditional independence assumption, modeling $f(x\mid Y=k)$ directly can be very difficult in high dimensions. Naive Bayes replaces one hard high-dimensional modeling problem with many easier one-dimensional modeling problems.

This assumption is usually false in real data, but the classifier can still work surprisingly well in practice.

Warning

Conditional independence is much stronger than ordinary independence. The model does not assume the features are marginally independent; it assumes they are independent after conditioning on the class.

Properties

Important properties:

1. Naive Bayes is usually very fast to train and evaluate.
2. It works naturally for multiclass classification.
3. It often performs well in high-dimensional problems such as text classification.
4. It can output posterior class probabilities, although these probabilities are often not well calibrated.
5. The decision rule depends on a product of class-conditional terms, so in practice it is often computed on the log scale for numerical stability.

Using logarithms, the prediction rule becomes

$$\hat{y}(x)=\arg \underset{k}{\max }\left[\log \mathbb{P}(Y=k)+\sum _{j=1}^p\log f(x_j\mid Y=k)\right].$$

Intuition

The model gives each class a score made of two parts: a prior preference for the class and evidence from each feature. Each feature contributes independently to the total score under the model assumption.

Warning

Since the model multiplies many probabilities together, it can assign extreme posterior probabilities even when the conditional independence assumption is not very accurate.

Worked Example

Example

Suppose we have binary classification with classes $Y\in \{0,1\}$ and two binary features $X_1,X_2$. Assume the estimated class priors are

$$\mathbb{P}(Y=1)=0.4,\mathbb{P}(Y=0)=\mathrm{0.6.}$$

Also suppose the estimated class-conditional probabilities are

$$\mathbb{P}(X_1=1\mid Y=1)=0.8,\mathbb{P}(X_2=1\mid Y=1)=0.7,$$

and

$$\mathbb{P}(X_1=1\mid Y=0)=0.3,\mathbb{P}(X_2=1\mid Y=0)=\mathrm{0.2.}$$

For a new observation with $x_1=1$ and $x_2=1$, naive Bayes assigns the class-1 score

$$\mathbb{P}(Y=1)\mathbb{P}(X_1=1\mid Y=1)\mathbb{P}(X_2=1\mid Y=1)=0.4\cdot 0.8\cdot 0.7=0.224$$

and the class-0 score

$$\mathbb{P}(Y=0)\mathbb{P}(X_1=1\mid Y=0)\mathbb{P}(X_2=1\mid Y=0)=0.6\cdot 0.3\cdot 0.2=\mathrm{0.036.}$$

Since $0.224>0.036$, the classifier predicts class 1.

This example shows the basic pattern: each class gets a prior term and then a multiplicative contribution from each feature. The predicted class is the one with the larger resulting score.

Common Variants

The main difference between common naive Bayes models is the form assumed for the class-conditional feature distributions.

1. Gaussian naive Bayes: for continuous features, assume

$$X_j\mid Y=k\sim N({\mu }_{jk},{\sigma }_{jk}^2).$$

2. Bernoulli naive Bayes: for binary features, model whether each feature is present or absent.
3. Multinomial naive Bayes: often used for count data such as bag-of-words text features.

The classifier formula is the same in all cases; only the choice of the feature model changes.

Estimation Methods

The parameters are usually estimated by frequency counts or by simple maximum likelihood calculations within each class.

Class Priors

The class prior probabilities are typically estimated by

$$\hat{\mathbb{P}}(Y=k)=\frac{n_k}{n}$$

where $n_k$ is the number of training examples in class $k$.

Class-Conditional Feature Models

The class-conditional parameters are estimated separately for each class.

For example, in Gaussian naive Bayes,

$${\hat{\mu }}_{jk}=\frac{1}{n_k}\sum _{i:y_i=k}{x}_{ij}$$

and

$${\hat{\sigma }}_{jk}^2=\frac{1}{n_k}\sum _{i:y_i=k}(x_{ij}-{\hat{\mu }}_{jk}{)}^2.$$

In Bernoulli or multinomial naive Bayes, the corresponding probabilities are estimated from class-specific counts.

Smoothing

For discrete naive Bayes models, it is common to use Laplace smoothing or additive smoothing to avoid zero probabilities.

Warning

Without smoothing, if a feature value never appears in the training data for a class, the estimated class-conditional probability can be zero. Since naive Bayes multiplies probabilities across features, one zero term can force the entire posterior score for that class to zero.

Interpretation

Smoothing prevents the model from becoming overconfident just because some feature/class combination was not observed in a finite sample.

Text Classification Example

For text classification, a common choice is multinomial naive Bayes.

Suppose:

1. Each document has a class label, such as spam or not spam.
2. The features are word counts from a vocabulary of size $V$.
3. $N_{jk}$ is the total number of times word $j$ appears in training documents from class $k$.

Then the class prior is estimated by simple class frequencies:

$$\hat{\mathbb{P}}(Y=k)=\frac{n_k}{n}$$

where $n_k$ is the number of training documents in class $k$.

The class-conditional word probabilities are estimated by normalized counts:

$$\hat{\mathbb{P}}(\text{word }j\mid Y=k)=\frac{N_{jk}}{{\sum }_{m=1}^V{N}_{mk}}.$$

With Laplace smoothing,

$$\hat{\mathbb{P}}(\text{word }j\mid Y=k)=\frac{N_{jk}+\alpha }{{\sum }_{m=1}^V{N}_{mk}+\alpha V}$$

where often $\alpha =1$.

Example

Suppose 40 out of 100 training emails are spam. Then

$$\hat{\mathbb{P}}(\text{spam})=\mathrm{0.4.}$$

If the word free appears 120 times in spam emails but only 10 times in non-spam emails, then the estimated probability of the word free given spam will be much larger than the estimated probability of free given non-spam. This makes free evidence in favor of the spam class.

At prediction time, the model combines the prior and the word evidence for each class. In log form, the class score is

$$\log \mathbb{P}(Y=k)+\sum _{j=1}^V{x}_j\log \mathbb{P}(\text{word }j\mid Y=k)$$

where $x_j$ is the count of word $j$ in the new document.

Model Selection

Compared with linear or logistic regression, model selection for naive Bayes is usually less about adding or removing many coefficients and more about choosing the feature representation and the class-conditional distribution.

Common choices include:

1. Choosing the variant: Gaussian, Bernoulli, or multinomial naive Bayes depending on the feature type.
2. Cross-validation: useful for comparing different feature representations, smoothing levels, or preprocessing choices.
3. Log loss: useful if you care about the quality of predicted probabilities.
4. ROC-AUC / PR-AUC: useful for binary classification when ranking quality matters.
5. Accuracy / error rate: often used as a simple baseline comparison metric.

Rule of Thumb

In naive Bayes, good performance often depends more on the feature encoding and the choice of class-conditional model than on complicated optimization or inference machinery.

Naive Bayes vs Logistic Regression

Naive Bayes and logistic regression are both standard probabilistic classifiers, but they make different modeling choices.

Naive Bayes is a generative model:

1. It models $\mathbb{P}(Y)$ and $f(X\mid Y)$.
2. It uses Bayes’ theorem to compute $\mathbb{P}(Y\mid X)$.
3. It relies on the conditional independence assumption to make the feature model tractable.

Logistic regression is a discriminative model:

1. It models $\mathbb{P}(Y\mid X)$ directly.
2. It does not try to model the full feature distribution.
3. It assumes linearity on the log-odds scale rather than conditional independence of features.

Reasons to prefer naive Bayes:

1. You want a very fast, simple baseline.
2. The feature dimension is large, and you want to avoid modeling a full joint class-conditional distribution over the features.
3. You have relatively limited data and want a model with simple class-conditional estimates.
4. The chosen feature model is a good fit for the data.

Reasons to prefer logistic regression:

1. You care about modeling the decision boundary directly.
2. The predictors are strongly correlated, so the naive conditional independence assumption is not very plausible.
3. You want coefficients with a direct log-odds interpretation.
4. You want predicted probabilities that are often better calibrated.

Rule of Thumb

Naive Bayes often works well as a fast high-dimensional baseline because the conditional independence assumption avoids the much harder problem of modeling a full joint feature distribution within each class. Logistic regression is often preferred when interpretability, calibration, and fewer independence assumptions are more important.

Warning

Neither model dominates the other in every setting. Naive Bayes can outperform logistic regression when its assumptions are useful approximations and the sample size is limited, while logistic regression often improves as more labeled data become available.

Model Assumptions

The assumptions below are the main ones that come up in interviews and in practice.

Assumption	Meaning	How to check	If violated
Correct feature model	The chosen class-conditional distribution is a reasonable fit for the feature type, such as Gaussian for continuous features or multinomial for counts.	Compare the data type and empirical behavior to the model choice.	Predictions can degrade because the class-conditional probabilities are poorly estimated.
Conditional independence	Given the class, the features are independent.	Think about whether features remain strongly related even after conditioning on the label.	Posterior probabilities can be distorted, though classification can still work well.
Representative class priors	The estimated class frequencies reflect the real deployment setting, unless priors are set manually.	Compare training class balance to the target population.	The model may systematically overpredict or underpredict some classes.
Sufficient data per class	Each class has enough observations to estimate its feature distributions reliably.	Check class counts, rare categories, and sparsity.	Parameter estimates become noisy and unstable, especially for minority classes.

Some additional practical diagnostics are worth remembering:

1. If the predicted probabilities are important, check calibration, since naive Bayes probabilities are often too extreme.
2. If many features are redundant copies of one another, the model may effectively count the same evidence multiple times.
3. For text and count data, smoothing is often essential.

Interview Summary

If the interviewer asks for the assumptions of naive Bayes, the safest compact answer is: choose an appropriate class-conditional distribution, assume the features are conditionally independent given the class, estimate reasonable class priors, and have enough data in each class to estimate the required probabilities.

Warning

A common mistake is to say that naive Bayes assumes the features are independent in general. The actual assumption is weaker and more specific: the features are independent conditional on the class label.