Statistical model reference
Naive Bayes classifier
Review when to use this method, its data requirements, implementation patterns, and interpretation guidance.
Description
A probabilistic classifier based on Bayes' theorem with an assumption of conditional independence between features. It calculates the probability of each class given the feature values and selects the most likely class, making it computationally efficient for high-dimensional data. Naive Bayes is particularly effective for text classification (spam detection, sentiment analysis), medical diagnosis, and recommendation systems when features can be reasonably assumed to be independent, despite often violating this assumption in practice while still performing well.
Use Cases
- text classification
- spam filtering
- sentiment analysis
- medical diagnosis
- recommendation systems
- real-time prediction
Requirements
- Sample Size: small, medium, large
- Missing Data: none, random
- Data Distribution: multinomial, bernoulli, gaussian, poisson
- Relationship Type: probabilistic, feature-independent
Variable Types
Dependent Variables
- categorical
- binary
Independent Variables
- continuous
- categorical
- binary
- count
Implementation
from sklearn.naive_bayes import (
GaussianNB,
MultinomialNB,
BernoulliNB,
ComplementNB,
CategoricalNB
)
from sklearn.feature_extraction.text import CountVectorizer
# For continuous features (assumes Gaussian distribution)
gnb = GaussianNB()
gnb.fit(X_train_cont, y_train)
# For text/count data
vectorizer = CountVectorizer()
X_train_text = vectorizer.fit_transform(text_data)
mnb = MultinomialNB()
mnb.fit(X_train_text, y_train)
# For binary features
bnb = BernoulliNB()
bnb.fit(X_train_binary, y_train)
# Get predicted probabilities
probs = gnb.predict_proba(X_test)
# Get most likely class
preds = gnb.predict(X_test)library(e1071)
library(tm) # For text mining
# Gaussian NB for continuous features
gnb <- naiveBayes(y ~ ., data = train_cont)
# Multinomial NB for count data
corpus <- Corpus(VectorSource(text_data))
dtm <- DocumentTermMatrix(corpus)
mnb <- naiveBayes(as.matrix(dtm), y_train)
# Bernoulli NB for binary features
bnb <- naiveBayes(y ~ ., data = train_binary, laplace = 1)
# Predictions
preds <- predict(gnb, test_cont, type = "class")
probs <- predict(gnb, test_cont, type = "raw")NAIVEBAYES
/TARGET target_var
/INPUT var1 var2 var3
/MODEL TYPE=MULTINOMIAL /* or GAUSSIAN, BERNOULLI */
/LAPLACE 1
/PRINT CLASSIFICATION
/SAVE PREDPROB(pred_prob).
* For text analysis:
TEXT ANALYSIS
/FEATURES TOKENS=TERMS
/MODEL TYPE=NAIVEBAYES
/TARGET target_var./* Gaussian Naive Bayes */
PROC NAIVEBAYES DATA=train
TESTDATA=test
DIST=GAUSSIAN;
CLASS target_var;
INPUT var1-var8 / LEVEL=INTERVAL;
OUTPUT OUT=scored PREDICTED=pred PROB=prob;
RUN;
/* Multinomial Naive Bayes for text */
PROC TEXTMINE DATA=text_data
OUTFEATURES=features;
DOCID doc_id;
TARGET target_var / LEVEL=NOMINAL;
TEXT text_var;
NAIVEBAYES OUTMODEL=model;
RUN;* Gaussian Naive Bayes
nbayes y x1 x2 x3, type(gaussian)
predict yhat
* Bernoulli Naive Bayes
nbayes y x1-x5, type(bernoulli) laplace(1)
* View class probabilities
nbayes predict, probabilitySynthetic Data Example
Comprehensive synthetic datasets demonstrating different Naive Bayes variants with appropriate feature distributions
R Code for Data Generation and Analysis
# Generate synthetic data for different Naive Bayes variants
set.seed(123)
library(tm)
library(e1071)
# 1. Gaussian Naive Bayes (continuous features)
n <- 500
mu_A <- c(0, 0)
mu_B <- c(3, 3)
sigma <- matrix(c(1, 0.5, 0.5, 1), nrow=2)
group_A <- MASS::mvrnorm(n/2, mu_A, sigma)
group_B <- MASS::mvrnorm(n/2, mu_B, sigma)
df_gaussian <- data.frame(
x1 = c(group_A[,1], group_B[,1]),
x2 = c(group_A[,2], group_B[,2]),
y = factor(rep(c("A", "B"), each=n/2))
)
# Visualize
ggplot(df_gaussian, aes(x1, x2, color=y)) +
geom_point(alpha=0.6) +
stat_ellipse() +
ggtitle("Gaussian Naive Bayes Data")
# 2. Multinomial Naive Bayes (text/count data)
docs <- c(
rep("apple banana fruit", 100),
rep("car vehicle engine", 100),
rep("apple car hybrid", 50)
)
labels <- factor(c(
rep("fruit", 100),
rep("vehicle", 100),
rep("mixed", 50)
))
corpus <- Corpus(VectorSource(docs))
dtm <- DocumentTermMatrix(corpus, control=list(
weighting = weightTf,
removePunctuation = TRUE,
stopwords = FALSE
))
# 3. Bernoulli Naive Bayes (binary features)
df_bernoulli <- data.frame(
x1 = rbinom(n, 1, 0.3),
x2 = rbinom(n, 1, 0.7),
x3 = rbinom(n, 1, 0.5),
y = factor(ifelse(
(x1 + x2 + x3) > 1,
"Class1",
"Class2"
))
)
# Train-test split
train_idx <- sample(1:n, 0.7*n)
# Gaussian NB
train_gauss <- df_gaussian[train_idx, ]
test_gauss <- df_gaussian[-train_idx, ]
gnb <- naiveBayes(y ~ ., data=train_gauss)
gnb_pred <- predict(gnb, test_gauss)
gnb_prob <- predict(gnb, test_gauss, type="raw")
# Multinomial NB
train_dtm <- dtm[train_idx, ]
test_dtm <- dtm[-train_idx, ]
train_labels <- labels[train_idx]
mnb <- naiveBayes(as.matrix(train_dtm), train_labels)
mnb_pred <- predict(mnb, as.matrix(test_dtm))
# Bernoulli NB
train_bern <- df_bernoulli[train_idx, ]
test_bern <- df_bernoulli[-train_idx, ]
bnb <- naiveBayes(y ~ ., data=train_bern, laplace=1)
bnb_pred <- predict(bnb, test_bern)
# Evaluate performance
cat("Gaussian NB Accuracy:", mean(gnb_pred == test_gauss$y), "\n")
cat("Multinomial NB Accuracy:", mean(mnb_pred == labels[-train_idx]), "\n")
cat("Bernoulli NB Accuracy:", mean(bnb_pred == test_bern$y), "\n")
# Show predicted probabilities for Gaussian NB
head(gnb_prob)
# Feature log probabilities for Multinomial NB
log_probs <- log(mnb$apriori) + apply(log(mnb$tables), 2, sum)
sort(log_probs, decreasing=TRUE)
Copy this code into your R environment to generate synthetic data and perform analysis with this model.
Expected Analysis Results
Console Output
Gaussian NB Accuracy: 0.973
Multinomial NB Accuracy: 0.933
Bernoulli NB Accuracy: 0.853
Top 5 most discriminative terms (Multinomial NB):
apple car banana vehicle fruit
12.45 11.32 9.87 8.54 7.21
Sample predicted probabilities (Gaussian NB):
A B
1 0.99999999 1.0479e-08
2 0.99999857 1.4258e-06
3 0.00000000 1.0000e+00
4 0.00000123 9.9999e-01Visualizations
These results are from running the R code on synthetic data. Your actual results may vary depending on your data.
Interpretation Guide
Need help interpreting the results of your Naive Bayes classifier analysis? Our comprehensive interpretation guide explains:
- How to read and understand model outputs
- Interpreting coefficients and effect sizes correctly
- Understanding diagnostic plots and visualizations
- Common pitfalls and how to avoid them
- Making valid conclusions from your analysis