Model library Support Vector Machines (SVM)
Statistical model reference

Support Vector Machines (SVM)

Review when to use this method, its data requirements, implementation patterns, and interpretation guidance.

Description

A supervised learning algorithm that finds the optimal hyperplane to separate classes in feature space, potentially after mapping data to higher dimensions using kernel functions. SVMs maximize the margin between different classes while handling nonlinear relationships through kernel tricks, making them effective for classification (SVC), regression (SVR), outlier detection, and applications with clear margin of separation in high-dimensional spaces. They are particularly robust against overfitting in high-dimensional spaces and effective when the number of dimensions exceeds the number of samples.

Use Cases
  • classification
  • regression
  • outlier detection
  • text classification
  • image recognition
  • bioinformatics
  • handwriting recognition
Requirements
  • Sample Size: small, medium
  • Missing Data: none
  • Data Distribution: any
  • Relationship Type: linear, non-linear, high-dimensional
Variable Types
Dependent Variables
  • continuous
  • categorical
  • binary
Independent Variables
  • continuous
Implementation
from sklearn.svm import SVC, SVR
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Classification with RBF kernel
svc = SVC(kernel='rbf', C=1.0, gamma='scale', probability=True)

# Hyperparameter tuning
param_grid = {
    'C': [0.1, 1, 10, 100],
    'gamma': [1, 0.1, 0.01, 0.001],
    'kernel': ['rbf', 'linear']
}
grid = GridSearchCV(SVC(), param_grid, refit=True, cv=5)
grid.fit(X_train_scaled, y_train)

# Regression
svr = SVR(kernel='rbf', C=100, gamma=0.1, epsilon=0.1)
svr.fit(X_train_scaled, y_train)

# Plot decision boundaries (for 2D data)
plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train)
ax = plt.gca()
xlim = ax.get_xlim()
ylim = ax.get_ylim()
xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 50),
             np.linspace(ylim[0], ylim[1], 50))
Z = grid.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
plt.contourf(xx, yy, Z, alpha=0.2)
plt.show()
Documentation 
library(e1071)
library(caret)

# Scale features
preProc <- preProcess(train_data, method=c("center", "scale"))
train_scaled <- predict(preProc, train_data)
test_scaled <- predict(preProc, test_data)

# Classification with tuned parameters
svm_model <- svm(
  y ~ ., 
  data = train_scaled,
  kernel = "radial",
  cost = 10,
  gamma = 0.1,
  probability = TRUE
)

# Regression
svm_reg <- svm(
  y ~ .,
  data = train_scaled,
  kernel = "radial",
  cost = 10,
  gamma = 0.1,
  epsilon = 0.1
)

# Tune parameters
tune_out <- tune.svm(
  y ~ .,
  data = train_scaled,
  kernel = "radial",
  cost = 10^(-1:2),
  gamma = c(0.1, 1, 10)
)

# Plot model (requires plotmo package)
library(plotmo)
plotmo(svm_model)
Documentation 
SVM
  /TARGET y
  /INPUT x1 x2 x3
  /KERNEL FUNCTION=RBF
  /C 1
  /GAMMA SCALE
  /PRINT MODELSUMMARY CLASSIFICATION
  /SAVE PREDVAL(pred_svm) PROBVAL(prob_svm).

* For regression:
SVM
  /TARGET y
  /INPUT x1 x2 x3
  /KERNEL FUNCTION=RBF
  /TYPE EPSILON_SVR
  /C 1
  /EPSILON 0.1
  /PRINT MODELSUMMARY.
Documentation 
PROC SVM DATA=train_data
  KERNEL=RBF C=1 GAMMA=0.5;
  INPUT x1 x2 x3 / LEVEL=INTERVAL;
  TARGET y / LEVEL=BINARY;  /* or LEVEL=INTERVAL for regression */
  SCORE DATA=test_data OUT=scored;
  SAVE MODEL=svm_model;
RUN;

/* For regression */
PROC SVM DATA=train_data
  KERNEL=RBF C=1 GAMMA=0.5 EPSILON=0.1
  TYPE=REGRESSION;
  INPUT x1 x2 x3 / LEVEL=INTERVAL;
  TARGET y / LEVEL=INTERVAL;
RUN;
Documentation 
* Classification
svmachines y x1 x2 x3, type(class) kernel(rbf) cost(1) gamma(0.1)
predict yhat

* Regression
svmachines y x1 x2 x3, type(reg) kernel(rbf) cost(1) gamma(0.1) epsilon(0.1)

* Plot decision boundary (if 2D)
svmachines plot
Documentation
Synthetic Data Example

Comprehensive synthetic datasets demonstrating SVM's ability to handle both linear and non-linear decision boundaries with different kernel functions

R Code for Data Generation and Analysis
# Generate synthetic data with different separation patterns
set.seed(123)
library(e1071)
library(ggplot2)

# Linear separation example
n <- 200
x1 <- runif(n, -1, 1)
x2 <- runif(n, -1, 1)
y_linear <- factor(ifelse(x1 + x2 > 0, "Class1", "Class2"))
df_linear <- data.frame(x1, x2, y=y_linear)

# Non-linear separation (circle)
angle <- runif(n, 0, 2*pi)
r <- runif(n, 0.5, 1)
x1 <- r*cos(angle)
x2 <- r*sin(angle)
y_circle <- factor(ifelse(r > 0.75, "Class1", "Class2"))
df_circle <- data.frame(x1, x2, y=y_circle)

# XOR pattern
grid <- expand.grid(seq(-1, 1, length=15), seq(-1, 1, length=15))
x1 <- grid[,1]
x2 <- grid[,2]
y_xor <- factor(ifelse(x1*x2 > 0, "Class1", "Class2"))
df_xor <- data.frame(x1, x2, y=y_xor)

# Visualize datasets
ggplot(df_linear, aes(x1, x2, color=y)) + geom_point() + ggtitle("Linear Separation")
ggplot(df_circle, aes(x1, x2, color=y)) + geom_point() + ggtitle("Non-linear (Circle)")
ggplot(df_xor, aes(x1, x2, color=y)) + geom_point() + ggtitle("XOR Pattern")

# Train SVM models with different kernels
svm_linear <- svm(y ~ ., data=df_linear, kernel="linear", cost=10)
svm_rbf <- svm(y ~ ., data=df_circle, kernel="radial", gamma=1, cost=10)
svm_poly <- svm(y ~ ., data=df_xor, kernel="polynomial", degree=2, coef0=1, cost=10)

# Create grid for decision boundary visualization
make_grid <- function(df) {
  rng <- apply(df[,1:2], 2, range)
  x1_seq <- seq(rng[1,1], rng[2,1], length=100)
  x2_seq <- seq(rng[1,2], rng[2,2], length=100)
  expand.grid(x1=x1_seq, x2=x2_seq)
}

grid_linear <- make_grid(df_linear)
grid_linear$pred <- predict(svm_linear, grid_linear)

grid_rbf <- make_grid(df_circle)
grid_rbf$pred <- predict(svm_rbf, grid_rbf)

grid_poly <- make_grid(df_xor)
grid_poly$pred <- predict(svm_poly, grid_poly)

# Plot decision boundaries
ggplot(grid_linear, aes(x1, x2, fill=pred)) + 
  geom_tile(alpha=0.2) +
  geom_point(data=df_linear, aes(color=y)) +
  ggtitle("Linear Kernel Decision Boundary")

ggplot(grid_rbf, aes(x1, x2, fill=pred)) + 
  geom_tile(alpha=0.2) +
  geom_point(data=df_circle, aes(color=y)) +
  ggtitle("RBF Kernel Decision Boundary")

ggplot(grid_poly, aes(x1, x2, fill=pred)) + 
  geom_tile(alpha=0.2) +
  geom_point(data=df_xor, aes(color=y)) +
  ggtitle("Polynomial Kernel Decision Boundary")

# Performance metrics
cat("Linear Kernel Accuracy:", mean(predict(svm_linear, df_linear) == df_linear$y), "\n")
cat("RBF Kernel Accuracy:", mean(predict(svm_rbf, df_circle) == df_circle$y), "\n")
cat("Polynomial Kernel Accuracy:", mean(predict(svm_poly, df_xor) == df_xor$y), "\n")
Copy this code into your R environment to generate synthetic data and perform analysis with this model.
Expected Analysis Results
Console Output
Linear Kernel Accuracy: 1 
RBF Kernel Accuracy: 0.955 
Polynomial Kernel Accuracy: 1 

Support Vectors Count:
Linear Model: 4 support vectors
RBF Model: 32 support vectors
Polynomial Model: 18 support vectors
Visualizations
Plot 1
Plot 2
Plot 3
These results are from running the R code on synthetic data. Your actual results may vary depending on your data.
Interpretation Guide

Need help interpreting the results of your Support Vector Machines (SVM) analysis? Our comprehensive interpretation guide explains:

  • How to read and understand model outputs
  • Interpreting coefficients and effect sizes correctly
  • Understanding diagnostic plots and visualizations
  • Common pitfalls and how to avoid them
  • Making valid conclusions from your analysis
View Interpretation Guide Download Guide
Additional Resources
Tutorials and Guides
Quick-R Statistical Methods
Scikit-learn Tutorials
UCLA Statistical Methods
Academic Papers and References
Find Research Papers
Latest Statistical Methods
Back to Home View All Models

Statistical assistant