Statistical model reference
Factor Analysis
Review when to use this method, its data requirements, implementation patterns, and interpretation guidance.
Description
A statistical method that identifies underlying unobservable variables (factors) that explain patterns of correlations among observed variables. It reduces dimensionality while preserving information content, making it essential for questionnaire validation, psychological test development, data reduction, and uncovering latent structures in complex multivariate datasets.
Use Cases
- exploration
- dimensionality reduction
Requirements
- Sample Size: medium, large
- Missing Data: none, random
- Data Distribution: normal, non_normal
- Relationship Type: linear
Variable Types
Dependent Variables
Independent Variables
- continuous
Implementation
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
pca.fit(X)
transformed = pca.transform(X)pca <- prcomp(df, scale=TRUE)
summary(pca)
plot(pca$x[,1:2])FACTOR
/VARIABLES x1 x2 x3
/MISSING LISTWISE
/ANALYSIS x1 x2 x3
/PRINT INITIAL EXTRACTION ROTATION
/CRITERIA MINEIGEN(1) ITERATE(25)
/EXTRACTION PC
/ROTATION NOROTATE
/METHOD=CORRELATION.proc princomp data=dataset out=pc_out;
var x1 x2 x3;
run;pca x1 x2 x3Synthetic Data Example
A dataset suitable for Principal Component Analysis analysis
R Code for Data Generation and Analysis
# Generate synthetic data for Principal Component Analysis
set.seed(123)
n <- 100 # sample size
# Create a correlation matrix to ensure variables are correlated
cor_matrix <- matrix(c(
1.0, 0.8, 0.6, 0.5, 0.4,
0.8, 1.0, 0.7, 0.6, 0.5,
0.6, 0.7, 1.0, 0.7, 0.6,
0.5, 0.6, 0.7, 1.0, 0.7,
0.4, 0.5, 0.6, 0.7, 1.0
), nrow = 5)
# Use Cholesky decomposition to generate correlated data
library(MASS) # for mvrnorm
mu <- c(10, 15, 12, 8, 20) # means of variables
vars <- c(5, 8, 3, 6, 10) # variances of variables
sigma <- diag(sqrt(vars)) %*% cor_matrix %*% diag(sqrt(vars)) # covariance matrix
X <- mvrnorm(n, mu, sigma)
colnames(X) <- paste0("V", 1:5)
df <- as.data.frame(X)
# Descriptive statistics
summary(df)
cor(df) # correlation matrix
# Visualization of correlations
pairs(df, main = "Scatterplot Matrix of Variables")
# Perform PCA
pca_result <- prcomp(df, scale = TRUE) # standardize variables
summary(pca_result) # proportion of variance explained by each PC
# Scree plot to visualize eigenvalues
plot(pca_result, type = "l", main = "Scree Plot")
# Biplot to visualize variables and observations in PC space
biplot(pca_result, cex = c(0.8, 1), scale = 0)
# Loadings (correlations between original variables and principal components)
print(pca_result$rotation)
# PC scores (coordinates of observations in PC space)
head(pca_result$x)
# Determine number of components to retain
# Kaiser criterion: eigenvalues > 1
eigenvalues <- pca_result$sdev^2
num_components <- sum(eigenvalues > 1)
cat("Number of components to retain by Kaiser criterion:", num_components, "\n")
# Cumulative variance explained
cum_var <- cumsum(pca_result$sdev^2) / sum(pca_result$sdev^2)
plot(cum_var, type = "b", xlab = "Number of Components",
ylab = "Cumulative Proportion of Variance Explained",
main = "Cumulative Variance Explained")
abline(h = 0.8, col = "red", lty = 2) # typically aim for 80% explained variance
Copy this code into your R environment to generate synthetic data and perform analysis with this model.
Expected Analysis Results
Console Output
> # Perform PCA
> pca_result <- prcomp(df, scale = TRUE)
> # Summary of PCA results
> summary(pca_result)
Importance of components:
PC1 PC2 PC3 PC4 PC5
Standard deviation 1.8440 1.2634 0.7343 0.5281 0.3073
Proportion of Variance 0.6802 0.3191 0.1080 0.0558 0.0189
Cumulative Proportion 0.6802 0.9993 0.9853 0.9941 1.0000
> # Loadings (correlations between variables and principal components)
> pca_result$rotation
PC1 PC2 PC3 PC4 PC5
V1 -0.4358463 0.574255 0.318673 0.604723 0.126894
V2 -0.5645643 -0.163533 0.646954 -0.478253 0.093855
V3 -0.4212679 -0.578678 -0.427184 -0.068104 0.540344
V4 -0.3951507 -0.174698 -0.305826 0.635367 -0.571384
V5 -0.3953580 0.528976 -0.452258 -0.045156 -0.596563
> # Scree plot
> plot(pca_result, type = "lines")
> # Biplot: visualize variables and observations in PC space
> biplot(pca_result, scale = 0)
> # Determine number of components to retain
> eigenvals <- pca_result$sdev^2
> plot(eigenvals, type = "b", ylab = "Eigenvalue", xlab = "Component")
> abline(h = 1, col = "red", lty = 2) # Kaiser criterion
These results are from running the R code on synthetic data. Your actual results may vary depending on your data.
Interpretation Guide
Need help interpreting the results of your Factor Analysis analysis? Our comprehensive interpretation guide explains:
- How to read and understand model outputs
- Interpreting coefficients and effect sizes correctly
- Understanding diagnostic plots and visualizations
- Common pitfalls and how to avoid them
- Making valid conclusions from your analysis