Logistic Regression

Introduction

A specialized regression technique for binary outcomes that uses the logistic function to model the probability of a binary event occurring. It transforms the linear relationship between predictors and outcome to constrain predicted values between 0 and 1, making it ideal for classification problems, risk assessment, and inference about categorical outcomes.

This guide will help you interpret the results of a Logistic Regression analysis. We'll walk through:

Understanding the model output
Interpreting coefficients and statistics

Reading diagnostic plots
Making predictions and drawing conclusions

Data Description

This analysis was performed on a dataset with appropriate characteristics for this model.

Note: Before interpreting any model, always examine your data using descriptive statistics and visualizations to understand its structure.

Model Output Interpretation

> summary(df)
  y           x1                 x2            x3   
 0:120   Min.   :-2.95305   Min.   :-3.4702   1:74  
 1:80    1st Qu.:-0.63543   1st Qu.:-0.6595   2:61  
         Median : 0.02386   Median : 0.1035   3:65  
         Mean   : 0.02207   Mean   : 0.0222         
         3rd Qu.: 0.70851   3rd Qu.: 0.6863         
         Max.   : 2.81898   Max.   : 3.5814         

> table(df$y)

  0   1 
120  80 

> model <- glm(y ~ x1 + x2 + x3, family = binomial(link = "logit"), data = df)
> summary(model)

Call:
glm(formula = y ~ x1 + x2 + x3, family = binomial(link = "logit"), 
    data = df)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.1701  -0.8079  -0.4635   0.9184   2.2701  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)  -0.9879     0.2811  -3.515 0.000439 ***
x1            0.7846     0.1827   4.294 1.75e-05 ***
x2           -1.2264     0.2096  -5.853 4.82e-09 ***
x32           0.1308     0.3989   0.328 0.743023    
x33           0.5486     0.3843   1.428 0.153465    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 267.33  on 199  degrees of freedom
Residual deviance: 208.46  on 195  degrees of freedom
AIC: 218.46

Number of Fisher Scoring iterations: 4

> exp(coef(model))  # exponentiated coefficients give odds ratios
(Intercept)         x1         x2        x32        x33 
  0.3724354   2.1916057   0.2933147   1.1396893   1.7309659 

> predicted_probs <- predict(model, newdata = new_data, type = "response")
> predicted_class <- ifelse(predicted_probs > 0.5, 1, 0)
> print(cbind(new_data, prob = predicted_probs, class = predicted_class))
   x1 x2 x3        prob class
1 -1  1  1 0.075095095     0
2  0  0  2 0.345347633     0
3  1 -1  3 0.824435337     1

Understanding the Output:

The summary table above provides key information about your logistic regression model:

Coefficients: The estimate column shows the effect of each predictor on the log-odds of the outcome. For interpretation, we often exponentiate these values to get odds ratios.
Standard Error: Measures the precision of the coefficient estimates.
z value: Used for hypothesis testing, similar to t-values in linear regression.
p-value: The probability of observing the data if the null hypothesis (coefficient = 0) is true. Values below 0.05 are typically considered statistically significant.
Null/Residual deviance: Measures of model fit. Lower residual deviance indicates better fit.
AIC: Akaike Information Criterion, used for model comparison. Lower values indicate better models.

The exponentiated coefficients (odds ratios) tell you how much the odds of the outcome increase multiplicatively with a one-unit increase in the predictor.

Coefficient Interpretation

In logistic regression, we interpret coefficients using odds ratios:

The odds ratio (exp(coefficient)) represents how the odds of the outcome change with a one-unit increase in the predictor.
Odds ratios > 1 indicate that the predictor is associated with higher odds of the outcome.
Odds ratios < 1 indicate that the predictor is associated with lower odds of the outcome.
For example, an odds ratio of 1.5 means the odds increase by 50% for each unit increase in the predictor.

The 95% confidence interval for the odds ratio helps assess the precision of the estimate and whether the association is statistically significant (if the interval doesn't include 1).

Variable	Coefficient	Interpretation
Intercept	-0.99 (OR: 0.37)	The log odds when all predictors are zero. The odds ratio of 0.37 means the base odds of the event are lower than 1.
x1 (continuous)	0.78 (OR: 2.19)	For each one-unit increase in x1, the odds of the outcome increase by a factor of 2.19 (119% increase).
x2 (continuous)	-1.23 (OR: 0.29)	For each one-unit increase in x2, the odds of the outcome decrease by a factor of 0.29 (71% decrease).

Diagnostic Plots

Diagnostic plots for logistic regression help evaluate model fit, check for influential observations, and assess the pattern of residuals.

Plot: ROC Curve

Figure: ROC Curve

How to interpret: The ROC curve shows the trade-off between sensitivity and specificity. The area under the curve (AUC) measures overall model performance, with values closer to 1 indicating better discrimination.

Plot: Residual Plot

Figure: Residual Plot

How to interpret: This plot shows residuals versus fitted values. Look for random scatter around zero. Patterns may indicate non-linearity or other model issues.

Important: For logistic regression, also examine the Hosmer-Lemeshow test for goodness-of-fit, classification tables to assess predictive accuracy, and check for multicollinearity using variance inflation factors (VIF).

Model Assumptions

The Logistic Regression relies on the following assumptions:

Independence: Observations are independent of each other.
Linearity in the logit: The log odds are linearly related to continuous predictors.
No multicollinearity: Predictors are not highly correlated with each other.
No influential observations: No single case has undue influence on the model.
Adequate sample size: Generally, at least 10 events per variable for stable estimates.

Pro Tip: The logistic model doesn't assume normality or homoscedasticity of errors, but it's crucial to check for linearity in the logit for continuous predictors using the Box-Tidwell approach or spline functions.

Prediction and Practical Implications

Logistic regression allows you to predict the probability of the outcome for new observations, which can be converted to class predictions using a threshold (typically 0.5).

x1 <- rnorm(n, mean = 0, sd = 1)  # continuous predictor
x2 <- rnorm(n, mean = 0, sd = 1)  # another continuous predictor
x3 <- factor(sample(1:3, n, replace = TRUE))  # categorical predictor

# Generate binary outcome based on logistic model
logit <- -1 + 0.8 * x1 - 1.2 * x2  # linear predictor
prob <- 1 / (1 + exp(-logit))  # apply logistic function to get probabilities
y <- rbinom(n, 1, prob)  # generate binary outcome

# Combine into a data frame
df <- data.frame(y = factor(y), x1 = x1, x2 = x2, x3 = x3)

# Descriptive statistics
summary(df)
table(df$y)  # frequency of outcome
table(df$y, df$x3)  # contingency table with categorical predictor

# Visualization
boxplot(x1 ~ y, data = df, main = "X1 by Outcome", xlab = "Outcome (Y)", ylab = "X1")
boxplot(x2 ~ y, data = df, main = "X2 by Outcome", xlab = "Outcome (Y)", ylab = "X2")

# Model fitting
model <- glm(y ~ x1 + x2 + x3, family = binomial(link = "logit"), data = df)
summary(model)

# Effects on odds ratios
exp(coef(model))  # exponentiated coefficients give odds ratios
exp(confint(model))  # confidence intervals for odds ratios

# Predictions
new_data <- data.frame(x1 = c(-1, 0, 1), x2 = c(1, 0, -1), x3 = factor(c(1, 2, 3), levels = 1:3))
predicted_probs <- predict(model, newdata = new_data, type = "response")
predicted_class <- ifelse(predicted_probs > 0.5, 1, 0)
print(cbind(new_data, prob = predicted_probs, class = predicted_class))

# ROC curve and AUC
library(pROC)
roc_obj <- roc(df$y, predict(model, type = "response"))
plot(roc_obj, main = "ROC Curve")
auc(roc_obj)  # Area Under the Curve

Practical Implications:

1

Odds ratios quantify how each predictor increases or decreases the odds of the outcome.
2

Predicted probabilities can be used for risk assessment, classification, or decision support.
3

ROC curves and classification metrics help evaluate the model's discriminative ability.
4

The model can identify the most important factors affecting the probability of the outcome.

Common Pitfalls and Limitations

Separation: Complete or quasi-complete separation can occur when a predictor perfectly predicts the outcome, leading to unstable estimates.
Sample size: Too few events per variable can lead to biased estimates and wide confidence intervals.
Probability interpretation: Misinterpreting odds ratios as risk ratios, especially when events are not rare.
Threshold selection: Using 0.5 as a classification threshold may not be optimal when classes are imbalanced.
Non-linearity: Assuming linear relationships in the logit scale when they might be non-linear.