Linear Regression

Introduction

A powerful statistical approach that models the linear relationship between a dependent variable and one or more independent variables. It finds the best-fitting straight line through the data by minimizing the sum of squared residuals, making it ideal for prediction, exploration of relationships, and testing causal hypotheses when assumptions are met.

This guide will help you interpret the results of a Linear Regression analysis. We'll walk through:

Data Description

This analysis was performed on a dataset with appropriate characteristics for this model.

Model Output Interpretation

> summary(df)
       y                x1             x2             x3    
 Min.   : 4.83   Min.   : 5.40   Min.   :0.00   1:28     
 1st Qu.: 7.68   1st Qu.: 8.68   1st Qu.:0.00   2:31     
 Median : 8.74   Median :10.17   Median :0.00   3:41     
 Mean   : 8.87   Mean   :10.07   Mean   :0.46          
 3rd Qu.:10.20   3rd Qu.:11.49   3rd Qu.:1.00          
 Max.   :12.61   Max.   :14.65   Max.   :1.00          

> cor(df[, c("y", "x1", "x2")])
          y        x1        x2
y  1.000000 0.8046874 0.4486060
x1 0.804687 1.0000000 0.0322727
x2 0.448606 0.0322727 1.0000000

> model <- lm(y ~ x1 + x2 + x3, data = df)
> summary(model)

Call:
lm(formula = y ~ x1 + x2 + x3, data = df)

Residuals:
     Min       1Q   Median       3Q      Max 
-2.05111 -0.62366  0.01062  0.70315  2.10890 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   2.1344     0.4940   4.321 3.92e-05 ***
x1            0.4921     0.0457  10.769  < 2e-16 ***
x2            1.4975     0.1866   8.025 3.41e-12 ***
x32           0.1977     0.2534   0.780   0.4373    
x33           0.1741     0.2309   0.754   0.4527    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.9766 on 95 degrees of freedom
Multiple R-squared:  0.7324,	Adjusted R-squared:  0.7208 
F-statistic: 64.93 on 4 and 95 DF,  p-value: < 2.2e-16

> new_data <- data.frame(x1 = c(8, 10, 12), x2 = c(0, 1, 0), x3 = factor(c(1, 2, 3), levels = 1:3))
> predictions <- predict(model, newdata = new_data, interval = "confidence")
> print(cbind(new_data, predictions))
   x1 x2 x3      fit      lwr      upr
1  8  0  1  6.071198  5.67851  6.46389
2 10  1  2  8.576220  8.17784  8.97460
3 12  0  3 10.146060  9.63766 10.65446

> plot(model)

Understanding the Output:

The summary table above provides key information about your linear regression model:

• Coefficients: The estimate column shows the effect of each predictor on the outcome. For every one-unit increase in a predictor, the outcome changes by this amount, holding other variables constant.
• Standard Error: Measures the precision of the coefficient estimates.
• t value: The coefficient divided by its standard error, used for hypothesis testing.
• p-value: The probability of observing the data if the null hypothesis (coefficient = 0) is true. Values below 0.05 are typically considered statistically significant.
• R-squared: The proportion of variance in the dependent variable explained by the model (ranges from 0 to 1).
• F-statistic: Tests whether the model as a whole explains significant variation in the outcome.

Coefficient Interpretation

In linear regression, coefficients represent the change in the outcome variable associated with a one-unit change in the predictor, holding all other variables constant:

• The intercept is the expected value of the outcome when all predictors are zero.
• For continuous predictors, the coefficient represents the slope of the relationship between that predictor and the outcome.
• For binary predictors, the coefficient represents the difference in the outcome between the two groups.
• For categorical predictors, each coefficient represents the difference in the outcome compared to the reference category.

Statistical significance (p-value) helps determine which coefficients reflect real effects rather than random variation.

Diagnostic Plots

Diagnostic plots help assess the validity of the model's assumptions. These four standard plots provide visual checks for linearity, normality, homoscedasticity, and influential points.

Model Assumptions

The Linear Regression relies on the following assumptions:

Prediction and Practical Implications

Linear regression allows you to predict the value of the outcome variable for new observations. The prediction includes both a point estimate and a confidence interval.

x1 <- rnorm(n, mean = 10, sd = 2)  # continuous predictor
x2 <- rbinom(n, 1, 0.5)  # binary predictor
x3 <- factor(sample(1:3, n, replace = TRUE))  # categorical predictor with 3 levels
# Create outcome with a linear relationship plus some noise
y <- 2 + 0.5 * x1 + 1.5 * x2 + rnorm(n, mean = 0, sd = 1)
# Combine into a data frame
df <- data.frame(y = y, x1 = x1, x2 = x2, x3 = x3)

# Descriptive statistics
summary(df)
cor(df[, c("y", "x1", "x2")])
boxplot(y ~ x2, data = df, main = "Y by Binary Predictor", xlab = "X2", ylab = "Y")
plot(x1, y, main = "Scatterplot of Y vs X1", xlab = "X1", ylab = "Y")

# Model fitting
model <- lm(y ~ x1 + x2 + x3, data = df)
summary(model)

# Diagnostic plots
par(mfrow = c(2, 2))
plot(model)

# Predictions
new_data <- data.frame(x1 = c(8, 10, 12), x2 = c(0, 1, 0), x3 = factor(c(1, 2, 3), levels = 1:3))
predictions <- predict(model, newdata = new_data, interval = "confidence")
print(cbind(new_data, predictions))

Practical Implications:

Common Pitfalls and Limitations

Further Reading