View source for Multiple linear regression

{{RoundBoxTop|theme=2}}
{{Tertiary}}
{{notes}}
This learning resource summarises the main teaching points about '''multiple linear regression''' (MLR), including key concepts, principles, assumptions, and how to conduct and interpret MLR analyses. 

Prerequisites:
# [[Linear correlation|Correlation]]
# [[Simple linear regression|Linear regression]]
{{RoundBoxBottom}}
{{RightTOC}}

== What is MLR? ==
* Multiple linear regression (MLR) is a [[Multivariate statistics|multivariate statistical technique]] for examining the [[linear correlation]]s between two or more [[Independent variable|independent variable]]s (IVs) and a single [[Dependent variable|dependent variable]] (DV).
* [[Research question]]s suitable for MLR can be of the form "To what extent do X1, X2, and X3 (IVs) predict Y (DV)?"<br>e.g., "To what extent does people's age and gender (IVs) predict their levels of blood cholesterol (DV)?"
* MLR analyses can be visualised as [[path diagram]]s and/or [[w:Venn diagram|venn diagram]]s

{{center top}}
<gallery>
File:Example path diagram (conceptual).png|MLR studies the relation between two or more IVs and a single DV.
File:Vidrarias de Laboratorio.jpg|MLR is used to statistically 'distill' the relative contribution of two or more [[independent variable|IV]]s on a single [[dependent variable|DV]].
Image:Question mark.svg|What other ways can you think of to explain the purpose of MLR?
</gallery>
{{center bottom}}

== [[/Assumptions|Assumptions]] ==
{{/Assumptions}}
== [[/Types|Types]] ==
{{/Types}}
==Results==
* MLR analyses produce several diagnostic and outcome statistics which are summarised below and are important to understand.
* Make sure that you can learn how to find and interpret these statistics from [[statistical software]] output.

===Correlations===
Examine the [[linear correlation]]s between (usually as a correlation matrix, but also view the scatterplots):
* IVs
* each IV and the DV
* DVs (if there is more than 1)

===Effect sizes===
====''R''====
# (Big) ''R'' is the '''multiple correlation coefficient''' for the relationship between the predictor and outcome variables.
# Interpretation is similar to that for little ''r'' (the [[linear correlation]] between two variables), however ''R'' can only range from 0 to 1, with 0 indicating no relationship and 1 a perfect relationship. Large values of ''R'' indicate more variance explained in the DV.
# ''R'' can be squared and interpreted as for ''r''<sup>2</sup>, with a rough rule of thumb being .1 (small), .3 (medium), and .5 (large). These ''R''<sup>2</sup> values would indicate 10%, 30%, and 50% of the variance in the DV explained respectively. 
# When generalising findings to the population, the ''R''<sup>2</sup> for a sample tends to overestimate the ''R''<sup>2</sup> of the population. Thus, adjusted ''R''<sup>2</sup> is recommended when generalising from a sample, and this value will be adjusted downward based on the sample size; the smaller the sample size, the greater the reduction. 
# The statistical significance of ''R'' can be examined using an ''F'' test and its corresponding ''p'' level.
# Reporting example: ''R''<sup>2</sup> = .32, ''F''(6, 217) = 19.50, ''p'' = .001
## "6, 217" refers to the [[degrees of freedom]] - for more information, see about half-down [http://academic.udayton.edu/gregelvers/psy216/spss/1wayanova.htm this page]

====Cohen's ''&fnof;''<sup>2</sup>====
* [[w:Effect_size#Cohen.27s_.C6.922|Cohen's ''&fnof;''<sup>2</sup>]] is based on the ''R''<sup>2</sup> and is an alternate indicator of [[effect size]] for MLR.

===Coefficients===
An MLR analysis produces several useful statistics about each of the predictors. These '''regression coefficients''' are usually presented in a Results table ([https://apastyle.apa.org/style-grammar-guidelines/tables-figures/sample-tables#regression example]) which may include: 
* Constant (or Intercept) - the starting value for DV when the IVs are 0
* B (unstandardised) - used for building a prediction equation
* Confidence intervals for B - the probable range of population values for the Bs
* β (standardised) - the direction and relative strength of the predictors on a scale ranging from -1 to 1
* [[Correlation|Zero-order correlation]] (''r'') - the correlation between a predictor and the outcome variable
* [[Partial correlation]]s (''pr'') - the unique correlations between each IV and the DV (i.e., without the influence of other IVs) (labelled "partial" in SPSS output)
* [[Semi-partial correlation]]s (''sr'') - similar to partial correlations (labelled "part" in SPSS output); squaring this value provides the percentage of variance in the DV uniquely explained by each IV (''sr''<sup>2</sup>)
* ''t'', ''p'' - indicates the statistical significance of each predictor. [https://stats.stackexchange.com/questions/93130/when-doing-a-t-test-for-the-significance-of-a-regression-coefficient-why-is-the Degrees of freedom for ''t'' is ''n'' - ''p'' - 1].

===Equation===
* A prediction equation can be derived from the regression coefficients in a MLR analysis.
* The equation is of the form<br>
<math>\hat{Y} = bx + a</math> (for predicted values) or <br>
<math>Y = bx + a + e</math> (for observed values)

===Residuals===
A residual is the difference between the actual value of a DV and its predicted value. Each case will have a residual for each MLR analysis. Three key [[#assumptions|assumptions]] can be tested using plots of residuals:
# Linearity: IVs are linearly related to DV
# Normality of residuals
# Equal variances (Homoscedasticity)

==Power==
* [http://www.danielsoper.com/statcalc3/calc.aspx?id=9 Post-hoc statistical power calculator for MLR] (danielsoper.com)

==Advanced concepts==
* [[Partial correlation]]s
* Use of hierarchical regression to partial out or remove the effect of 'control' variables
* Interactions between IVs
* [[w:Mediation (statistics)|Moderation and mediation]]

==Writing up==
When writing up the results of an MLR, consider describing:
* Assumptions: How were they tested? To what extent were the assumptions met?
* Correlations: What are they? Consider correlations between the IVs and the DV separately to the correlations between the IVs.
* Regression coefficients: Report a table and interpret
* Causality: Be aware of the limitations of the analysis - it may be consistent with a causal relationship, but it is unlikely to prove causality
* See also: [[Survey research and design in psychology/Assessment#Sample write-ups|Sample write-ups]]

==FAQ==
===What if there are univariate outliers?===
Basically, explore and consider what the implications might be - do these "outliers" impact on the [[#assumptions|assumptions]]? A lot depends on how "outliers" are defined. It is probably better to consider distributions in terms of the shape of the histogram and skewness and kurtosis, and whether these values are unduely impacting on the estimates of linear relations between variables. In other words, what are the implications? Ultimately, the researcher needs to decide whether the outliers are so severe that they are unduely influencing results of analyses or whether they are relatively benign. If unsure, explore, test, try the analyses with and without these values etc. If still unsure, be conservative and remove the data points or recode the data.

==See also==
{{wikipedia}}
* Tutorials/Activities
** [[/Activity/Life expectancy/|Life expectancy MLR activity]]
** [[/Quiz|MLR quiz]] (Practice)
** [[Survey research and design in psychology/Tutorials/Multiple linear regression|MLR data analysis tutorial]]
* Lectures
** [[Survey research and design in psychology/Lectures/Multiple linear regression I|Multiple linear regression I]]
** [[Survey research and design in psychology/Lectures/Multiple linear regression II|Multiple linear regression II]]
* Other
** [http://pareonline.net/getvn.asp?n=2&v=8 Four assumptions of multiple regression that researchers should always test] (Osborne & Waters,  2002)
** [[Least-Squares Fitting]]
** [[Logistic regression]]
** [[commons:Category:Multiple linear regression|Multiple linear regression]] (Commons)

===References===
# Allen & Bennett 13.3.2.1 Assumptions (pp. 178-179)
# Francis 5.1.4 Practical Issues and Assumptions (pp. 126-128)
# Green, S. B. (1991). [http://dx.doi.org/10.1207/s15327906mbr2603_7 How many subjects does it take to do a regression analysis?]. ''Multivariate Behavioral Research'', ''26'', 499-510.
# Knofczynski, G. T., & Mundfrom, D. (2008). [http://epm.sagepub.com/content/68/3/431.short Sample sizes when using multiple linear regression for prediction]. ''Educational and Psychological Measurement'', ''68'', 431-442.
# Wilson Van Voorhis, C. R. & Morgan, B. L. (2007). [http://www.tqmp.org/Content/vol03-2/p043/p043.pdf Understanding power and rules of thumb for determining sample sizes]. ''Tutorials in Quantitative Methods for Psychology'', ''3''(2), 43-50.

==External links==
* [http://www.westgard.com/lesson42.htm Correlation and simple least squares regression] (Zady, 2000)
* [http://www.statsoft.com/textbook/multiple-regression Multiple regression] (Statsoft)
* [http://eric.ed.gov/ERICWebPortal/detail?accno=ED470205 Multiple regression assumptions] (ERIC Digest)

[[Category:Multiple linear regression]]