Meta-analysis

Subject classification: this is a statistics resource.

Completion status: this resource is ~50% complete.

Meta-analysis is a systematic technique for reviewing, analysing, and summarising quantitative research studies on specific topics or questions.

This page provides information and resources about how to conduct a meta-analysis. The target audience includes post-graduate students conducting a meta-analysis or beginning researchers in meta-analysis. These pages could also be used by students involved in research methods coursework.

Introduction

1. A meta-analysis is quantitative technique for conducting a "study of studies".
2. Use of meta-analysis has flourished, particularly in the social, health, and medical sciences, since it was developed in the 1970s
3. Meta-analysis was initially developed in response to controversy over traditional, subjective literature review methods (specifically, at the time, those used in to review the psychotherapy outcome studies).

Lecture slides

1. Practical meta-analysis (Lecture slides; Wilson, 1999)
2. How to do meta-analysis (Lecture slides; Basu, 2005)
3. Meta-analysis: combining information(Lecture Slides; LeBauer, 2010) — includes outline of three approaches to inferences: moment matching, maximum likelihoo, and Bayesian.

How to do a meta-analysis

1. Meta-analysis involves analysing the summary data from many studies. It can be performed by hand, using a spreadsheet and formulae, using scripts, syntax or macros with generic statistics software packages, or by using dedicated meta-analysis software packages.
2. Before starting, identify a clear question(s), e.g., "What are the outcomes of psychotherapy?"
   1. Questions can also involve the effect of independent variables, e.g., "Are the outcomes of psychotherapy similar for males and females?"
   2. Read other related meta-analyses to get a feel for the kinds of questions asked.
   3. Make sure that any independent variables (IVs) and dependent variables (DVs) are very clearly defined.
   4. Because of the importance of establishing a well-defined question and variables, developing a peer-reviewed proposal for a meta-analytic study is strongly recommended.
3. It can be helpful to identify several similar or related meta-analytic studies as models for your meta-analytic study. Consider the strengths and weaknesses of their methodologies.
4. Establish clear criteria for selection of studies, e.g., does it need to be published in a peer-reviewed journal, or will you also accept theses and non-peer reviewed papers (e.g., conference papers)?
5. Conduct an exhaustive and systematic literature search, recording your steps along the way (important for the Method - must allow replication)
6. Create a "coding sheet" - this is the list of fields (variables) you want to extract from each study, and how each of the variables are to be coded - get this peer-reviewed, otherwise you will limit the potential/quality of your analyses
7. Enter the data - one study per row, but note that there may be multiple outcomes and/or groups of interest for each study, in which case each of these will receive their own row in the database, with a column to code which type of outcome was measured.
8. Analyse the data using spreadsheet formulae, or by writing syntax commands for a generic statistics package, or by using a dedicated meta-analysis software package (with in-built meta-analysis tools).

Effect sizes

1. Central to understanding meta-analysis is an understanding of effect sizes.
2. The chief value of effect sizes in the context of meta-analysis is that they provide a way to standardise effects across studies using different measures, allowing for common analysis.
3. There are many possible effect sizes, but essentially there are two commonly reported types in meta-analysis:
   1. Correlational: e.g., r (product-moment correlation)
   2. Mean differences: e.g., Cohen's d, Hedge's g, etc.

Limitations

1. An important limitation of meta-analysis is that its results can only be as good as the original data is valid.
2. Meta-analysis can only analyse the role of independent variables in explaining variance in dependent variables if sufficient data is provided in the original studies.
3. "Apples and oranges" effect - i.e., there is a risk/tendency in meta-analysis to average/mash together disparate effects.
4. Can lack in qualitative insight (e.g., as may be more likely to be contributed by an expert conducting a traditional literature review).

Example meta-analytic studies

1. Hattie, J., Biggs, J., Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis. Review of Educational Research, 66, 99-136.
2. Hattie, J., Marsh, H. W., Neill, J. T., & Richards, G. E. (1997). Adventure education and Outward Bound: Out-of-class experiences that make a lasting difference. Review of Educational Research, 67, 43-87.
3. Purdie, N., Hattie, J., Carroll, A. (2002). A review of the research on interventions for attention deficit hyperactivity disorder: What works best? Review of Educational Research, 77, 61-99.

Software

Comparison table

Some dedicated meta-analysis software includes:

Name	URL	License	$Cost	Trial or Demo?	Version	Notes
MetaXL	http://www.epigear.com	Shareware (no commercial redistribution, no derivatives)	Free	No	3.0	Can run bias adjusted (quality effects) meta-analyses and the IVhet model (alternative to the RE model) in addition to conventional (IV, RE, MH, Peto) models. This software creates forest and funnel plots and in addition has implemented the Doi plot and LFK index which are new tools for publication bias assessment.
CMA	http://www.meta-analysis.com	Proprietary	~1000	Yes	2
Metawin	http://www.metawinsoft.com	Proprietary	150
MIX	http://www.meta-analysis-made-easy.com	Proprietary	0-210	Yes	2.0	Student and academic licenses available
Cochrane's Review Manager (RevMan)	http://community.cochrane.org/tools/review-production-tools/revman-5/about#http://www.cc-ims.net/RevMan	Proprietary	Free for academic use, 730 GBP with discount for 4+ copies for commercial use.	Yes	5	For organising reviews; for MA, see [1]
rmeta	http://cran.r-project.org/web/packages/rmeta/index.html	GNU	0	NA	2.16
Meta-Analysis Calculator	http://www.lyonsmorris.com/lyons/maCalc/index.cfm	CC0	0	N/A	3.5	The Meta-Analysis Calculator is a web based application that conducts a Hunter-Schmidt (2004) type Meta-Analysis correcting for variation due to sampling and reliability. A future edition will allow for range restriction corrections. The results and the effect-size data set can be exported in Excel format and downloaded to the user's system.

Alternative (software

Non-dedicated, generic statistics software which can be used for conducting meta-analysis include:

• R
• Stata
• SPSS
• Excel
• DStat
• NCSS

Other comparisons/lists

1. w:Meta-analysis#Software
2. http://www.um.es/facpsi/metaanalysis/software.php
3. http://www.lehanathabane.com/personal/metalinks.htm
4. http://www.med.umich.edu/csp/Course%20materials/Fall%202005/Rogers_Meta%20Analysis%20software%20packages.pdf

Tasks

• To do

References

Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Sage. Thousand Oaks, CA.

See also

• Effect size
• Interest group

External links

• Practical meta-analysis effect size calculator (Wilson)
• Meta-analysis (psychwiki.com)
• Meta-analysis research methodology (Wilderdom)
• Meta-analysis stuff (Wilson)
• Meta-Analysis:Methods of Accumulating Results Across Research Domains (Online Paper)