Appendix C
Research Checklist

Purpose

• increase statistical power

• resolve uncertainty

• improve estimate of effect size

• answer questions not posed at the start of individual trials.

Study Design

• Protocol

  • criteria for inclusion

  • methods to be used

• Literature Search

  • consult electronic databases

  • reviews

  • textbooks

  • experts in the field

  • review references in trials found

• List of Trials Analyzed and Log of Rejected Trials

  • generate list

  • enumerate and reason for exclusion

• Treatment Assignment

  • record method of control in primary study

  • pooling of randomized vs. non-randomized data

• Ranges of Characteristics and Treatments

  • present range of subject characteristics in primary studies

  • present range of treatment methods in primary studies

Combinability
A major issue in pooling data is whether the results of separate trials can be meaningfully combined. (451)

• Criteria detailed

• Measurement of homogeneity

Control and Measurement of Potential Bias

• Selection Bias (blinding technique for accepting and rejecting)

  • look at methods only

  • code method and result separately

• Data-Extraction Bias

  • proper blinding of extraction process

  • multiple coders or statistical validation of primary coder

• Source of Support Stated

Statistical Analysis

• Statistical Method

  • any recognized technique

  • no simple addition of successes across trials

• Statistical Errors

  • awareness of Type I

  • awareness of Type II

• Confidence Intervals

  • of difference between success rate of treatments

• Subgroup Analysis

  • purpose of meta-analysis

Sensitivity Analysis

• depends on statistical test chosen

• data combined to give different conclusions

• Quality Assessment

  • methodologic rigor

  • scientific quality

  • randomization process

  • measurement of subject compliance

  • blinding of subjects and observers

  • statistical analysis

  • handling of withdrawals in primary studies

• Varying Methods Sensitivity Analysis

  • to establish result spectrum

• Publication Bias

  • calculate number of negative studies needed to refute findings

  • published vs. Unpublished

Application of Results

• Caveats

  • how should results be used/focus of observation and recommendation

  • opportunities for research

  • research method refinements that are needed

• Economic Impact

  • of adopting new methods

Remaining Problems

• appropriateness of meta-analysis

  • efficacy established through definition and design

  • role exists when definitive randomized trial impractical or impossible

  • gives quantitative estimates of weight of available evidence

• meta-analysis question stated and relevant

• Limitations

  • location of primary studies

  • statistical analysis

  • publication bias vs. study quality

• Sufficient Information for Reader Conclusion About Validity

• Scientific Rigor of Meta-Analysis

  • clear question

  • clear method to be used

  • intraobserver and interobserver variability estimates

  • identify and minimize bias

  • establish validity of pooling

Sources:
Sacks, H. S., Berrier, J., Reitman, D., Ancona-Berk, V. A.,Chalmers, T. C. (1987, February 14). Meta-analyses of randomized controlled trials. The New England Journal of Medicine, (316), 8, 450-455.

Cooper & Hedges, 1994.