Guidelines for Multiple Testing in Impact Evaluations of Educational Interventions.

Publication Date: May 2008

Publisher: Mathematica Policy Research, Inc.

Author(s): Peter Z. Schochet

Research Area: Education

Keywords: Statistical Power; Educational Interventions; Multiple Testing

Type: Report

Abstract:

Statistical procedures that correct for multiple testing typically result in hypothesis tests with reduced statistical power because adjustment methods reduce the likelihood of identifying real differences between contrasted groups. There is disagreement among researchers about the use of multiple testing procedures and the appropriate trade-off between type I error and statistical power (type II error). These guidelines were developed to handle multiple testing in education research. In addition, the report provides details on the nature of the multiple testing problem and the statistical solutions that have been proposed; the creation of composite outcomes measures; and the Bayesian hypothesis testing approach.