Published in volume 109, pages 71-76 of AEA Papers and Proceedings, May 2019, Abstract: Concerns about the dissemination of spurious results have led to calls for pre-analysis plans … We thank Uber for donating ride-sharing services, Alameda County for donating vaccinations, and the Lenoirs for subletting their clinic. A pre-analysis plan is a credibly fixed plan of how a researcher will collect and analyze data, which is submitted before a project begins. Tracking Survey 1 Questionnaire; Midline 2.

Report; Household Questionnaire; Child Questionnaire; Data. A pre-analysis plan pre-specifies in a precise way the analysis to be run before examining the data. Tracking Survey 2 Questionnaire; Endline. Administration, pre-specified statistical analysis plans are required that explicitly spell out how data will be handled—and these analysis plans must be finalized and archived before researchers actually run regressions on the unblinded data (Food and Drug Administration 1998). paper was submitted for pre-publication re-analysis to the Abdul Latif Jameel Poverty Action Lab (J-PAL), where a code replication exercise was conducted on the analysis.

A researcher can specify variables, data cleaning procedures, regression specifications, and so on. The Editors of Psychological Science encourage psychological scientists to preregister their research plans before collecting data (or at least before analyzing data). Improving research transparency through easier, faster access to studies in the AEA RCT Registry .

For more details about J-PAL’s replication work, visit https://osf.io/be432/.

This post was originally published here on the J-PAL Blog. By Keesler Welch, Senior Research Associate (Trial Registry), J-PAL Global, and James Turitto, Senior Research Manager, J-PAL Global. Posted on: 15 August 2019.

J-PAL and the American Economic Association (AEA) have been working together over the past year to create digital object identifiers, or DOIs, for each trial registered in the AEA Registry for Randomized Controlled Trials. But pre-specifying analysis plans comes at a cost. Jann Spiess, Microsoft Research. Registry Entry; Initial Analysis Plan, February 2016; Executed Analysis Plan with August 2018 Amendments Baseline. Report; Household Questionnaire; Child Questionnaire; Midline 1.

James Turitto. We … Authors: Keesler Welch. Pre-Analysis Plans in the Machine-Learning Era . AEA Analysis Plan. Sendhil Mullainathan, University of Chicago. Augmenting Pre-Analysis Plans with Machine Learning by Jens Ludwig, Sendhil Mullainathan and Jann Spiess. Jens Ludwig, University of Chicago. Though pre-analysis plans have been lauded in the popular press and across the social sciences, we will argue that enthusiasm for pre-analysis plans should be tempered for several reasons.

If the regressions are pre-specified in advance and researchers are required to report all the results they pre-specify, data-mining problems are greatly reduced.

Download link.

J-PAL and the American Economic Association (AEA) have been working together …

Improving research transparency through easier, faster access to studies in the AEA RCT Registry August 15, 2019.