A new document has been added to the Society for Political Methodology Website.

Title: How conditioning on post-treatment variables can ruin your experiment and what to do about it

Author(s): Jacob M. Montgomery, Brendan Nyhan, Michelle Torres

Entry Date: 2016-11-03

Keyword(s): experiments, post-treatment, bias

Abstract: In principle, experiments offer a straightforward method for social scientists to accurately estimate causal effects. However, scholars often unwittingly distort treatment effect estimates by conditioning on variables that could be affected by their experimental manipulation. Typical examples include controlling for post-treatment variables in statistical models or eliminating observations based on post-treatment criteria. Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal effects. Moreover, problems associated with conditioning on post-treatment variables remain largely unrecognized in the field, which we show frequently publishes experimental studies using these practices in our discipline's most prestigious journals. We demonstrate the severity of experimental post-treatment bias analytically and document the magnitude of the potential distortions it induces using visualizations and reanalyses of real-world data. We conclude by providing applied researchers with recommendations for best practice.

http://polmeth.wustl.edu/node/1477

**************************************************************
               Political Methodology E-Mail List
   Editors: Jonathan Homola  <[log in to unmask]>
               Michelle Torres     <[log in to unmask]>
**************************************************************
     Send messages to [log in to unmask]
  To join the list, cancel your subscription, or modify
                 your subscription settings visit:

             http://polmeth.wustl.edu/mailing-list

**************************************************************