title:         The difference between ``significant'' and ``not significant'' is not itself statistically significant
authors:       Andrew Gelman, Hal Stern
entrydate:     2005-12-23 08:25:49
keywords:      multilevel modeling, multiple comparisons,
replication, statistical significance
abstract:      A common error in statistical analyses is to summarize comparisons by declarations of statistical significance or non-significance.  There are a number of difficulties with this approach.  First is the oft-cited dictum that statistical significance is not the same as practical significance.  Another difficulty is that this dichotomization into significant and non-significant results encourages the dismissal of observed differences in favor of the usually less interesting null hypothesis of no difference.

Here, we focus on a less commonly noted  problem, namely that changes in statistical significance are not themselves significant.  By this, we are not merely making the commonplace observation that  any particular threshold is arbitrary---for example, only a small change is required to move an estimate from a 5.1% significance level to 4.9%, thus moving it into statistical significance.  Rather, we are pointing out that even large changes in significance levels can correspond to small, non-significant changes in the underlying variables.   We illustrate with a theoretical and an applied example.

http://polmeth.wustl.edu/retrieve.php?id=573

**********************************************************
             Political Methodology E-Mail List
        Editor: Karen Long Jusko <[log in to unmask]>
**********************************************************
        Send messages to [log in to unmask]
  To join the list, cancel your subscription, or modify
           your subscription settings visit:

          http://polmeth.wustl.edu/polmeth.php

**********************************************************