title: The difference between ``significant'' and ``not significant'' is not itself statistically significant authors: Andrew Gelman, Hal Stern entrydate: 2005-12-23 08:25:49 keywords: multilevel modeling, multiple comparisons, replication, statistical significance abstract: A common error in statistical analyses is to summarize comparisons by declarations of statistical significance or non-significance. There are a number of difficulties with this approach. First is the oft-cited dictum that statistical significance is not the same as practical significance. Another difficulty is that this dichotomization into significant and non-significant results encourages the dismissal of observed differences in favor of the usually less interesting null hypothesis of no difference. Here, we focus on a less commonly noted problem, namely that changes in statistical significance are not themselves significant. By this, we are not merely making the commonplace observation that any particular threshold is arbitrary---for example, only a small change is required to move an estimate from a 5.1% significance level to 4.9%, thus moving it into statistical significance. Rather, we are pointing out that even large changes in significance levels can correspond to small, non-significant changes in the underlying variables. We illustrate with a theoretical and an applied example. http://polmeth.wustl.edu/retrieve.php?id=573 ********************************************************** Political Methodology E-Mail List Editor: Karen Long Jusko <[log in to unmask]> ********************************************************** Send messages to [log in to unmask] To join the list, cancel your subscription, or modify your subscription settings visit: http://polmeth.wustl.edu/polmeth.php **********************************************************