Readers of this list who have not seen it likely will find of interest the
following editorial from last week's Science:

 

SCIENCE

sciencemag.org

4 JULY 2014 . VOL 345 Issue 6192

 

 

"Raising the Bar"

 

Marcia McNutt

Editor-in-Chief

Science

 

"Numbers. Lots and lots of numbers. It is hard to find a paper published in
Science or any other journal that is not full of numbers. Interpretation of
those numbers provides the basis for the conclusions, as well as an
assessment of the confidence in those conclusions. But unfortunately, there
have been far too many cases where the quantitative analysis of those
numbers has been flawed, causing doubt about the authors' interpretation and
uncertainty about the result. Furthermore, it is not realistic to expect
that a technical reviewer, chosen for her or his expertise in the topical
subject matter or experimental protocol, will also be an expert in data
analysis. For that reason, with much help from the American Statistical
Association, Science has established, effective 1 July 2014, a Statistical
Board of Reviewing Editors (SBoRE), consisting of experts in various aspects
of statistics and data analysis, to provide better oversight of the
interpretation of observational data.  

 

"For those familiar with the role of Science's Board of Reviewing Editors
(BoRE), the function of the SBoRE will be slightly different. Members of the
BoRE perform a rapid quality check of manuscripts and recommend which should
receive in-depth review by technical specialists. Members of the SBoRE will
receive manuscripts that have been identified by editors, BoRE members, or
possibly reviewers as needing additional scrutiny of the data analysis or
statistical treatment. The SBoRE member assesses what the issue is that
requires screening and suggests experts from the statistics community to
provide it. 

 

"So why is Science taking this additional step? Readers must have confidence
in the conclusions published in our journal. We want to continue to take
reasonable measures to verify the accuracy of those results. We believe that
establishing the SBoRE will help avoid honest mistakes and raise the
standards for data analysis, particularly when sophisticated approaches are
needed. But even when taking added precautions, no review system is
infallible, and no combination of reviewers can be expected to uncover all
of the ways in which the interpretation of results may have gone wrong. In
particular, it is difficult for reviewers to detect whether authors have
approached the study with a lack of bias in their data collection and
presentation.  

 

"I recall a study that I conducted years ago involving a global analysis of
some oceanographic features that I was modeling to understand the rheology
of oceanic plates on million-year time scales. I had only a handful of data
points-perhaps a dozen or so-and the fit to my model failed a significance
test. Clearly, throwing out a few of the data points by declaring them
'outliers' would have improved the fit dramatically, and in fact I even
recall a reviewer of the paper commenting: 'Can't you make the data fit the
model better?'

 

"Really? 

 

"The editor published the paper despite the lousy fit of the model to the
data. It was not too long before it was realized that those "outliers" were
the key to a more complete understanding of the long-term rheological
behavior of the oceanic plates. Although the model in the earlier paper
needed an overhaul, the fundamental observations, because they were
presented without bias, inspired much further progress in the field.  

 

"In the years since, I have been amazed at how many scientists have never
considered that their data might be presented with bias. There are
fundamental truths that may be missed when bias is unintentionally
overlooked, or worse yet, when data are "massaged." Especially as we enter
an era of "big data," we should raise the bar ever higher in scrutinizing
the analyses that take us from observations to understanding."

 

Kim

 

Kim Quaile Hill, Ph.D.
Cullen-McFadden Professor of Political Science, 
Presidential Professor for Teaching Excellence, and
Eppright Professor of Teaching Excellence
Department of Political Science
Texas A&M University
MS 4348
College Station, TX 77845
Email:  [log in to unmask] <mailto:[log in to unmask]> 

 


**********************************************************
             Political Methodology E-Mail List
   Editors: Ethan Porter        <[log in to unmask]>
            Gregory Whitfield   <[log in to unmask]>
**********************************************************
        Send messages to [log in to unmask]
  To join the list, cancel your subscription, or modify
           your subscription settings visit:

          http://polmeth.wustl.edu/polmeth.php

**********************************************************