Doing so might set a helpful example for some major philanthropic funders
of social science research.  Some well-intentioned, but suboptimally
informed, institutions waste a nontrivial amount of money on dubious
"research" projects.


On Tue, Jul 15, 2014 at 8:11 PM, caesaigh <[log in to unmask]> wrote:

> Thanks for this. Perhaps our discipline is overdue in establishing- at
> some point/place -something similar. Too much dross makes print too often
> using a BTWBS format.
>
> C
>
> Sent from my iPhone
>
> > On 15 Jul 2014, at 16:31, Kim Hill <[log in to unmask]> wrote:
> >
> >
> >
> > Readers of this list who have not seen it likely will find of interest
> the
> > following editorial from last week's Science:
> >
> >
> >
> > SCIENCE
> >
> > sciencemag.org
> >
> > 4 JULY 2014 . VOL 345 Issue 6192
> >
> >
> >
> >
> >
> > "Raising the Bar"
> >
> >
> >
> > Marcia McNutt
> >
> > Editor-in-Chief
> >
> > Science
> >
> >
> >
> > "Numbers. Lots and lots of numbers. It is hard to find a paper published
> in
> > Science or any other journal that is not full of numbers. Interpretation
> of
> > those numbers provides the basis for the conclusions, as well as an
> > assessment of the confidence in those conclusions. But unfortunately,
> there
> > have been far too many cases where the quantitative analysis of those
> > numbers has been flawed, causing doubt about the authors' interpretation
> and
> > uncertainty about the result. Furthermore, it is not realistic to expect
> > that a technical reviewer, chosen for her or his expertise in the topical
> > subject matter or experimental protocol, will also be an expert in data
> > analysis. For that reason, with much help from the American Statistical
> > Association, Science has established, effective 1 July 2014, a
> Statistical
> > Board of Reviewing Editors (SBoRE), consisting of experts in various
> aspects
> > of statistics and data analysis, to provide better oversight of the
> > interpretation of observational data.
> >
> >
> >
> > "For those familiar with the role of Science's Board of Reviewing Editors
> > (BoRE), the function of the SBoRE will be slightly different. Members of
> the
> > BoRE perform a rapid quality check of manuscripts and recommend which
> should
> > receive in-depth review by technical specialists. Members of the SBoRE
> will
> > receive manuscripts that have been identified by editors, BoRE members,
> or
> > possibly reviewers as needing additional scrutiny of the data analysis or
> > statistical treatment. The SBoRE member assesses what the issue is that
> > requires screening and suggests experts from the statistics community to
> > provide it.
> >
> >
> >
> > "So why is Science taking this additional step? Readers must have
> confidence
> > in the conclusions published in our journal. We want to continue to take
> > reasonable measures to verify the accuracy of those results. We believe
> that
> > establishing the SBoRE will help avoid honest mistakes and raise the
> > standards for data analysis, particularly when sophisticated approaches
> are
> > needed. But even when taking added precautions, no review system is
> > infallible, and no combination of reviewers can be expected to uncover
> all
> > of the ways in which the interpretation of results may have gone wrong.
> In
> > particular, it is difficult for reviewers to detect whether authors have
> > approached the study with a lack of bias in their data collection and
> > presentation.
> >
> >
> >
> > "I recall a study that I conducted years ago involving a global analysis
> of
> > some oceanographic features that I was modeling to understand the
> rheology
> > of oceanic plates on million-year time scales. I had only a handful of
> data
> > points-perhaps a dozen or so-and the fit to my model failed a
> significance
> > test. Clearly, throwing out a few of the data points by declaring them
> > 'outliers' would have improved the fit dramatically, and in fact I even
> > recall a reviewer of the paper commenting: 'Can't you make the data fit
> the
> > model better?'
> >
> >
> >
> > "Really?
> >
> >
> >
> > "The editor published the paper despite the lousy fit of the model to the
> > data. It was not too long before it was realized that those "outliers"
> were
> > the key to a more complete understanding of the long-term rheological
> > behavior of the oceanic plates. Although the model in the earlier paper
> > needed an overhaul, the fundamental observations, because they were
> > presented without bias, inspired much further progress in the field.
> >
> >
> >
> > "In the years since, I have been amazed at how many scientists have never
> > considered that their data might be presented with bias. There are
> > fundamental truths that may be missed when bias is unintentionally
> > overlooked, or worse yet, when data are "massaged." Especially as we
> enter
> > an era of "big data," we should raise the bar ever higher in scrutinizing
> > the analyses that take us from observations to understanding."
> >
> >
> >
> > Kim
> >
> >
> >
> > Kim Quaile Hill, Ph.D.
> > Cullen-McFadden Professor of Political Science,
> > Presidential Professor for Teaching Excellence, and
> > Eppright Professor of Teaching Excellence
> > Department of Political Science
> > Texas A&M University
> > MS 4348
> > College Station, TX 77845
> > Email:  [log in to unmask] <mailto:[log in to unmask]>
> >
> >
> >
> >
> > **********************************************************
> >             Political Methodology E-Mail List
> >   Editors: Ethan Porter        <[log in to unmask]>
> >            Gregory Whitfield   <[log in to unmask]>
> > **********************************************************
> >        Send messages to [log in to unmask]
> >  To join the list, cancel your subscription, or modify
> >           your subscription settings visit:
> >
> >          http://polmeth.wustl.edu/polmeth.php
> >
> > **********************************************************
>
> **********************************************************
>              Political Methodology E-Mail List
>    Editors: Ethan Porter        <[log in to unmask]>
>             Gregory Whitfield   <[log in to unmask]>
> **********************************************************
>         Send messages to [log in to unmask]
>   To join the list, cancel your subscription, or modify
>            your subscription settings visit:
>
>           http://polmeth.wustl.edu/polmeth.php
>
> **********************************************************
>

**********************************************************
             Political Methodology E-Mail List
   Editors: Ethan Porter        <[log in to unmask]>
            Gregory Whitfield   <[log in to unmask]>
**********************************************************
        Send messages to [log in to unmask]
  To join the list, cancel your subscription, or modify
           your subscription settings visit:

          http://polmeth.wustl.edu/polmeth.php

**********************************************************