We would all be well advised to follow their advice to examine
gradients, trace plots, the Hessian, and likelihood profiles and to
avoid an overreliance on numerical optimization when more information
can be given to the solver that is also useful for the analyst in
assessing convergence.  The BUGS bug is a different story.

But, my point was that the value of that entire exchange is at least
in part that such an exchange could take place and that the free flow
of information is permitted by the replication policies of the
journals.  We have no idea how many such results are floating around
in political science journals.  What is worse is that it seems
acceptable to employ the second strategy cited by McCullough/Vinod
upon a request for data; make it impossible for them to try through
sloppy documentation or just ignore the request.  In the end, they
will take up the work of scholars who devote precious time to making
their work easy to replicate leaving all the discoverable falsehoods
unexplored.

The argument that I am making is that this entire discussion about
software and programming languages has a self-evident solution in
transparency.  What code from what software, what data, etc. etc. is
something that should be a part of peer review and/or the vetting of
results.  That it is not seems to say quite a bit.

Best,
RWW

On Oct 10, 2006, at 1:08 PM, Jasjeet Singh Sekhon wrote:

>
> The subsequent article by Drukker and Wiggins (2004) (cf. McCullough
> and Vinod's (2004)) does not alter McCullough and Vinod's original
> point regarding the poor quality of statistical software for even
> estimating a *probit*.  Although, in the data example, the Hessian can
> be made well behaved after a rescaling, software programs, such as
> Gauss, were reporting solutions under a scaling where there should be
> none given the limitations of computer numerical precision, and the
> software packages failed to report the problem.  (note: the software
> was not doing an internal rescaling which is what good software should
> do with these problems).  Since the well scaled solution isn't very
> different from Shachar and Nalebuff's (1999) original solution, their
> substantive finding survives, but the issue remains that software was
> claiming to have converged even though the the Hessian was
> ill-conditioned. And, in general, the well scaled solution need not be
> near the poorly scaled one. Rescaling has long been known to be an
> important issue in optimization and indeed even for inverting (X'X)-1.
> See, for example, Gill, Murray and Wright (1981).
>
> And if this issue is too subtle for you to worry about regarding
> software packages, recall the infamous WinBUGS syntax interpreter bug
> which parsed -(b-c) as -b-c. Screwing up the basic properties of
> arithmetic: http://www.cognigencorp.com/nonmem/nm/99jun042004.html
>
> Jas.
>
> =======================================
> Jasjeet S. Sekhon
>
> Associate Professor
> Travers Department of Political Science
> Survey Research Center
> UC Berkeley
>
> http://sekhon.berkeley.edu/
> V: 510-642-9974  F: 617-507-5524
> =======================================
>
>
>
> Robert W. Walker writes:
>> Just to clarify Jas's comment about McCullough and Vinod (2003), the
>> followups in the AER show that McCullough and Vinod were largely
>> incorrect [see the replies by Wiggins and Drukker (2004) and
>> McCullough and Vinod (2004)], though there is much to be said for the
>> spirit of their paper.  In the end, the solution indeed exists with
>> some rescaling and the solution is quite close to the result
>> originally reported by Shachar and Nalebuff.  My suspicion is that
>> they were unlucky, there are plenty of false numerical solutions out
>> there; they just found one that appeared false and was not.
>>
>> Their conclusion [in 2004 and other papers] concerns a replication
>> policy and the need for both code and data.  Whether one is
>> inherently skeptical or otherwise of the programming talents of
>> social scientists is a non-issue; we have no way of verifying this
>> conclusion without evidence and we seem to have decided, as a
>> discipline, that we would just rather not know.  The most pernicious
>> of falsehoods survive under the guise of science without the critical
>> attributes that make them scientific -- reproducibility.
>>
>> Best,
>> RWW
>>
>> Robert W. Walker
>> Assistant Professor
>> Department of Political Science
>> Program in Applied Statistics and Computation
>> Washington University in Saint Louis
>> Campus Box 1063
>> One Brookings Drive
>> Saint Louis, Missouri 63130-3899
>> rww at wustl.edu
>> http://rww.wustl.edu
>>
>> **********************************************************
>>              Political Methodology E-Mail List
>>         Editor: Karen Long Jusko <[log in to unmask]>
>> **********************************************************
>>         Send messages to [log in to unmask]
>>   To join the list, cancel your subscription, or modify
>>            your subscription settings visit:
>>
>>           http://polmeth.wustl.edu/polmeth.php
>>
>> **********************************************************

Robert W. Walker
Assistant Professor
Department of Political Science
Program in Applied Statistics and Computation
Washington University in Saint Louis
Campus Box 1063
One Brookings Drive
Saint Louis, Missouri 63130-3899
rww at wustl.edu
http://rww.wustl.edu

**********************************************************
             Political Methodology E-Mail List
        Editor: Karen Long Jusko <[log in to unmask]>
**********************************************************
        Send messages to [log in to unmask]
  To join the list, cancel your subscription, or modify
           your subscription settings visit:

          http://polmeth.wustl.edu/polmeth.php

**********************************************************