Hi Steve, I think your "manual" matching procedure is likely to be 
better than propensity score matching for your application.  PSM works 
when the data fit the conditions of the EPBR class of methods 
(multivariate normal and some other sampling processes fit, but probably 
not your data), and then is designed to work only in expectation across 
experiments, whereas you need treatment-control balance in your 
particular sample.  Moreover, you'd need to get the PSM logit or other 
specification at least approximately correct, and the usual logit fit 
diagnostics are not helpful: if the specification is correct, it 
balances, but the only way to check if its correct is to check balance 
after the fact, and if you can do that it no longer matters whether the 
specification is correct (Dan, Kosuke, Liz and I in our PA article last 
year call this the "propensity score tautology").  Since you say your 
manual procedure guarantees balance, you don't need to worry about all 
this; you already have what you need.  You can implement something close 
to your manual procedure via "coarsened exact matching", which also has 
other desirable statistical properties without the data restrictions, 
and some software too; see http://gking.harvard.edu/cem.

Your idea of balancing on geographic distance (no random walk necessary) 
is also a really good idea since you're likely to be controlling for a 
variety of otherwise unmeasured confounders, although you want to be 
sure to go far enough from the treated unit so that the treated person 
is not influencing the control's outcomes.

Gary
---
http://gking.harvard.edu


On 09/30/2008 01:14 PM, Steven Finkel wrote:
> Dear Colleagues:
> I am conducting an evaluation of a nation-wide (non-US) program in which individuals were treated, mostly but not always one time, and on a voluntary (self-selected) basis.  Based on past experience, I know that treated individuals are likely to be less poor and less socially-isolated than the average person, with a few other less-pronounced demographic differences.  The evaluation is to be a one-shot, post-only design comparing a random sample of treated individuals with a control group.  The treatment sample will be drawn from lists kept by the organizations that implemented the program. 
>
> I have a question concerning the best sampling method for obtaining the control group.
>
> In past evaluations, I have matched on certain demographic characteristics by using a random walk procedure in each treated individual's neighborhood to find a person with similar demographics who was not treated.  I have sometimes used propensity-score matching in conjunction with this procedure when doing the statistical analyses.  Would it be just as good, better or worse to obtain a control group with a national random sample and simply use PSM in the analyses?  This would seem to give a better control group in terms of generalizability, and also would seem to correspond to the advice given in some textbook treatments of PSM in terms of running the first stage logistic regression to generate the probability of treatment.  It would also be cheaper to implement. On the other hand, the "manual" matching sampling method is intuitively straightforward, appeals to the non-statisticians who are funding the study, and also guarantees balance on place of
>  residence and other specific demographics I specify in the matching procedure.
>
> Any insights or advice about the merits and drawbacks of these approaches would be very much appreciated.
>
> Best,
>
> Steve Finkel
> Department of Political Science
> University of Pittsburgh
>
> **********************************************************
>              Political Methodology E-Mail List
>    Editors: Melanie Goodrich, <[log in to unmask]>
>             Delia Bailey, <[log in to unmask]>
> **********************************************************
>         Send messages to [log in to unmask]
>   To join the list, cancel your subscription, or modify
>            your subscription settings visit:
>
>           http://polmeth.wustl.edu/polmeth.php
>
> **********************************************************
>   

**********************************************************
             Political Methodology E-Mail List
   Editors: Melanie Goodrich, <[log in to unmask]>
            Delia Bailey, <[log in to unmask]>
**********************************************************
        Send messages to [log in to unmask]
  To join the list, cancel your subscription, or modify
           your subscription settings visit:

          http://polmeth.wustl.edu/polmeth.php

**********************************************************