Micah,

Before choosing a statistic it might be useful to ask yourself two questions:

First, what is the unit of agreement?  You might either ask to what
extent your coders agree on phrases, or to what extent they agree
about the keywords that will be extracted from those phrases and put
into the search engine.  For sub-phrase search keywords, agreement on
the latter may be both higher and more relevant with respect to your
final purpose.

Second, do you intend to have a gold standard coding?  A gold standard
coding might be produced either by coding all the platforms yourself,
or by the less taxing methods described in King and Lowe (2003).

With these answers in hand, a sensible discussion on choosing a
statistic is: http://ourworld.compuserve.com/homepages/jsuebersax/agree.htm

Will Lowe
Methods and Data Institute
University of Nottingham


On 10/14/06, Micah Weinberg, University of North Carolina at Chapel
Hill <[log in to unmask]> wrote:
> Folks:
>
> What is a good measure of intercoder reliability for the following situation:
>
> Two coders review political texts (party platforms) to extract
> (stemmed) partisan frames (in this case frame=phrase) to be used as
> keyword searches in digital archives of newspapers.  There has been
> substantial discussion ahead of time as to what constitutes a partisan
> frame as well as review of texts (outside of the sample) to discuss
> examples of such frames.  Each coder independently develops a list of
> stemmed phrases.  I would like to calculate the intercoder reliability
> using these lists.
>
> Thank you very much!
>
> Micah Weinberg
> Doctoral Candidate
> Department of Political Science
>
> **********************************************************
>              Political Methodology E-Mail List
>         Editor: Karen Long Jusko <[log in to unmask]>
> **********************************************************
>         Send messages to [log in to unmask]
>   To join the list, cancel your subscription, or modify
>            your subscription settings visit:
>
>           http://polmeth.wustl.edu/polmeth.php
>
> **********************************************************
>

**********************************************************
             Political Methodology E-Mail List
        Editor: Karen Long Jusko <[log in to unmask]>
**********************************************************
        Send messages to [log in to unmask]
  To join the list, cancel your subscription, or modify
           your subscription settings visit:

          http://polmeth.wustl.edu/polmeth.php

**********************************************************