Hi Micah:
My read of recent computational linguistics lit, where they do this a
whole lot, suggests that kappa (either Cohen's or Siegel and
Castellan's) is still pretty well regarded for two-coder agreement. Or
at least there's no consensus on anything better.
In addition to the web page Will cites, I'd suggest looking at the
following:
www.temple.edy/ispr/mmc/reliability
http://pareonline.net/getvn.asp?v=9&n=4
Also, Di Eugenio and Glass, "The Kappa statistic: a second look"
Computational Linguistics 32 (1): 95-101 (2004), available at
http://www.mglass.org/papers/papers.html
And for a very nice take on multicoder agreement, including
Krippendorf's alpha:
Artstein & Poesio, "Kappa^3 = Alpha (or Beta)", NLE Technical Note
05-1, CS Technical Report CSM-437, ISSN 1744-8050 (September 29, 2005)
Good luck -
Anna Maria
Anna Maria Ortiz, Ph.D.
Senior Statistician
Applied Research and Methods
U.S. Government Accountability Office
(202) 512-2788
[log in to unmask]
>>> Will Lowe <[log in to unmask]> 10/15/2006 10:08 AM >>>
Micah,
Before choosing a statistic it might be useful to ask yourself two
questions:
First, what is the unit of agreement? You might either ask to what
extent your coders agree on phrases, or to what extent they agree
about the keywords that will be extracted from those phrases and put
into the search engine. For sub-phrase search keywords, agreement on
the latter may be both higher and more relevant with respect to your
final purpose.
Second, do you intend to have a gold standard coding? A gold standard
coding might be produced either by coding all the platforms yourself,
or by the less taxing methods described in King and Lowe (2003).
With these answers in hand, a sensible discussion on choosing a
statistic is:
http://ourworld.compuserve.com/homepages/jsuebersax/agree.htm
Will Lowe
Methods and Data Institute
University of Nottingham
On 10/14/06, Micah Weinberg, University of North Carolina at Chapel
Hill <[log in to unmask]> wrote:
> Folks:
>
> What is a good measure of intercoder reliability for the following
situation:
>
> Two coders review political texts (party platforms) to extract
> (stemmed) partisan frames (in this case frame=phrase) to be used as
> keyword searches in digital archives of newspapers. There has been
> substantial discussion ahead of time as to what constitutes a
partisan
> frame as well as review of texts (outside of the sample) to discuss
> examples of such frames. Each coder independently develops a list
of
> stemmed phrases. I would like to calculate the intercoder
reliability
> using these lists.
>
> Thank you very much!
>
> Micah Weinberg
> Doctoral Candidate
> Department of Political Science
>
> **********************************************************
> Political Methodology E-Mail List
> Editor: Karen Long Jusko <[log in to unmask]>
> **********************************************************
> Send messages to [log in to unmask]
> To join the list, cancel your subscription, or modify
> your subscription settings visit:
>
> http://polmeth.wustl.edu/polmeth.php
>
> **********************************************************
>
**********************************************************
Political Methodology E-Mail List
Editor: Karen Long Jusko <[log in to unmask]>
**********************************************************
Send messages to [log in to unmask]
To join the list, cancel your subscription, or modify
your subscription settings visit:
http://polmeth.wustl.edu/polmeth.php
**********************************************************
**********************************************************
Political Methodology E-Mail List
Editor: Karen Long Jusko <[log in to unmask]>
**********************************************************
Send messages to [log in to unmask]
To join the list, cancel your subscription, or modify
your subscription settings visit:
http://polmeth.wustl.edu/polmeth.php
**********************************************************
|
