The Alpha resources page

This is a companion page to the paper Inter-coder agreement for computational linguistics, a survey of coefficients of agreement (such as Kappa and Alpha) to appear in Computational Linguistics. 

Whereas Kappa is reasonably well-known and software for computing it can be easily found and is incorporated in many statistical packages, Alpha is less understood and more difficult to use. The aim of this page is to keep track of resources concerning alpha, such as introductory material,    software, and papers using Alpha for computational linguistics. 

Documentation

The main source of documentation on  Alpha is Klaus Krippendorff's Content Analysis textbook - we recommend the 2nd edition as it contains a much more extensive discussion of Alpha:

Klaus Krippendorff. Content Analysis, An Introduction to Its Methodology 2nd Edition; 413 pages.  Thousand Oaks, CA: Sage Publications, 2004.

Krippendorff also wrote a paper specifically on the differences between Alpha and Kappa:

Klaus Krippendorff (2008 in press), Systematic and Random Disagreement and the Reliability of Nominal Data. Communication Methods and Measures.

Further information and  a number of useful practical manuals can be found in Klaus Krippendorff's page. In particular, we would suggest looking at   http://www.asc.upenn.edu/usr/krippendorff/webreliability.doc

For the use of Alpha in Computational Linguistics, apart from our paper, we recommend having a look at Passonneau's LREC 2004 paper. 

Software

Papers on using alpha in Computational Linguistics

Using coefficients of agreement

We list  here some useful papers and / or websites on coefficients of agreement not necessarily focusing on alpha

The ARRAU project was funded by the EPSRC, grant number GR/S76434/01

Modified 23 July 2008; maintained by Massimo Poesio.