The Alpha resources page
This is a companion page to the paper Inter-coder agreement for computational linguistics, a survey of coefficients of agreement (such as Kappa and Alpha) to appear in Computational Linguistics.
Whereas Kappa is reasonably well-known and software for computing it can be easily found and is incorporated in many statistical packages, Alpha is less understood and more difficult to use. The aim of this page is to keep track of resources concerning alpha, such as introductory material, software, and papers using Alpha for computational linguistics.
Documentation
The main source of documentation on Alpha is Klaus Krippendorff's Content Analysis textbook - we recommend the 2nd edition as it contains a much more extensive discussion of Alpha:
Klaus Krippendorff. Content Analysis, An Introduction to Its Methodology 2nd Edition; 413 pages. Thousand Oaks, CA: Sage Publications, 2004.
Krippendorff also wrote a paper specifically on the differences between Alpha and Kappa:
Klaus Krippendorff (2008 in press), Systematic and Random Disagreement and the Reliability of Nominal Data. Communication Methods and Measures.
Further information and a number of useful practical manuals can be found in Klaus Krippendorff's page. In particular, we would suggest looking at http://www.asc.upenn.edu/usr/krippendorff/webreliability.doc
For the use of Alpha in Computational Linguistics, apart from our paper, we recommend having a look at Passonneau's LREC 2004 paper.
Software
- From Klaus Krippendorff's page
- Krippendorff has scripts for computing alpha but we are not sure about their availaibility
- See http://www.asc.upenn.edu/usr/krippendorff/webreliability.doc
- Andrew Hayes and Klaus Krippendorff's SPSS macros
- Available from http://www.comm.ohio-state.edu/ahayes/
- Select “SPSS and SAS Macros” then KALPHA.SPS
- the package is described at www.asc.upenn.edu/usr/krippendorff/CMM_alpha_final.doc
- Thomas Lippincott's and Becky Passonneau's Python scripts
- Thomas Lippincott and Becky Passonneau developed a Python script that can be used to compute a variety of agreement coefficients.
- Download the script
- Documentation
- Ron Artstein's Alpha Calculation script
- Ron Artstein wrote several scripts to compute alpha for the experiments ran in the ARRAU project. This script is a version for general use, computing interval alpha.
- Download the script
- Documentation
- Another SAS macro for computing Alpha
- Computing Alpha in R
- http://pbil.univ-lyon1.fr/library/concord/html/kripp.alpha.html
- (See also http://cran.r-project.org/bin/macosx/2.1/check/concord-check.ex)
- A general site on software for computing reliability
- The excellent site http://www.temple.edu/sct/mmc/reliability/ reviews the available software for computing reliability
Papers on using alpha in Computational Linguistics
- Ron Artstein and Massimo Poesio.
Inter-coder agreement for computational linguistics. (PDF, 41 pages)
Survey article, to appear in Computational Linguistics. - Passonneau, Rebecca; Lippincott, Tom; Yano, Tae; Klavans, Judith. 2008. Relation between Agreement Measures on Human Labeling and Machine Learning Performance Results from an Art History Domain. In Proc. of Language Resources and Evaluation (LREC). pdf
- Massimo Poesio and Ron Artstein.
Anaphoric annotation in the ARRAU corpus. (PDF)
LREC 2008, Marrakech, Morocco, May 2008. - Ron Artstein and Massimo Poesio.
The Arrau corpus of anaphoric relations. (PDF slides)
American Association for Corpus Linguistics, Provo, Utah, March 2008. - Nenkova, Ani; Passonneau, Rebecca J.; McKeown, Kathleen. 2007. The pyramid method: incorporating human content selection variation in summarization evaluation. ACM Transactions on Speech and Language Processing, 4(2). pdf
- Passonneau, Rebecca J. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In Proceedings of the Fifth Internation al Conference on Language Resources and Evaluation (LREC). pdf
- Ron Artstein and Massimo Poesio.
Identifying reference to abstract objects in dialogue. (PDF)
brandial 2006 proceedings, Potsdam, Germany, September 2006. - Ron Artstein and Massimo Poesio.
Bias decreases in proportion to the number of annotators. (PDF)
Proceedings of FG-MoL 2005, pages 141-150. Edinburgh, Scotland, August 2005. - Massimo Poesio and Ron Artstein.
Annotating (anaphoric) ambiguity. (PDF)
Corpus linguistics, Birmingham, England, July 2005. - Massimo Poesio and Ron Artstein.
The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. (PDF)
Proceedings of the Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, pages 76-83. Ann Arbor, Michigan, June 2005. - Ron Artstein and Massimo Poesio.
Kappa3 = alpha (or beta).
Technical Report CSM-437, University of Essex Department of Computer Science, September 2005.
Superseded by journal article Inter-coder agreement for computational linguistics, above. - Nenkova, Ani and Rebecca J. Passonneau. 2004. Evaluating Content Selection in Summarization: the Pyramid Method. NAACL. pdf
- Passonneau, Rebecca J. 2004. Computing Reliability for Coreference Annotation. In: Proceedings of Language Resources and Evaluation (LREC). pdf
Using coefficients of agreement
We list here some useful papers and / or websites on coefficients of agreement not necessarily focusing on alpha
- Jean Carletta, (1996). Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics, 22(2), 249-254. PDF.
- Ron Artstein and Massimo Poesio.
Inter-coder agreement for computational linguistics. (PDF, 41 pages)
Survey article, to appear in Computational Linguistics. - John Uebersax's page, and in particular http://ourworld.compuserve.com/homepages/jsuebersax/agree.htm
The ARRAU project was funded by the EPSRC, grant number GR/S76434/01