Summary of the paper

Title Word Sense Annotation of Polysemous Words by Multiple Annotators
Authors Rebecca J. Passonneau, Ansaf Salleb-Aoussi, Vikas Bhardwaj and Nancy Ide
Abstract We describe results of a word sense annotation task using WordNet, involving half a dozen well-trained annotators on ten polysemous words for three parts of speech. One hundred sentences for each word were annotated. Annotators had the same level of training and experience, but interannotator agreement (IA) varied across words. There was some effect of part of speech, with higher agreement on nouns and adjectives, but within the words for each part of speech there was wide variation. This variation in IA does not correlate with number of senses in the inventory, or the number of senses actually selected by annotators. In fact, IA was sometimes quite high for words with many senses. We claim that the IA variation is due to the word meanings, contexts of use, and individual differences among annotators. We find some correlation of IA with sense confusability as measured by a sense confusion threshhold (CT). Data mining for association rules on a flattened data representation indicating each annotator's sense choices identifies outliers for some words, and systematic differences among pairs of annotators on others.
Topics Word Sense Disambiguation, Text mining, Lexicon, lexical database
Full paper Word Sense Annotation of Polysemous Words by Multiple Annotators
Slides Word Sense Annotation of Polysemous Words by Multiple Annotators
Bibtex @InProceedings{PASSONNEAU10.922,
  author = {Rebecca J. Passonneau and Ansaf Salleb-Aoussi and Vikas Bhardwaj and Nancy Ide},
  title = {Word Sense Annotation of Polysemous Words by Multiple Annotators},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  month = {may},
  date = {19-21},
  address = {Valletta, Malta},
  editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias},
  publisher = {European Language Resources Association (ELRA)},
  isbn = {2-9517408-6-7},
  language = {english}
Powered by ELDA © 2010 ELDA/ELRA