Summary of the paper

Title Fine-grained Linguistic Evaluation of Question Answering Systems
Authors Sarra El Ayari, Brigitte Grau and Anne-Laure Ligozat
Abstract Question answering systems are complex systems using natural language processing. Some evaluation campaigns are organized to evaluate such systems in order to propose a classification of systems based on final results (number of correct answers). Nevertheless, teams need to evaluate more precisely the results obtained by their systems if they want to do a diagnostic evaluation. There are no tools or methods to do these evaluations systematically. We present REVISE, a tool for glass box evaluation based on diagnostic of question answering system results.
Topics Question Answering, Corpus (creation, annotation, etc.), Evaluation methodologies
Full paper Fine-grained Linguistic Evaluation of Question Answering Systems
Slides -
Bibtex @InProceedings{ELAYARI10.520,
  author = {Sarra El Ayari and Brigitte Grau and Anne-Laure Ligozat},
  title = {Fine-grained Linguistic Evaluation of Question Answering Systems},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  month = {may},
  date = {19-21},
  address = {Valletta, Malta},
  editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias},
  publisher = {European Language Resources Association (ELRA)},
  isbn = {2-9517408-6-7},
  language = {english}
Powered by ELDA © 2010 ELDA/ELRA