LREC 2006 - Proceedings sorted by papers

Title	A Model for Context-Based Evaluation of Language Processing Systems and its Application to Machine Translation Evaluation
Authors	A. Popescu-belis, P. Estrella, M. King, N. Underwood
Abstract	In this paper, we propose a formal framework that takes into account the influence of the intended context of use of an NLP system on the procedure and the metrics used to evaluate the system. We introduce in particular the notion of a context-dependent quality model and explain how it can be adapted to a given context of use. More specifically, we define vector-space representations of contexts of use and of quality models, which are connected by a generic contextual quality model (GCQM). For each domain, experts in evaluation are needed to build a GCQM based on analytic knowledge and on previous evaluations, using the mechanism proposed here. The main inspiration source for this work is the FEMTI framework for the evaluation of machine translation, which implements partly the present model, and which is described briefly along with insights from other domains.
Keywords	Evaluation, Context-based evaluation, Evaluation tools, Machine translation evaluation
Full paper	A Model for Context-Based Evaluation of Language Processing Systems and its Application to Machine Translation Evaluation

Title

A Model for Context-Based Evaluation of Language Processing Systems and its Application to Machine Translation Evaluation

Authors

A. Popescu-belis, P. Estrella, M. King, N. Underwood

Abstract

In this paper, we propose a formal framework that takes into account the influence of the intended context of use of an NLP system on the procedure and the metrics used to evaluate the system. We introduce in particular the notion of a context-dependent quality model and explain how it can be adapted to a given context of use. More specifically, we define vector-space representations of contexts of use and of quality models, which are connected by a generic contextual quality model (GCQM). For each domain, experts in evaluation are needed to build a GCQM based on analytic knowledge and on previous evaluations, using the mechanism proposed here. The main inspiration source for this work is the FEMTI framework for the evaluation of machine translation, which implements partly the present model, and which is described briefly along with insights from other domains.

Keywords

Evaluation, Context-based evaluation, Evaluation tools, Machine translation evaluation

Full paper

A Model for Context-Based Evaluation of Language Processing Systems and its Application to Machine Translation Evaluation