LREC 2000 2nd International Conference on Language Resources & Evaluation

Previous Paper   Next Paper

Title Target Suites for Evaluating the Coverage of Text Generators
Authors Bateman John A. (Linguistics and Literature, University of Bremen,
Hartley Anthony F. (Information Technology Research Institute, University of Brighton, UK,
Keywords Automatic annotation, Evaluation, Grammatical coverage, Natural Language Generation, NLG, Surface realization, Target suites, Test suites
Session Session EP1 - Evaluation and Written Area
Full Paper, 92.pdf
Abstract Our goal is to evaluate the grammatical coverage of the surface realization component of a natural language generation system by means of target suites. We consider the utility of re-using for this purpose test suites designed to assess the coverage of natural language analysis / understanding systems. We find that they are of some interest, in helping inter-system comparisons and in providing an essential link to annotated corpora. But they have limitations. First, they contain a high proportion of ill-formed items which are inappropriate as targets for generation. Second, they omit phenomena such as discourse markers which are key issues in text production. We illustrate a partial remedy for this situation in the form of a text generator that annotates its own output to an externally specified standard, the TSNLP scheme.