LREC 2000 - Papers

LREC 2000 2^nd International Conference on Language Resources & Evaluation

Conference Papers

Papers by paper title: A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Papers by ID number: 1-50, 51-100, 101-150, 151-200, 201-250, 251-300, 301-350, 351-377.

List of all papers and abstracts.

Previous Paper Next Paper

Title Target Suites for Evaluating the Coverage of Text Generators

Authors Bateman John A. (Linguistics and Literature, University of Bremen, bateman@uni-bremen.de)
Hartley Anthony F. (Information Technology Research Institute, University of Brighton, UK, Tony.Hartley@itri.brighton.ac.uk)

Keywords Automatic annotation, Evaluation, Grammatical coverage, Natural Language Generation, NLG, Surface realization, Target suites, Test suites

Session Session EP1 - Evaluation and Written Area

Abstract Our goal is to evaluate the grammatical coverage of the surface realization component of a natural language generation system by means of target suites. We consider the utility of re-using for this purpose test suites designed to assess the coverage of natural language analysis / understanding systems. We find that they are of some interest, in helping inter-system comparisons and in providing an essential link to annotated corpora. But they have limitations. First, they contain a high proportion of ill-formed items which are inappropriate as targets for generation. Second, they omit phenomena such as discourse markers which are key issues in text production. We illustrate a partial remedy for this situation in the form of a text generator that annotates its own output to an externally specified standard, the TSNLP scheme.

ce="Verdana">

Title	Target Suites for Evaluating the Coverage of Text Generators
Authors	Bateman John A. (Linguistics and Literature, University of Bremen, bateman@uni-bremen.de) Hartley Anthony F. (Information Technology Research Institute, University of Brighton, UK, Tony.Hartley@itri.brighton.ac.uk)
Keywords	Automatic annotation, Evaluation, Grammatical coverage, Natural Language Generation, NLG, Surface realization, Target suites, Test suites
Session	Session EP1 - Evaluation and Written Area
Abstract	Our goal is to evaluate the grammatical coverage of the surface realization component of a natural language generation system by means of target suites. We consider the utility of re-using for this purpose test suites designed to assess the coverage of natural language analysis / understanding systems. We find that they are of some interest, in helping inter-system comparisons and in providing an essential link to annotated corpora. But they have limitations. First, they contain a high proportion of ill-formed items which are inappropriate as targets for generation. Second, they omit phenomena such as discourse markers which are key issues in text production. We illustrate a partial remedy for this situation in the form of a text generator that annotates its own output to an externally specified standard, the TSNLP scheme.