|LREC 2000 2nd International Conference on Language Resources & Evaluation
Previous Paper Next Paper
|Target Suites for Evaluating the Coverage of Text Generators
|Bateman John A. (Linguistics and Literature, University of Bremen, firstname.lastname@example.org)
Hartley Anthony F. (Information Technology Research Institute, University of Brighton, UK, Tony.Hartley@itri.brighton.ac.uk)
|Automatic annotation, Evaluation, Grammatical coverage, Natural Language Generation, NLG, Surface realization, Target suites, Test suites
|Session EP1 - Evaluation and Written Area
|Our goal is to evaluate the grammatical coverage of the surface realization component of a natural language generation system by means of target suites. We consider the utility of re-using for this purpose test suites designed to assess the coverage of natural language analysis / understanding systems. We find that they are of some interest, in helping inter-system comparisons and in providing an essential link to annotated corpora. But they have limitations. First, they contain a high proportion of ill-formed items which are inappropriate as targets for generation. Second, they omit phenomena such as discourse markers which are key issues in text production. We illustrate a partial remedy for this situation in the form of a text generator that annotates its own output to an externally specified standard, the TSNLP scheme.