LREC 2000 2nd International Conference on Language Resources & Evaluation

Previous Paper   Next Paper

Title Evaluation for Darpa Communicator Spoken Dialogue Systems
Authors Walker Marilyn (AT& T Labs - Research, 180 Park Ave, Florham Park, N.J. 07932, U.S.A.,
Hirschman Lynette (The MITRE Corporation, 202 Burlington Road;Bedford,MA 01730 USA,
Aberdeen John (The MITRE Corporation, 202 Burlington Rd., Bedford, MA 01730, U.S.A.,
Session Session SO5 - Evaluation of Dialogue
Full Paper, 191.pdf
Abstract The overall objective of the DARPA COMMUNICATOR project is to support rapid, cost-effective development of multi-modal speech-enabled dialogue systems with advanced conversational capabilities, such as plan optimization, explanation and negotiation. In order to make this a reality, we need to find methods for evaluating the contribution of various techniques to the users’ willingness and ability to use the system. This paper reports on the approach to spoken dialogue system evaluation that we are applying in the COMMUNICATOR program. We describe our overall approach, the experimental design, the logfile standard, and the metrics applied in the experimental evaluation planned for June of 2000.