LREC 2000 2nd International Conference on Language Resources & Evaluation

Previous Paper   Next Paper

Title A Parallel English-Japanese Query Collection for the Evaluation of On-Line Help Systems
Authors Sutcliffe Richard F.E. (Department of Computer Science and Information Systems, University of Limerick, Limerick, Ireland,
Kurohashi Sadao (Graduate School of Informatics, Kyoto University, Yoshida-honmachi, Sakyo, Kyoto 606-8501, Japan,
Keywords Parallel Multilingual Evaluation Materials
Session Session EP1 - Evaluation and Written Area
Full Paper, 248.pdf
Abstract An experiment concerning the creation of parallel evaluation data for information retrieval is presented. A set of English queries was gathered for the domain of wordprocessing using Lotus Ami Pro. A set of Japanese queries was then created from these. The answers to the queries were elicited from eight respondents comprising four native speakers of each language. We first describe how the queries were created and the answers elicited. We then present analyses of the responses in each language. The results show a lower level of agreeement between respondents than was expected. We discuss a refinement of the elicitation process which is designed to address this problem as well as measuring the integrity of individual respondents.