Summary of the paper

Title BLEU+: a Tool for Fine-Grained BLEU Computation
Authors A. Cuneyd Tantug, Kemal Oflazer and Ilknur Durgar El-Kahlout
Abstract We present a tool, BLEU+, which implements various extension to BLEU computation to allow for a better understanding of the translation performance, especially for morphologically complex languages. BLEU+ takes into account both “closeness” in morphological structure, “closeness” of the root words in the WordNet hierarchy while comparing tokens in the candidate and reference sentence. In addition to gauging performance at a finer level of granularity, BLEU+ also allows the computation of various upper bound oracle scores: comparing all tokens considering only the roots allows us to get an upper bound when all errors due to morphological structure are fixed, while comparing tokens in an error-tolerant way considering minor morpheme edit operations, allows us to get a (more realistic) upper bound when tokens that differ in morpheme insertions/deletions and substitutions are fixed. We use BLEU+ in the fine-grained evaluation of the output of our English-to-Turkish statistical MT system.
Language Language-independent
Topics Evaluation methodologies, Tools, systems, applications
Full paper BLEU+: a Tool for Fine-Grained BLEU Computation
Slides -
Bibtex @InProceedings{TANTUG08.382,
  author = {A. Cuneyd Tantug, Kemal Oflazer and Ilknur Durgar El-Kahlout},
  title = {BLEU+: a Tool for Fine-Grained BLEU Computation},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  month = {may},
  date = {28-30},
  address = {Marrakech, Morocco},
  editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Daniel Tapias},
  publisher = {European Language Resources Association (ELRA)},
  isbn = {2-9517408-4-0},
  note = {},
  language = {english}

Powered by ELDA © 2008 ELDA/ELRA