1.4 KiB
1.4 KiB
- The Bilingual Evaluation Understudy Score, or BLEU for short, is a metric for evaluating a generated sentence to a reference sentence.
- A perfect match results in a score of 1.0, whereas a perfect mismatch results in a score of 0.0.
- The score was developed to evaluate the prediction made by automatic machine translation systems. Even though it is not perfect, it offers 5 compelling benefits:
- quick and expensive to calculate
- easy to understand
- language independent
- it correlates highly with human evaluation
- it has been widely adopted
- The BLEU score was proposed by Kishore Papineni, et. al. in their 2002 paper BLEU: a Method for Automatic Evaluation of Machine Translation“.
- The approach works by counting matching n-grams in the candidate translation to n-grams in the reference text, where 1-gram or unigram would be each token and a bigram comparison would be each word pair. The comparison is made regardless of word order.
The primary programming task for a BLEU implementor is to compare n-grams of the candidate with the n-grams of the reference translation and count the number of matches. These matches are position-independent. The more the matches, the better the candidate translation is.
- ((63063d27-0931-46bb-89ba-6436d127cdbf))
- ((63063df8-a6dc-4bee-96b2-2512719da00b)) #card id:: 65c8d455-5ace-4f50-a571-e08b0196a72d