It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information
Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt
Standard
It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information. / Bugliarello, Emanuele; Mielke, Sabrina J.; Anastasopoulos, Antonios; Cotterell, Ryan; Okazaki, Naoaki.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online : Association for Computational Linguistics (ACL), 2020. s. 1640-1649.Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - GEN
T1 - It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information
AU - Bugliarello, Emanuele
AU - Mielke, Sabrina J.
AU - Anastasopoulos, Antonios
AU - Cotterell, Ryan
AU - Okazaki, Naoaki
PY - 2020/7/1
Y1 - 2020/7/1
N2 - The performance of neural machine translation systems is commonly evaluated in terms of BLEU. However, due to its reliance on target language properties and generation, the BLEU metric does not allow an assessment of which translation directions are more difficult to model. In this paper, we propose cross-mutual information (XMI): an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. XMI allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty of the target-side generation component independent of the translation task. We then present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems. Code for replicating our experiments is available online at https://github.com/e-bug/nmt-difficulty.
AB - The performance of neural machine translation systems is commonly evaluated in terms of BLEU. However, due to its reliance on target language properties and generation, the BLEU metric does not allow an assessment of which translation directions are more difficult to model. In this paper, we propose cross-mutual information (XMI): an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. XMI allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty of the target-side generation component independent of the translation task. We then present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems. Code for replicating our experiments is available online at https://github.com/e-bug/nmt-difficulty.
U2 - 10.18653/v1/2020.acl-main.149
DO - 10.18653/v1/2020.acl-main.149
M3 - Article in proceedings
SP - 1640
EP - 1649
BT - Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
PB - Association for Computational Linguistics (ACL)
CY - Online
T2 - 58th Annual Meeting of the Association for Computational Linguistics
Y2 - 5 July 2020 through 10 July 2020
ER -
ID: 255126547