| 2 |
Pushing the right buttons: adversarial evaluation of quality estimation
|
|
|
|
In: Proceedings of the Sixth Conference on Machine Translation ; 625 ; 638 (2022)
|
|
Abstract:
© (2021) The Authors. Published by Association for Computational Linguistics. This is an open access article available under a Creative Commons licence. The published version can be accessed at the following link on the publisher’s website: https://aclanthology.org/2021.wmt-1.67 ; Current Machine Translation (MT) systems achieve very good results on a growing variety of language pairs and datasets. However, they are known to produce fluent translation outputs that can contain important meaning errors, thus undermining their reliability in practice. Quality Estimation (QE) is the task of automatically assessing the performance of MT systems at test time. Thus, in order to be useful, QE systems should be able to detect such errors. However, this ability is yet to be tested in the current evaluation practices, where QE systems are assessed only in terms of their correlation with human judgements. In this work, we bridge this gap by proposing a general methodology for adversarial testing of QE for MT. First, we show that despite a high correlation with human judgements achieved by the recent SOTA, certain types of meaning errors are still problematic for QE to detect. Second, we show that on average, the ability of a given model to discriminate between meaningpreserving and meaning-altering perturbations is predictive of its overall performance, thus potentially allowing for comparing QE systems without relying on manual quality annotation.
|
|
Keyword:
adversarial evaluation; machine translation; quality estimation
|
|
URL: http://hdl.handle.net/2436/624376
|
|
BASE
|
|
Hide details
|
|
|
| 4 |
APE through neural and statistical MT with augmented data: ADAPT/DCU submission to the WMT 2019 APE Shared task
|
|
|
|
In: Shterionov, Dimitar orcid:0000-0001-6300-797X , Wagner, Joachim orcid:0000-0002-8290-3849 and do Carmo, Félix orcid:0000-0003-4193-3854 (2019) APE through neural and statistical MT with augmented data: ADAPT/DCU submission to the WMT 2019 APE Shared task. In: Fourth Conference on Machine Translation (WMT19), 01-02 Aug 2019, Florence, Italy. (2019)
|
|
BASE
|
|
Show details
|
|
|
| 5 |
Findings of the 2019 Conference on Machine Translation (WMT19)
|
|
|
|
In: Barrault, Loïc orcid:0000-0002-0634-6147 , Bojar, Ondřej orcid:0000-0002-0606-0050 , Costa-Jussà, Marta R. orcid:0000-0002-5703-520X , Federmann, Christian, Fishel, Mark and Graham, Yvette (2019) Findings of the 2019 Conference on Machine Translation (WMT19). In: Fourth Conference on Machine Translation, 1-2 Aug 2019, Florence, Italy. (2019)
|
|
BASE
|
|
Show details
|
|
|
| 7 |
Using domain-specific and collaborative resources for term translation
|
|
|
|
BASE
|
|
Show details
|
|
|
| 8 |
Findings of the 2018 Conference on Machine Translation (WMT18)
|
|
|
|
In: Bojar, Ondřej orcid:0000-0002-0606-0050 , Federmann, Christian, Fishel, Mark, Graham, Yvette and Haddow, Barry (2018) Findings of the 2018 Conference on Machine Translation (WMT18). In: Third Conference on Machine Translation, 31 Oct- 1 Nov 2018, Brussels, Belgium. (2018)
|
|
BASE
|
|
Show details
|
|
|
| 9 |
Findings of the 2018 conference on machine translation (WMT18)
|
|
|
|
In: Bojar, Ondřej orcid:0000-0002-0606-0050 , Federmann, Christian, Fishel, Mark, Graham, Yvette, Haddow, Barry, Huck, Matthias, Koehn, Philipp and Monz, Christof (2018) Findings of the 2018 conference on machine translation (WMT18). In: Third Conference on Machine Translation, Volume 2: Shared Task Papers, 31 Oct - 1 Nov 2018, Brussels, Belgium. (2018)
|
|
BASE
|
|
Show details
|
|
|
| 10 |
Findings of the 2017 conference on machine translation (WMT17)
|
|
|
|
In: Bojar, Ondřej orcid:0000-0002-0606-0050 , Chatterjee, Rajen, Federmann, Christian, Graham, Yvette, Haddow, Barry, Huang, Shujian, Huck, Matthias, Koehn, Philipp, Liu, Qun orcid:0000-0002-7000-1792 , Logacheva, Varvara, Monz, Christof, Negri, Matteo, Post, Matt, Rubino, Raphael, Specia, Lucia and Turchi, Marco (2017) Findings of the 2017 conference on machine translation (WMT17). In: Second Conference on Machine Translation (WMT17), 7-11 Sept 2017, Copenhagen, Denmark. ISBN 978-1-945626-96-8 (2017)
|
|
BASE
|
|
Show details
|
|
|
|
|
