4 |
Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications ...
|
|
|
|
Abstract:
Sentence-level Quality estimation (QE) of machine translation is traditionally formulated as a regression task, and the performance of QE models is typically measured by Pearson correlation with human labels. Recent QE models have achieved previously-unseen levels of correlation with human judgments, but they rely on large multilingual contextualized language models that are computationally expensive and make them infeasible for real-world applications. In this work, we evaluate several model compression techniques for QE and find that, despite their popularity in other NLP tasks, they lead to poor performance in this regression setting. We observe that a full model parameterization is required to achieve SoTA results in a regression task. However, we argue that the level of expressiveness of a model in a continuous range is unnecessary given the downstream applications of QE, and show that reframing QE as a classification problem and evaluating QE models using classification metrics would better reflect ... : EMNLP 2021 ...
|
|
Keyword:
Computation and Language cs.CL; FOS Computer and information sciences
|
|
URL: https://arxiv.org/abs/2109.08627 https://dx.doi.org/10.48550/arxiv.2109.08627
|
|
BASE
|
|
Hide details
|
|
5 |
A Generative Framework for Simultaneous Machine Translation ...
|
|
|
|
BASE
|
|
Show details
|
|
6 |
Continual Quality Estimation with Online Bayesian Meta-Learning ...
|
|
|
|
BASE
|
|
Show details
|
|
7 |
SentSim: Crosslingual Semantic Evaluation of Machine Translation ...
|
|
|
|
BASE
|
|
Show details
|
|
10 |
What Makes a Scientific Paper be Accepted for Publication? ...
|
|
|
|
BASE
|
|
Show details
|
|
11 |
MultiSubs: A Large-scale Multimodal and Multilingual Dataset ...
|
|
|
|
BASE
|
|
Show details
|
|
12 |
Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP
|
|
|
|
In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (2021)
|
|
BASE
|
|
Show details
|
|
13 |
Classifying Dyads for Militarized Conflict Analysis
|
|
|
|
In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (2021)
|
|
BASE
|
|
Show details
|
|
14 |
Efficient Sampling of Dependency Structure
|
|
|
|
In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (2021)
|
|
BASE
|
|
Show details
|
|
15 |
Searching for More Efficient Dynamic Programs
|
|
|
|
In: Findings of the Association for Computational Linguistics: EMNLP 2021 (2021)
|
|
BASE
|
|
Show details
|
|
16 |
“Let Your Characters Tell Their Story”: A Dataset for Character-Centric Narrative Understanding
|
|
|
|
In: Findings of the Association for Computational Linguistics: EMNLP 2021 (2021)
|
|
BASE
|
|
Show details
|
|
17 |
A Bayesian Framework for Information-Theoretic Probing
|
|
|
|
In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (2021)
|
|
BASE
|
|
Show details
|
|
18 |
Findings of the WMT 2021 Shared Task on Quality Estimation ...
|
|
|
|
BASE
|
|
Show details
|
|
19 |
Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation ...
|
|
|
|
BASE
|
|
Show details
|
|
20 |
The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification ...
|
|
|
|
BASE
|
|
Show details
|
|
|
|