| 3 |
On the differences of Bert and MT encoder spaces and how to address them in translation tasks ...
|
|
|
|
BASE
|
|
Show details
|
|
|
| 4 |
An Empirical Investigation of Word Alignment Supervision for Zero-Shot Multilingual Neural Machine Translation ...
|
|
|
|
BASE
|
|
Show details
|
|
|
| 5 |
A Systematic Study of Inner-Attention-Based Sentence Representations in Multilingual Neural Machine Translation
|
|
|
|
In: Computational Linguistics, Vol 46, Iss 2, Pp 387-424 (2020) (2020)
|
|
Abstract:
Neural machine translation has considerably improved the quality of automatic translations by learning good representations of input sentences. In this article, we explore a multilingual translation model capable of producing fixed-size sentence representations by incorporating an intermediate crosslingual shared layer, which we refer to as attention bridge. This layer exploits the semantics from each language and develops into a language-agnostic meaning representation that can be efficiently used for transfer learning. We systematically study the impact of the size of the attention bridge and the effect of including additional languages in the model. In contrast to related previous work, we demonstrate that there is no conflict between translation performance and the use of sentence representations in downstream tasks. In particular, we show that larger intermediate layers not only improve translation quality, especially for long sentences, but also push the accuracy of trainable classification tasks. Nevertheless, shorter representations lead to increased compression that is beneficial in non-trainable similarity tasks. Similarly, we show that trainable downstream tasks benefit from multilingual models, whereas additional language signals do not improve performance in non-trainable benchmarks. This is an important insight that helps to properly design models for specific applications. Finally, we also include an in-depth analysis of the proposed attention bridge and its ability to encode linguistic properties. We carefully analyze the information that is captured by individual attention heads and identify interesting patterns that explain the performance of specific settings in linguistic probing tasks.
|
|
Keyword:
Computational linguistics. Natural language processing; P98-98.5
|
|
URL: https://doaj.org/article/e9259d25119a4e979698d24f54c4a5f0
https://doi.org/10.1162/coli_a_00377
|
|
BASE
|
|
Hide details
|
|
|
| 6 |
Multilingual NMT with a language-independent attention bridge ...
|
|
|
|
BASE
|
|
Show details
|
|
|
|
|
