1 |
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT
|
|
|
|
In: https://hal.inria.fr/hal-03161685 ; 2021 (2021)
|
|
Abstract:
Accepted at EACL 2021 ; Multilingual pretrained language models have demonstrated remarkable zero-shot cross-lingual transfer capabilities. Such transfer emerges by fine-tuning on a task of interest in one language and evaluating on a distinct language, not seen during the fine-tuning. Despite promising results, we still lack a proper understanding of the source of this transfer. Using a novel layer ablation technique and analyses of the model's internal representations, we show that multilingual BERT, a popular multilingual language model, can be viewed as the stacking of two sub-networks: a multilingual encoder followed by a task-specific language-agnostic predictor. While the encoder is crucial for cross-lingual transfer and remains mostly unchanged during fine-tuning, the task predictor has little importance on the transfer and can be reinitialized during fine-tuning. We present extensive experiments with three distinct tasks, seventeen typologically diverse languages and multiple domains to support our hypothesis.
|
|
Keyword:
[INFO.INFO-TT]Computer Science [cs]/Document and Text Processing
|
|
URL: https://hal.inria.fr/hal-03161685
|
|
BASE
|
|
Hide details
|
|
2 |
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT
|
|
|
|
In: EACL 2021 - The 16th Conference of the European Chapter of the Association for Computational Linguistics ; https://hal.inria.fr/hal-03239087 ; EACL 2021 - The 16th Conference of the European Chapter of the Association for Computational Linguistics, Apr 2021, Kyiv / Virtual, Ukraine ; https://2021.eacl.org/ (2021)
|
|
BASE
|
|
Show details
|
|
3 |
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT ...
|
|
|
|
BASE
|
|
Show details
|
|
5 |
Measuring and Improving Consistency in Pretrained Language Models ...
|
|
|
|
BASE
|
|
Show details
|
|
6 |
Amnesic Probing: Behavioral Explanation With Amnesic Counterfactuals ...
|
|
|
|
BASE
|
|
Show details
|
|
7 |
It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT ...
|
|
|
|
BASE
|
|
Show details
|
|
8 |
The Extraordinary Failure of Complement Coercion Crowdsourcing ...
|
|
|
|
BASE
|
|
Show details
|
|
10 |
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals ...
|
|
|
|
BASE
|
|
Show details
|
|
11 |
Evaluating Models' Local Decision Boundaries via Contrast Sets ...
|
|
|
|
BASE
|
|
Show details
|
|
12 |
Unsupervised Distillation of Syntactic Information from Contextualized Word Representations ...
|
|
|
|
BASE
|
|
Show details
|
|
13 |
How Large Are Lions? Inducing Distributions over Quantitative Attributes ...
|
|
|
|
BASE
|
|
Show details
|
|
14 |
Where’s My Head? Definition, Data Set, and Models for Numeric Fused-Head Identification and Resolution
|
|
|
|
In: Transactions of the Association for Computational Linguistics, Vol 7, Pp 519-535 (2019) (2019)
|
|
BASE
|
|
Show details
|
|
|
|