| 1 |
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation
|
|
|
|
In: Zhang, Biao; Bapna, Ankur; Sennrich, Rico; Firat, Orhan (2021). Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation. In: International Conference on Learning Representations, Virtual, 3 May 2021 - 7 May 2021, ICLR. (2021)
|
|
Abstract:
Using a mix of shared and language-specific (LS) parameters has shown promise in multilingual neural machine translation (MNMT), but the question of when and where LS capacity matters most is still under-studied. We offer such a study by proposing conditional language-specific routing (CLSR). CLSR employs hard binary gates conditioned on token representations to dynamically select LS or shared paths. By manipulating these gates, it can schedule LS capacity across sub-layers in MNMT subject to the guidance of translation signals and budget constraints. Moreover, CLSR can easily scale up to massively multilingual settings. Experiments with Transformer on OPUS-100 and WMT datasets show that: 1) MNMT is sensitive to both the amount and the position of LS modeling: distributing 10%-30% LS computation to the top and/or bottom encoder/decoder layers delivers the best performance; and 2) one-to-many translation benefits more from CLSR compared to many-to-one translation, particularly with unbalanced training data. Our study further verifies the trade-off between the shared capacity and LS capacity for multilingual translation. We corroborate our analysis by confirming the soundness of our findings as foundation of our improved multilingual Transformers. Source code and models are available at https://github.com/bzhangGo/zero/tree/iclr2021_clsr. Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics One-sentence Summary: We investigate and improve parameter-sharing strategies in multilingual Transformers by utilizing conditional computation.
|
|
Keyword:
000 Computer science; 410 Linguistics; Institute of Computational Linguistics; knowledge & systems
|
|
URL: https://doi.org/10.5167/uzh-208876
https://openreview.net/forum?id=Wj4ODo0uyCF
https://www.zora.uzh.ch/id/eprint/208876/
https://www.zora.uzh.ch/id/eprint/208876/1/share_or_not_learning_to_sched.pdf
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
On Biasing Transformer Attention Towards Monotonicity
|
|
|
|
In: Rios, Annette; Amrhein, Chantal; Aepli, Noëmi; Sennrich, Rico (2021). On Biasing Transformer Attention Towards Monotonicity. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, 6 June 2021 - 11 June 2021. Association for Computational Linguistics, 4474-4488. (2021)
|
|
BASE
|
|
Show details
|
|
|
| 3 |
Negation typology and general representation models for cross-lingual zero-shot negation scope resolution in Russian, French, and Spanish
|
|
|
|
In: Shaitarova, Anastassia; Rinaldi, Fabio (2021). Negation typology and general representation models for cross-lingual zero-shot negation scope resolution in Russian, French, and Spanish. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, Online, 6 June 2021 - 11 June 2021, ACL Anthology. (2021)
|
|
BASE
|
|
Show details
|
|
|
| 4 |
Progressive Transformer-Based Generation of Radiology Reports
|
|
|
|
In: Nooralahzadeh, Farhad; Perez Gonzalez, Nicolas; Frauenfelder, Thomas; Fujimoto, Koji; Krauthammer, Michael (2021). Progressive Transformer-Based Generation of Radiology Reports. In: Empirical Methods in Natural Language Processing (EMNLP), Punta Cana, 7 November 2021 - 11 November 2021. ACL Anthology, 2824-2832. (2021)
|
|
BASE
|
|
Show details
|
|
|
| 5 |
Multi-Level Modelling for Upstream Text Processing
|
|
|
|
In: Ruzsics, Tatiana. Multi-Level Modelling for Upstream Text Processing. 2021, University of Zurich, Faculty of Arts. (2021)
|
|
BASE
|
|
Show details
|
|
|
| 6 |
Evaluation of Dialogue Systems
|
|
|
|
In: Deriu, Jan Milan. Evaluation of Dialogue Systems. 2021, University of Zurich, Faculty of Arts. (2021)
|
|
BASE
|
|
Show details
|
|
|
| 7 |
Biomedical Text Mining for Etiological Factor Identification in Mental Health Publications
|
|
|
|
In: Ellendorff, Tilia. Biomedical Text Mining for Etiological Factor Identification in Mental Health Publications. 2021, University of Zurich, Faculty of Arts. (2021)
|
|
BASE
|
|
Show details
|
|
|
| 8 |
Robust Neural Machine Translation Systems
|
|
|
|
In: Müller, Mathias. Robust Neural Machine Translation Systems. 2021, University of Zurich, Faculty of Arts. (2021)
|
|
BASE
|
|
Show details
|
|
|
| 9 |
Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas
|
|
In: Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas. Edited by: Mager, Manuel; Oncevay, Arturo; Rios, Annette; Meza Ruiz, Ivan Vladimir; Palmer, Alexis; Neubig, Graham; Kann, Katharina (2021). Online: Association for Computational Linguistics. (2021)
|
|
BASE
|
|
Show details
|
|
|
| 10 |
Exploring the Importance of Source Text in Automatic Post-Editing for Context-Aware Machine Translation
|
|
|
|
In: Wang, Chaojun; Hardmeier, Christian; Sennrich, Rico (2021). Exploring the Importance of Source Text in Automatic Post-Editing for Context-Aware Machine Translation. In: Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), Reykjavik, Iceland (Online), 31 May 2021 - 2 June 2021. ACL Anthology, 326-335. (2021)
|
|
BASE
|
|
Show details
|
|
|
| 11 |
Exploring German Multi-Level Text Simplification
|
|
|
|
In: Spring, Nicolas; Rios, Annette; Ebling, Sarah (2021). Exploring German Multi-Level Text Simplification. In: International Conference on Recent Advances in Natural Language Processing (RANLP 2021), online, 1 September 2021 - 3 September 2021. ACL Anthology, 1339-1349. (2021)
|
|
BASE
|
|
Show details
|
|
|
| 12 |
Benchmarking Data-driven Automatic Text Simplification for German
|
|
|
|
In: Säuberli, Andreas; Ebling, Sarah; Volk, Martin (2020). Benchmarking Data-driven Automatic Text Simplification for German. In: Gala, Nuria; Wilkens, Rodrigo. Proceedings of the 1st Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI). Marseille: European Language Resources Association, 41-48. (2020)
|
|
BASE
|
|
Show details
|
|
|
| 13 |
Using Lexical-Semantic Concepts for Fine-Grained Classification in the Embedding Space
|
|
|
|
In: Amsler, Michael. Using Lexical-Semantic Concepts for Fine-Grained Classification in the Embedding Space. 2020, University of Zurich, Faculty of Arts. (2020)
|
|
BASE
|
|
Show details
|
|
|
| 14 |
Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English
|
|
|
|
In: Tang, Gongbo; Sennrich, Rico; Nivre, Joakim (2020). Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English. In: Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, 8 December 2020 - 13 December 2020, 4251-4262. (2020)
|
|
BASE
|
|
Show details
|
|
|
| 15 |
Vowel and consonant length in four Alemannic dialects and their influence on the respective varieties of Swiss Standard German
|
|
|
|
In: Zihlmann, Urban (2020). Vowel and consonant length in four Alemannic dialects and their influence on the respective varieties of Swiss Standard German. Wiener Linguistische Gazette, 86:1-46. (2020)
|
|
BASE
|
|
Show details
|
|
|
| 16 |
Real-Time Sign Language Detection using Human Pose Estimation
|
|
|
|
In: Moryossef, Amit; Tsochantaridis, Ioannis; Aharoni, Roee; Ebling, Sarah; Narayanan, Srini (2020). Real-Time Sign Language Detection using Human Pose Estimation. arXiv.org 04637, University of Zurich. (2020)
|
|
BASE
|
|
Show details
|
|
|
| 17 |
Domain robustness in neural machine translation
|
|
|
|
In: Müller, Mathias; Rios, Annette; Sennrich, Rico (2020). Domain robustness in neural machine translation. In: 14th Conference of the Association for Machine Translation in the Americas (AMTA 2020), Virtual, 6 October 2020 - 9 October 2020. Association for Machine Translation in the Americas, 151-164. (2020)
|
|
BASE
|
|
Show details
|
|
|
| 18 |
Characterizing speech rhythm using spectral coherence between jaw displacement and speech temporal envelope
|
|
|
|
In: He, Lei; Zhang, Yu (2020). Characterizing speech rhythm using spectral coherence between jaw displacement and speech temporal envelope. Loquens, 7(2):e074. (2020)
|
|
BASE
|
|
Show details
|
|
|
| 19 |
In Neural Machine Translation, What Does Transfer Learning Transfer?
|
|
|
|
In: Aji, Alham Fikri; Bogoychev, Nikolay; Heafield, Kenneth; Sennrich, Rico (2020). In Neural Machine Translation, What Does Transfer Learning Transfer? In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 5 July 2020 - 10 July 2020, 7701-7710. (2020)
|
|
BASE
|
|
Show details
|
|
|
| 20 |
Benchmarking Automated Review Response Generation for the Hospitality Domain
|
|
|
|
In: Kew, Tannon; Amsler, Michael; Ebling, Sarah (2020). Benchmarking Automated Review Response Generation for the Hospitality Domain. In: Workshop on Natural Language Processing in E-Commerce, Barcelona, Spain, 12 December 2020. Association for Computational Linguistics, 43-52. (2020)
|
|
BASE
|
|
Show details
|
|
|
|
|
