| 1 |
A New Generation of Perspective API: Efficient Multilingual Character-level Transformers ...
|
|
|
|
BASE
|
|
Show details
|
|
|
| 2 |
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization ...
|
|
Tay, Yi; Tran, Vinh Q.; Ruder, Sebastian; Gupta, Jai; Chung, Hyung Won; Bahri, Dara; Qin, Zhen; Baumgartner, Simon; Yu, Cong; Metzler, Donald. - : arXiv, 2021
|
|
Abstract:
State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings. In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. To this end, we introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level. Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, ... : ICLR 2022 Camera Ready ...
|
|
Keyword:
Artificial Intelligence cs.AI; Computation and Language cs.CL; FOS Computer and information sciences; Machine Learning cs.LG
|
|
URL: https://dx.doi.org/10.48550/arxiv.2106.12672
https://arxiv.org/abs/2106.12672
|
|
BASE
|
|
Hide details
|
|
|
| 3 |
Are Pretrained Convolutions Better than Pretrained Transformers? ...
|
|
|
|
BASE
|
|
Show details
|
|
|
| 4 |
StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling ...
|
|
|
|
BASE
|
|
Show details
|
|
|
|
|
