1 |
Towards Best Practices for Training Multilingual Dense Retrieval Models ...
|
|
|
|
BASE
|
|
Show details
|
|
3 |
Sparsifying Sparse Representations for Passage Retrieval by Top-$k$ Masking ...
|
|
|
|
BASE
|
|
Show details
|
|
5 |
Aligning Cross-Lingual Entities with Multi-Aspect Information ...
|
|
|
|
BASE
|
|
Show details
|
|
6 |
End-to-End Open-Domain Question Answering with BERTserini ...
|
|
|
|
BASE
|
|
Show details
|
|
7 |
What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning ...
|
|
|
|
Abstract:
Pretrained transformer-based language models have achieved state of the art across countless tasks in natural language processing. These models are highly expressive, comprising at least a hundred million parameters and a dozen layers. Recent evidence suggests that only a few of the final layers need to be fine-tuned for high quality on downstream tasks. Naturally, a subsequent research question is, "how many of the last layers do we need to fine-tune?" In this paper, we precisely answer this question. We examine two recent pretrained language models, BERT and RoBERTa, across standard tasks in textual entailment, semantic similarity, sentiment analysis, and linguistic acceptability. We vary the number of final layers that are fine-tuned, then study the resulting change in task-specific effectiveness. We show that only a fourth of the final layers need to be fine-tuned to achieve 90% of the original quality. Surprisingly, we also find that fine-tuning all layers does not always help. ... : 5 pages ...
|
|
Keyword:
Computation and Language cs.CL; FOS Computer and information sciences
|
|
URL: https://arxiv.org/abs/1911.03090 https://dx.doi.org/10.48550/arxiv.1911.03090
|
|
BASE
|
|
Hide details
|
|
8 |
Simple BERT Models for Relation Extraction and Semantic Role Labeling ...
|
|
|
|
BASE
|
|
Show details
|
|
9 |
Lucene for Approximate Nearest-Neighbors Search on Arbitrary Dense Vectors ...
|
|
|
|
BASE
|
|
Show details
|
|
10 |
Simple Attention-Based Representation Learning for Ranking Short Social Media Posts ...
|
|
|
|
BASE
|
|
Show details
|
|
11 |
ARCHITECTURE, MODELS, AND ALGORITHMS FOR TEXTUAL SIMILARITY
|
|
|
|
BASE
|
|
Show details
|
|
12 |
Integrating Lexical and Temporal Signals in Neural Ranking Models for Searching Social Media Streams ...
|
|
|
|
BASE
|
|
Show details
|
|
13 |
Searching to Translate and Translating to Search: When Information Retrieval Meets Machine Translation
|
|
|
|
BASE
|
|
Show details
|
|
14 |
Frontiers, Challenges, and Opportunities for Information Retrieval – Report from SWIRL 2012, The Second Strategic Workshop on Information Retrieval in Lorne
|
|
|
|
BASE
|
|
Show details
|
|
16 |
COMPUTATIONAL ANALYSIS OF THE CONVERSATIONAL DYNAMICS OF THE UNITED STATES SUPREME COURT
|
|
|
|
BASE
|
|
Show details
|
|
20 |
Multiple Alternative Sentene Compressions as a Tool for Automatic Summarization Tasks
|
|
|
|
BASE
|
|
Show details
|
|
|
|