1 |
Entropy Analysis of Heart Rate Variability in Different Sleep Stages
|
|
|
|
In: Entropy (Basel) (2022)
|
|
BASE
|
|
Show details
|
|
2 |
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts ...
|
|
|
|
Abstract:
Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear. In this work, we study the computational patterns of FFNs and observe that most inputs only activate a tiny ratio of neurons of FFNs. This phenomenon is similar to the sparsity of the human brain, which drives research on functional partitions of the human brain. To verify whether functional partitions also emerge in FFNs, we propose to convert a model into its MoE version with the same parameters, namely MoEfication. Specifically, MoEfication consists of two phases: (1) splitting the parameters of FFNs into multiple functional partitions as experts, and (2) building expert routers to decide which experts will be used for each input. Experimental results show that MoEfication can conditionally use 10% to 30% of FFN parameters while maintaining over 95% original performance for different models ... : Accepted to ACL Findings 2022 ...
|
|
Keyword:
Computation and Language cs.CL; FOS Computer and information sciences
|
|
URL: https://dx.doi.org/10.48550/arxiv.2110.01786 https://arxiv.org/abs/2110.01786
|
|
BASE
|
|
Hide details
|
|
4 |
Rethinking Stealthiness of Backdoor Attack against NLP Models ...
|
|
|
|
BASE
|
|
Show details
|
|
5 |
RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models ...
|
|
|
|
BASE
|
|
Show details
|
|
6 |
Serial magnetic resonance imaging changes of pseudotumor lesions in retinal vasculopathy with cerebral leukoencephalopathy and systemic manifestations: a case report
|
|
|
|
In: BMC Neurol (2021)
|
|
BASE
|
|
Show details
|
|
7 |
Ecoacoustics and multispecies semiosis: naming, semantics, semiotic characteristics, and competencies
|
|
|
|
BASE
|
|
Show details
|
|
8 |
Hand gestures facilitate novel segment learning (Xi et al., 2020) ...
|
|
|
|
BASE
|
|
Show details
|
|
9 |
Hand gestures facilitate novel segment learning (Xi et al., 2020) ...
|
|
|
|
BASE
|
|
Show details
|
|
10 |
Evaluation of poverty-stricken families in rural areas using a novel casebased reasoning method for probabilistic linguistic term sets
|
|
|
|
BASE
|
|
Show details
|
|
11 |
Laryngeal Diffuse Large B-Cell Lymphoma Presenting as Laryngeal Stenosis
|
|
|
|
BASE
|
|
Show details
|
|
12 |
Bivariate Entropy Analysis of Electrocardiographic RR–QT Time Series
|
|
|
|
In: Entropy (Basel) (2020)
|
|
BASE
|
|
Show details
|
|
13 |
Differences in Working Memory With Emotional Distraction Between Proficient and Non-proficient Bilinguals
|
|
|
|
In: Front Psychol (2020)
|
|
BASE
|
|
Show details
|
|
15 |
Relation Between Working Memory Capacity of Biological Movements and Fluid Intelligence
|
|
|
|
BASE
|
|
Show details
|
|
17 |
Detection of epileptic seizure based on entropy analysis of short-term EEG
|
|
|
|
BASE
|
|
Show details
|
|
18 |
Detection of epileptic seizure based on entropy analysis of short-term EEG
|
|
|
|
BASE
|
|
Show details
|
|
20 |
The Fable of Recognition: A Study of Northrop Frye as a Prophet
|
|
|
|
In: English Language Teaching; Vol 4, No 3 (2011); p54 (2011)
|
|
BASE
|
|
Show details
|
|
|
|