1 |
Fairlex: A multilingual benchmark for evaluating fairness in legal text processing ...
|
|
|
|
Abstract:
We present a benchmark suite of four datasets for evaluating the fairness of pre-trained legal language models and the techniques used to fine-tune them for downstream tasks. Our benchmarks cover four jurisdictions (European Council, USA, Swiss, and Chinese), five languages (English, German, French, Italian, and Chinese), and fairness across five attributes (gender, age, nationality/region, language, and legal area). In our experiments, we evaluate pre-trained language models using several group-robust fine-tuning techniques and show that performance group disparities are vibrant in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparities. Furthermore, we provide a quantitative and qualitative analysis of our results, highlighting open challenges in the development of robustness methods in legal NLP. ...
|
|
Keyword:
fairlex; fairness; legal
|
|
URL: https://dx.doi.org/10.5281/zenodo.6322643 https://zenodo.org/record/6322643
|
|
BASE
|
|
Hide details
|
|
2 |
Fairlex: A multilingual benchmark for evaluating fairness in legal text processing ...
|
|
|
|
BASE
|
|
Show details
|
|
5 |
FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing ...
|
|
|
|
BASE
|
|
Show details
|
|
6 |
Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks ...
|
|
|
|
BASE
|
|
Show details
|
|
8 |
Factual Consistency of Multilingual Pretrained Language Models ...
|
|
|
|
BASE
|
|
Show details
|
|
9 |
Zero-Shot Dependency Parsing with Worst-Case Aware Automated Curriculum Learning ...
|
|
|
|
BASE
|
|
Show details
|
|
10 |
How Conservative are Language Models? Adapting to the Introduction of Gender-Neutral Pronouns ...
|
|
|
|
BASE
|
|
Show details
|
|
11 |
Replicating and Extending "Because Their Treebanks Leak": Graph Isomorphism, Covariants, and Parser Performance ...
|
|
|
|
BASE
|
|
Show details
|
|
12 |
The Impact of Positional Encodings on Multilingual Compression ...
|
|
|
|
BASE
|
|
Show details
|
|
13 |
Minimax and Neyman–Pearson Meta-Learning for Outlier Languages ...
|
|
|
|
BASE
|
|
Show details
|
|
14 |
Evaluation of Summarization Systems across Gender, Age, and Race ...
|
|
|
|
BASE
|
|
Show details
|
|
17 |
Replicating and Extending ``Because Their Treebanks Leak'': Graph Isomorphism, Covariants, and Parser Performance ...
|
|
|
|
BASE
|
|
Show details
|
|
18 |
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color ...
|
|
|
|
BASE
|
|
Show details
|
|
20 |
Minimax and Neyman–Pearson Meta-Learning for Outlier Languages ...
|
|
|
|
BASE
|
|
Show details
|
|
|
|