| 1 |
Differential Evaluation: a Qualitative Analysis of Natural Language Processing System Behavior Based Upon Data Resistance to Processing
|
|
|
|
In: Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems ; EVAL4NLP, 2nd Workshop on "Evaluation & Comparison of NLP Systems", EMNLP 2021 ; https://hal.archives-ouvertes.fr/hal-03432331 ; EVAL4NLP, 2nd Workshop on "Evaluation & Comparison of NLP Systems", EMNLP 2021, Nov 2021, Punta Cana, Dominican Republic (2021)
|
|
Abstract:
International audience ; Most of the time, when dealing with a particular Natural Language Processing task, systems are compared on the basis of global statistics such as recall, precision, F1-score, etc. While such scores provide a general idea of the behavior of these systems, they ignore a key piece of information that can be useful for assessing progress and discerning remaining challenges: the relative difficulty of test instances. To address this shortcoming, we introduce the notion of differential evaluation which effectively defines a pragmatic partition of instances into gradually more difficult bins by leveraging the predictions made by a set of systems. Comparing systems along these difficulty bins enables us to produce a finergrained analysis of their relative merits, which we illustrate on two use-cases: a comparison of systems participating in a multi-label text classification task (CLEF eHealth 2018 ICD-10 coding), and a comparison of neural models trained for biomedical entity detection (BioCreative V chemical-disease relations dataset).
|
|
Keyword:
[INFO.INFO-IR]Computer Science [cs]/Information Retrieval [cs.IR]; [INFO.INFO-TT]Computer Science [cs]/Document and Text Processing; [INFO]Computer Science [cs]; [SHS.LANGUE]Humanities and Social Sciences/Linguistics
|
|
URL: https://hal.archives-ouvertes.fr/hal-03432331
https://hal.archives-ouvertes.fr/hal-03432331/file/Differential_Evaluation__Assessing_Natural_Language_Processing_SystemPerformance_Based_Upon_Data_Resistance_to_Processing_final.pdf
https://hal.archives-ouvertes.fr/hal-03432331/document
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters
|
|
|
|
In: International Conference on Computational Linguistics ; https://hal.archives-ouvertes.fr/hal-03100665 ; International Conference on Computational Linguistics, Dec 2020, Barcelona (on line), Spain. pp.6903-6915 ; https://coling2020.org/ (2020)
|
|
BASE
|
|
Show details
|
|
|
| 3 |
Embedding Strategies for Specialized Domains: Application to Clinical Entity Recognition
|
|
|
|
In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop ; https://hal.archives-ouvertes.fr/hal-02860947 ; Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, Jul 2019, Florence, France. pp.295-301, ⟨10.18653/v1/P19-2041⟩ (2019)
|
|
BASE
|
|
Show details
|
|
|
|
|
