| 1 |
Differential Evaluation: a Qualitative Analysis of Natural Language Processing System Behavior Based Upon Data Resistance to Processing
|
|
|
|
In: Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems ; EVAL4NLP, 2nd Workshop on "Evaluation & Comparison of NLP Systems", EMNLP 2021 ; https://hal.archives-ouvertes.fr/hal-03432331 ; EVAL4NLP, 2nd Workshop on "Evaluation & Comparison of NLP Systems", EMNLP 2021, Nov 2021, Punta Cana, Dominican Republic (2021)
|
|
Abstract:
International audience ; Most of the time, when dealing with a particular Natural Language Processing task, systems are compared on the basis of global statistics such as recall, precision, F1-score, etc. While such scores provide a general idea of the behavior of these systems, they ignore a key piece of information that can be useful for assessing progress and discerning remaining challenges: the relative difficulty of test instances. To address this shortcoming, we introduce the notion of differential evaluation which effectively defines a pragmatic partition of instances into gradually more difficult bins by leveraging the predictions made by a set of systems. Comparing systems along these difficulty bins enables us to produce a finergrained analysis of their relative merits, which we illustrate on two use-cases: a comparison of systems participating in a multi-label text classification task (CLEF eHealth 2018 ICD-10 coding), and a comparison of neural models trained for biomedical entity detection (BioCreative V chemical-disease relations dataset).
|
|
Keyword:
[INFO.INFO-IR]Computer Science [cs]/Information Retrieval [cs.IR]; [INFO.INFO-TT]Computer Science [cs]/Document and Text Processing; [INFO]Computer Science [cs]; [SHS.LANGUE]Humanities and Social Sciences/Linguistics
|
|
URL: https://hal.archives-ouvertes.fr/hal-03432331
https://hal.archives-ouvertes.fr/hal-03432331/file/Differential_Evaluation__Assessing_Natural_Language_Processing_SystemPerformance_Based_Upon_Data_Resistance_to_Processing_final.pdf
https://hal.archives-ouvertes.fr/hal-03432331/document
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
Embedding Strategies for Specialized Domains: Application to Clinical Entity Recognition
|
|
|
|
In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop ; https://hal.archives-ouvertes.fr/hal-02860947 ; Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, Jul 2019, Florence, France. pp.295-301, ⟨10.18653/v1/P19-2041⟩ (2019)
|
|
BASE
|
|
Show details
|
|
|
| 3 |
CLEF eHealth 2017 Multilingual Information Extraction task Overview: ICD10 Coding of Death Certificates in English and French.
|
|
|
|
In: Workshop of the Cross-Language Evaluation Forum ; https://hal.archives-ouvertes.fr/hal-01665374 ; Workshop of the Cross-Language Evaluation Forum, CEUR-WS, Jan 2017, Dublin, Ireland (2017)
|
|
BASE
|
|
Show details
|
|
|
| 4 |
Oublier ce qu'on sait, pour mieux apprendre ce qu'on ne sait pas : une étude sur les contraintes de type dans les modèles CRF
|
|
|
|
In: Actes de la 22e conférence sur le Traitement Automatique des Langues Naturelles. Articles longs ; Conférence sur le Traitement Automatique des Langues Naturelles ; https://hal.archives-ouvertes.fr/hal-01634997 ; Conférence sur le Traitement Automatique des Langues Naturelles, ATALA, 2015, Caen, France (2015)
|
|
BASE
|
|
Show details
|
|
|
| 5 |
LIMSI$@$WMT'15 : Translation Task
|
|
|
|
In: Proceedings of the Tenth Workshop on Statistical Machine Translation ; https://hal.archives-ouvertes.fr/hal-02912383 ; Proceedings of the Tenth Workshop on Statistical Machine Translation, Sep 2015, Lisbon, Portugal. pp.145-151, ⟨10.18653/v1/W15-3016⟩ (2015)
|
|
BASE
|
|
Show details
|
|
|
| 6 |
Automatic language identity tagging on word and sentence-level in multilingual text sources: a case-study on Luxembourgish
|
|
|
|
In: International Conference on Language Resources and Evaluation ; https://hal.archives-ouvertes.fr/hal-01843401 ; International Conference on Language Resources and Evaluation, May 2014, Reykjavik, Iceland (2014)
|
|
BASE
|
|
Show details
|
|
|
| 7 |
Discriminative training of a phoneme confusion model for a dynamic lexicon in ASR
|
|
|
|
In: Interspeech 2013 ; Annual Conference of the International Speech Communication Association ; https://hal.archives-ouvertes.fr/hal-01843427 ; Annual Conference of the International Speech Communication Association, Jan 2013, Lyon, France (2013)
|
|
BASE
|
|
Show details
|
|
|
| 8 |
Automatic named entity pre-annotation for out-of-domain human annotation
|
|
|
|
In: Linguistic Annotation Workshop ; https://hal.archives-ouvertes.fr/hal-01831229 ; Linguistic Annotation Workshop, ACL, Jan 2013, Sofia, Bulgaria (2013)
|
|
BASE
|
|
Show details
|
|
|
| 9 |
Combining an expert-based medical entity recognizer to a machine-learning system: methods and a case-study
|
|
|
|
In: Biomedical Informatics Insights ; https://hal.archives-ouvertes.fr/hal-01972779 ; Biomedical Informatics Insights, 2013, 13p (2013)
|
|
BASE
|
|
Show details
|
|
|
| 10 |
Designing an improved discriminative word aligner
|
|
|
|
In: International Conference on Intelligent Text Processing and Computational Linguistics ; https://hal.archives-ouvertes.fr/hal-01960730 ; International Conference on Intelligent Text Processing and Computational Linguistics, Jan 2011, Tokyo, Japan (2011)
|
|
BASE
|
|
Show details
|
|
|
|
|
