| 1 |
Reproducibility in Computational Linguistics: Are We Willing to Share?
|
|
|
|
In: Computational Linguistics, Vol 44, Iss 4, Pp 641-649 (2018) (2018)
|
|
Abstract:
This study focuses on an essential precondition for reproducibility in computational linguistics: the willingness of authors to share relevant source code and data. Ten years after Ted Pedersen’s influential “Last Words” contribution in Computational Linguistics, we investigate to what extent researchers in computational linguistics are willing and able to share their data and code. We surveyed all 395 full papers presented at the 2011 and 2016 ACL Annual Meetings, and identified whether links to data and code were provided. If working links were not provided, authors were requested to provide this information. Although data were often available, code was shared less often. When working links to code or data were not provided in the paper, authors provided the code in about one third of cases. For a selection of ten papers, we attempted to reproduce the results using the provided data and code. We were able to reproduce the results approximately for six papers. For only a single paper did we obtain the exact same results. Our findings show that even though the situation appears to have improved comparing 2016 to 2011, empiricism in computational linguistics still largely remains a matter of faith. Nevertheless, we are somewhat optimistic about the future. Ensuring reproducibility is not only important for the field as a whole, but also seems worthwhile for individual researchers: The median citation count for studies with working links to the source code is higher.
|
|
Keyword:
Computational linguistics. Natural language processing; P98-98.5
|
|
URL: https://doi.org/10.1162/coli_a_00330
https://doaj.org/article/155f8b6efe174058b23fd84833a03cc7
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
Synchronic Patterns of Tuscan Phonetic Variation and Diachronic Change: Evidence from a Dialectometric Study
|
|
|
|
In: http://urd.let.rug.nl/nerbonne/papers/Montemagni_Wieling_DeJonge_Nerbonne_LLC-2011.pdf (2013)
|
|
BASE
|
|
Show details
|
|
|
| 3 |
A Cognitively Grounded Measure of Pronunciation Distance
|
|
|
|
In: http://www.let.rug.nl/~gooskens/pdf/publ_PLoS_ONE_2014.pdf (2013)
|
|
BASE
|
|
Show details
|
|
|
| 4 |
Quantitative social dialectology: explaining linguistic variation geographically and socially
|
|
|
|
In: ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/d8/a3/PLoS_One_2011_Sep_1_6(9)_e23613.tar.gz (2011)
|
|
BASE
|
|
Show details
|
|
|
| 5 |
Multiple sequence alignments in linguistics
|
|
|
|
In: http://www.martijnwieling.nl/files/Prokic-Wieling-Nerbonne-2009.pdf (2009)
|
|
BASE
|
|
Show details
|
|
|
| 6 |
Phonetic variation in the traditional English dialects: a computational analysis
|
|
|
|
In: http://urd.let.rug.nl/nerbonne/papers/WielingShackletonNerbonne-2011.pdf (2007)
|
|
BASE
|
|
Show details
|
|
|
| 7 |
Dialect pronunciation comparison and spoken word recognition
|
|
|
|
In: http://www.martijnwieling.nl/files/cohort.pdf (2007)
|
|
BASE
|
|
Show details
|
|
|
| 8 |
An aggregate analysis of pronunciation in the Goeman-Taeldeman-Van Reenen-Project data. Taal en Tongval
|
|
|
|
In: http://www.let.rug.nl/%7Eheeringa/dialectology/papers/tet06.pdf (2007)
|
|
BASE
|
|
Show details
|
|
|
| 9 |
LEXICAL DIFFERENCES BETWEEN TUSCAN DIALECTS AND STANDARD ITALIAN: A SOCIOLINGUISTIC ANALYSIS USING GENERALIZED ADDITIVE MIXED MODELING
|
|
|
|
In: http://urd.let.rug.nl/nerbonne/papers/Wieling-etal-Language-2013.pdf
|
|
BASE
|
|
Show details
|
|
|
| 10 |
Linguistic advergence and divergence in north-western Catalan: A dialectometric investigation of dialect leveling and border effects
|
|
|
|
In: http://urd.let.rug.nl/nerbonne/papers/Valls-Wieling-Nerbonne-LLC-2013-28(1).pdf
|
|
BASE
|
|
Show details
|
|
|
| 11 |
Dialect Pronunciation Comparison and Spoken Word Recognition
|
|
|
|
In: http://odur.let.rug.nl/~nerbonne/papers/Wieling-Nerbonne-Cohort-2007.pdf
|
|
BASE
|
|
Show details
|
|
|
| 12 |
Inducing phonetic distances from dialect variation
|
|
|
|
In: http://www.clinjournal.org/sites/default/files/Wieling_upd.pdf
|
|
BASE
|
|
Show details
|
|
|
| 13 |
Automatically measuring the strength of foreign accents in English
|
|
|
|
In: http://urd.let.rug.nl/nerbonne/papers/WielingEtAl-Accents-Validating-2013-final1.pdf
|
|
BASE
|
|
Show details
|
|
|
| 14 |
Inducing phonetic distances from dialect variation
|
|
|
|
In: http://urd.let.rug.nl/nerbonne/papers/Wieling-MN-CLIN-final-2011-Nov-15.pdf
|
|
BASE
|
|
Show details
|
|
|
| 15 |
SOME FURTHER DIALECTOMETRICAL STEPS
|
|
|
|
In: http://urd.let.rug.nl/nerbonne/papers/Further-Steps-Preprint-2010.pdf
|
|
BASE
|
|
Show details
|
|
|
| 16 |
pronunciation
|
|
|
|
In: http://urd.let.rug.nl/nerbonne/papers/Wieling-MN-JPhon-2011-Dec-12.pdf
|
|
BASE
|
|
Show details
|
|
|
| 17 |
Multiple sequence alignments in linguistics
|
|
|
|
In: http://odur.let.rug.nl/~nerbonne/papers/prokic-wieling-nerbonne-2009-eacl.pdf
|
|
BASE
|
|
Show details
|
|
|
| 18 |
Evaluating the pairwise string alignment of pronunciations
|
|
|
|
In: http://odur.let.rug.nl/~nerbonne/papers/WielingProkicNerbonne-2009-LaTeCH-SHELT&R.pdf
|
|
BASE
|
|
Show details
|
|
|
| 19 |
Evaluating the pairwise string alignment of pronunciations
|
|
|
|
In: http://aclweb.org/anthology-new/W/W09/W09-0304.pdf
|
|
BASE
|
|
Show details
|
|
|
| 20 |
Multiple sequence alignments in linguistics, in
|
|
|
|
In: http://aclweb.org/anthology-new/W/W09/W09-0303.pdf
|
|
BASE
|
|
Show details
|
|
|
|
|
