1 |
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus ...
|
|
|
|
Abstract:
The need for raw large raw corpora has dramatically increased in recent years with the introduction of transfer learning and semi-supervised learning methods to Natural Language Processing. And while there have been some recent attempts to manually curate the amount of data necessary to train large language models, the main way to obtain this data is still through automatic web crawling. In this paper we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR that could prove more suitable to pre-train large generative language models as well as hopefully other applications in Natural Language Processing and Digital Humanities. ... : 12 pages, 6 figures, 2 tables ...
|
|
Keyword:
Computation and Language cs.CL; FOS Computer and information sciences
|
|
URL: https://dx.doi.org/10.48550/arxiv.2201.06642 https://arxiv.org/abs/2201.06642
|
|
BASE
|
|
Hide details
|
|
2 |
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages ...
|
|
|
|
BASE
|
|
Show details
|
|
5 |
Méthodes pour la représentation informatisée de données lexicales / Methoden der Speicherung lexikalischer Daten ...
|
|
|
|
BASE
|
|
Show details
|
|
7 |
A prototype for projecting HPSG syntactic lexica towards LMF ...
|
|
|
|
BASE
|
|
Show details
|
|
9 |
Stabilizing knowledge through standards - A perspective for the humanities ...
|
|
|
|
BASE
|
|
Show details
|
|
12 |
Standardization of the formal representation of lexical information for NLP ...
|
|
|
|
BASE
|
|
Show details
|
|
15 |
International Standard for a Linguistic Annotation Framework ...
|
|
|
|
BASE
|
|
Show details
|
|
16 |
Multimodal Meaning Representation for Generic Dialogue Systems Architectures ...
|
|
|
|
BASE
|
|
Show details
|
|
18 |
Unification of multi-lingual scientific terminological resources using the ISO 16642 standard. The TermSciences initiative ...
|
|
|
|
BASE
|
|
Show details
|
|
19 |
Un modèle générique d'organisation de corpus en ligne: application à la FReeBank ...
|
|
|
|
BASE
|
|
Show details
|
|
|
|