1 |
Unsupervised Creation of Normalisation Dictionaries for Micro-Blogs in Arabic, French and English
|
|
|
|
In: 19th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing 2018) ; https://hal.archives-ouvertes.fr/hal-01795348 ; 19th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing 2018), Mar 2018, Hanoi, Vietnam (2018)
|
|
Abstract:
International audience ; Text normalisation is a necessity to correct and make more sense of the micro-blogs messages, for information retrieval purposes. Unfortunately, tools and resources of text normalisation are rarely shared. In this paper, an approach is presented based on an unsupervised method for text normalisation using distributed representations of words, known also as "word embedding", applied on Arabic, French and English Languages. In addition, a tool will be supplied to create dictionaries for micro-blogs normalisation, in a form of pairs of misspelled word with its standard-form word, in the languages: Arabic, French and English. The tool will be available as open source including the resources: word embedding's models (with vocabulary size of million words for Arabic language model, million words for English language model and thousand words for French language model), and also three normalisation dictionaries of thousand pairs in Arabic language, thousand pairs in French language and thousand pairs in English language. The evaluation of the tool shows an average in N ormalisation success of % for English language, .% for Arabic Language and % for French Language. Also, the results of using an English language normalisation dictionary with a sentiment analysis tool for micro-blog's messages, show an increase in f-measure from. to .
|
|
Keyword:
[INFO.INFO-CL]Computer Science [cs]/Computation and Language [cs.CL]; [INFO.INFO-IR]Computer Science [cs]/Information Retrieval [cs.IR]; [INFO.INFO-SI]Computer Science [cs]/Social and Information Networks [cs.SI]; [INFO.INFO-TT]Computer Science [cs]/Document and Text Processing; [INFO]Computer Science [cs]; arabic; dictionaries; french; micro-blogs; multilingual; normalisation; unsuper- vised; word embedding
|
|
URL: https://hal.archives-ouvertes.fr/hal-01795348 https://hal.archives-ouvertes.fr/hal-01795348/document https://hal.archives-ouvertes.fr/hal-01795348/file/AmalHtait_CICLing2018.pdf
|
|
BASE
|
|
Hide details
|
|
2 |
Unsupervised Creation of Normalization Dictionaries for Micro-Blogs in Arabic, French and English
|
|
|
|
In: ISSN: 1405-5546 ; EISSN: 2007-9737 ; Computación y sistemas ; https://hal.archives-ouvertes.fr/hal-01958675 ; Computación y sistemas, Instituto Politécnico Nacional IPN Centro de Investigación en Computación, 2018, 19th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing 2018), 22 (3), pp.729-737. ⟨10.13053/CyS-22-3-3034⟩ ; https://www.cys.cic.ipn.mx/ojs/index.php/CyS/article/view/3034/2514 (2018)
|
|
BASE
|
|
Show details
|
|
|
|