Multi-Label Language Identification for Similar Languages

Abstract

Language identification is a part of most natural language processing pipelines. Despite sometimes being proclaimed as a solved problem there are scenarios which occur with frequency when gathering data from the Internet which are underserved by test datasets. We create one such dataset, CHALIS, that tests the distinction of several linguistically close languages as well as encountered transliteration transformations of texts. We also explore the use of transformer based models using character-level tokenization through finetuning the CANINE-c model. We find the the model to not be best suited for language identification task, however some techniques used during the finetuning could serve as an inspiration for future work.

Identifikace jazyka je souc ̌a 'stı ́ve ̌ts ̌iny pipeline pro zpracova ́nı ́pr ̌irozene ́ho jazyka. Pr ̌estoz ̌e je ne ̌kdy oznac ̌ova ́na jako vyr ̌es ̌eny ́ proble ́m, existujı ́pr ̌ı ́pady, ktere ́ se pr ̌i sbe ̌ru dat z internetu vyskytujı ́, a ktere ́ nejsou dostatec ̌ne ̌ pokryty souc ̌asny ́mi testovacı ́mi sadami. Vytvor ̌ili jsme jednu takovou sadu, CHALIS, ktera ́ testuje rozlis ̌enı ́ ne ̌kolika blı ́zky ́ch jazyku ̊, a take ́ ne ̌ktere ́ transliterace textu, se ktery ́mi se lze setkat. Da ́le zkouma ́me vyuz ̌itı ́ modelu ̊ zaloz ̌eny ́ch na transformerech s tokenizacı ́ na u ́rovni znaku ̊ prostr ̌ednictvı ́m dotre ́nova ́nı ́ modelu CANINE-c. Zjis ̌t 'ujeme, z ̌e tento model nenı ́pro u ́lohu identifikace ja- zyka nejvhodne ̌js ̌ı ́, nicme ́ne ̌ ne ̌ktere ́ techniky pouz ̌ite ́ be ̌hem dolade ̌nı ́ mohou poslouz ̌it jako inspirace pro budoucı ́ pra ́ci.