Nouveau Recherche PDF, HTML, DOCX et bien d'autres formats.

Prépublication · 2026

When does text encoding matter for language modeling?

Eugene Jang, Catherine Arnett et al. — Monde

Source : arXiv

Résumé

UTF-8's role as the de facto encoding standard in language models introduces cross-lingual inequities due to its variable-length design. However, the effect that this encoding choice has on model performance remains largely unexamined. In this work, we investigate how text encoding impacts the performance of byte-level language models, comparing UTF-8 with a wide range of other encoding strategies, including other Unicode-based approaches, language-specific encodings, and our own customized encodings. To facilitate our analysis, we propose Aether: a hybrid encoding scheme that uses script-spec…

Actions

Citation