Tucano: Advancing Neural Text Generation for Portuguese

Resource information

Authors

Nicholas Kluge Corrêa, Aniket Sen, Sophia Falk, Shiza Fatimah

Research area

Language technology

Application domain

General AI research

AI methods

Deep learning, Transformers, Natural language processing

Research topics

cs.CL, cs.AI, cs.LG

arXiv record date

2024-11-12

Publication

arXiv

DOI

10.1016/j.patter.2025.101325

Upstream last modified

Jul 25, 2025

Last record update

Nov 3, 2026

Abstract

Significant advances have been made in natural language processing in recent years. However, our current deep learning approach to language modeling requires substantial resources in terms of data and computation. One of the side effects of this data-hungry paradigm is the current schism between languages, separating those considered high-resource, where most of the development happens and resources are available, and the low-resource ones, which struggle to attain the same level of performance and autonomy. This study aims to introduce a new set of resources to stimulate the future development of neural text generation in Portuguese. In this work, we document the development of GigaVerbo, a concatenation of deduplicated Portuguese text corpora amounting to 200 billion tokens. Via this corpus, we trained a series of decoder-transformers named Tucano. Our models perform equal or superior to other Portuguese and multilingual language models of similar size in several Portuguese benchmarks. The evaluation of our models also reveals that model performance on many currently available benchmarks used by the Portuguese NLP community has little to no correlation with the scaling of token ingestion during training, highlighting the limitations of such evaluations when it comes to the assessment of Portuguese generative language models. All derivatives of our study are openly released on GitHub and Hugging Face. See https://nkluge-correa.github.io/Tucano/