Avaliação da Generalização de Modelos BERTimbau na Detecção de Desinformação em Português com Análise Qualitativa de Dados

Abstract

The growing dissemination of information on the internet has intensified the spread of fake news, making automatic fake news detection a relevant problem. In this context, pre-trained language models have shown significant performance in natural language processing tasks, including text classification in Portuguese. This work evaluates the performance of the BERTimbau model in classifying news as true or false, using multiple datasets with distinct characteristics. To ensure a balanced evaluation, the datasets were previously balanced according to their classes. The experiments were conducted to analyze not only the model performance within the training domain, but also its generalization ability across different datasets. Additionally, a qualitative analysis step based on a large language model was incorporated to examine textual patterns, differences among datasets, and recurring aspects associated with the classifier's correct and incorrect predictions. The results indicate that, although the model achieves good performance on data similar to those used during training, there is a significant performance reduction in out-of-domain scenarios, highlighting limitations in generalization. These findings reinforce the importance of robustness evaluation in real-world applications of language models.