Abordagens Multimodais com Fusão de Dados em Aprendizado Profundo

Abstract

Deep neural networks, especially language and vision models, have been widely used in real problems in recent years. Usually models apply the use of only one type of data/information (text, image, video, audio) in learning problems, also called unimodal models. However, given the growing amount of unstructured information and the variety of existing data formats, new approaches have been developed with the aim of establishing strategies that enable the use of multiple data in the same learning model. This work explores data fusion in Multimodal Machine Learning (ML) models. The proposal of this thesis explores a simple strategy that uses mathematical operations to merge the different types of data between the layers of the multimodal architecture, mechanisms of attention and residual connections. Another proposal explores the use of multimodal knowledge distillation to optimize the performance of deep learning models, transferring knowledge between modalities of the same domain. The main advance of this work was to use arithmetic operations, attention mechanisms and residual connections in multimodal approaches with data fusion. This allowed obtaining complementary representations about the modalities, which led to a better convergence without significant difference with the state-of-the-art.