Hydra Flow - Pipeline Automatizada para ETL de Dados

Abstract

This paper presents Hydra Flow, a data pipeline based on the ETL (Extract, Transform and Load) architecture for collecting, transforming, and storing data obtained from web APIs in JSON format. The solution uses Apache Airflow for orchestration, MinIO as S3-compatible storage, and Apache Parquet for persisting processed data. The architecture is organized into raw, staged, and curated layers, enabling the separation of raw, processed, and refined data for analytical purposes. During the ingestion stage, jq expressions are used for data extraction and initial filtering, while subsequent stages perform data standardization and transformation. The obtained results demonstrate the feasibility of using open-source tools to build automated, modular, and maintainable data pipelines.