UDP Data Automation

UDP Data Automation is a modular data integration framework built with Apache Airflow and Python.

It implements a generic, template-based ETL process (Extract, Transform, Load) that automates and standardizes data workflows for various formats (Excel, CSV, SDE/MSSQL).

The system features a DAG generator that reads configuration data from the UDP Manager, an UDP Data-Management-Tool, (via REST API) to automatically create and update Airflow DAGs for each dataset, enabling flexible, asynchronous, and maintainable data processing pipelines.