
Apache Airflow
Apache Software Foundation · Platform to author, schedule and monitor workflows in Python
The industry standard for data pipelines.
Apache Airflow is the most popular data orchestration tool, with 47,488 GitHub stars. It is a platform to programmatically author, schedule and monitor workflows. You define workflows (Dags) as Python code, which makes them easier to maintain, version, test and share, and the Airflow scheduler runs your tasks on an array of workers while following their dependencies. A rich web UI shows pipelines running in production so you can monitor progress and troubleshoot. Airflow uses the Jinja templating engine and a wide range of built-in operators, and its modular architecture uses a message queue to orchestrate any number of workers. Beyond data pipelines, it is widely used for machine learning and increasingly for AI and LLM workloads. Its README says the project knows of around 500 organizations using Airflow. It is an Apache project under the Apache License 2.0.
Source: github.com/apache/airflow
Kestra30,235 GitHub stars
Prefect24,159 GitHub stars





