What Is Apache Airflow? The Workflow Orchestrator Explained

Apache Airflow is an open-source platform for authoring, scheduling, and monitoring data workflows defined as Python DAGs.

Apache Airflow is the workhorse orchestrator of data engineering. It lets you define complex multi-step workflows in Python, schedule them, and watch them run, with built-in retries, dependencies, and a web UI that shows the state of every task.

What Airflow is

Airflow is a workflow orchestration platform. You write a DAG - a directed acyclic graph of tasks - as Python code, and Airflow schedules and runs the tasks in dependency order, handling retries, backfills, and alerting. It is widely used to run ETL and ELT pipelines.

DAGs and tasks

Each Airflow DAG is a collection of tasks with dependencies between them. A task might run a SQL query, call an API, or trigger a Spark job. Airflow figures out what can run in parallel and what must wait, and it reruns failed tasks according to your retry policy.

Scheduling and monitoring

Airflow schedules DAGs on cron-like intervals and tracks every run in its metadata database, surfacing status, logs, and durations in a web UI. Operators can rerun a failed task or backfill a date range without rewriting code.

Airflow and CI/CD

Airflow orchestrates data pipelines, while CI/CD ships the DAG code that defines them. A good setup lints and unit-tests DAGs in CI - checking they parse and have no cycles - before deploying them to Airflow.

Testing Airflow DAGs in CI

steps:
  - run: pip install apache-airflow
  - run: python -c "import dags.my_pipeline"   # import check
  - run: pytest tests/dags                      # task unit tests

Latchkey note

CI jobs that validate DAGs install Airflow and its providers, which is heavy. On Latchkey, caching that environment between runs makes DAG-validation jobs fast, and auto-retry covers transient package-index or provider download failures during install.

Key takeaways

Apache Airflow authors, schedules, and monitors data workflows defined as Python DAGs.
It runs tasks in dependency order with built-in retries, backfills, and a monitoring UI.
Airflow orchestrates pipelines; CI/CD ships and tests the DAG code that defines them.