Apache Airflow Fundamentals

Apache Airflow develops, schedules, and monitors batch-oriented workflows. You define a workflow in Python. Airflow turns that definition into scheduled task instances, tracks their state, and exposes their history through a user interface and API.

The central mental model is a control plane for repeatable work.

Python Dag -> parsed definition -> Dag run -> task instances -> external systems
                    |                |              |
             metadata database  scheduler       executor

Airflow coordinates work. It does not need to perform every computation itself. A task can submit SQL, start a container, call an API, run Python, or invoke another system through a provider.

Why workflow orchestration exists

A data pipeline rarely consists of one operation. You might wait for an input, validate it, transform it, publish a table, and notify an owner. Each step has dependencies, retries, credentials, logs, and an operating schedule.

A collection of scripts can run those steps. It becomes difficult to answer basic operational questions as the collection grows:

Which step is ready to run?
Which input period does this run cover?
What failed, and how many times did it retry?
Can you rerun one step without repeating every step?
What happened on a past date?
How many workflows may use a scarce service at once?

Airflow records this control state. It schedules eligible work and gives operators one place to inspect it.

A Dag describes the workflow

A Dag is the Airflow model for a workflow. The name comes from directed acyclic graph. Directed edges express dependency order. Acyclic means a task cannot eventually depend on itself.

The Dag definition contains tasks, dependencies, schedule information, and operational settings. It describes the workflow. It is not one execution of that workflow.

extract -> validate -> transform -> publish
               |                       |
               +------> quarantine     +-> notify

Each node is a task. Each edge says which task is upstream and which is downstream. By default, a task waits for its upstream tasks to succeed. Trigger rules can change that condition for branching, cleanup, and other control-flow needs.

Keep the graph focused on orchestration. The Dag should reveal the major units of work and their dependencies. Business computation belongs inside tasks or in the external systems that tasks invoke.

Airflow parses Dag files repeatedly. Top-level code therefore runs during parsing, not only during a scheduled run. Network calls, database queries, and expensive imports at the top level slow parsing and can overload dependencies. Put runtime work inside tasks.

A Dag run is one execution in time

A Dag run is one instance of a Dag. Several Dag runs can exist for the same Dag. Each run has its own task instances and state.

For a time-based schedule, a Dag run usually represents a data interval. A daily run might represent data from one midnight to the next. The scheduler normally creates the run after that interval ends, when the complete interval is available.

Apache Airflow Fundamentals

Why workflow orchestration exists

A Dag describes the workflow

A Dag run is one execution in time

Relevant careers