Apache Airflow Fundamentals

Apache Airflow develops, schedules, and monitors batch-oriented workflows. You define a workflow in Python. Airflow turns that definition into scheduled task instances, tracks their state, and exposes their history through a user interface and API.

The central mental model is a control plane for repeatable work.

Python Dag -> parsed definition -> Dag run -> task instances -> external systems
                    |                |              |
             metadata database  scheduler       executor

Airflow coordinates work. It does not need to perform every computation itself. A task can submit SQL, start a container, call an API, run Python, or invoke another system through a provider.

Why workflow orchestration exists

A data pipeline rarely consists of one operation. You might wait for an input, validate it, transform it, publish a table, and notify an owner. Each step has dependencies, retries, credentials, logs, and an operating schedule.

A collection of scripts can run those steps. It becomes difficult to answer basic operational questions as the collection grows:

Which step is ready to run?
Which input period does this run cover?
What failed, and how many times did it retry?
Can you rerun one step without repeating every step?
What happened on a past date?
How many workflows may use a scarce service at once?

Airflow records this control state. It schedules eligible work and gives operators one place to inspect it.

A Dag describes the workflow

A Dag is the Airflow model for a workflow. The name comes from directed acyclic graph. Directed edges express dependency order. Acyclic means a task cannot eventually depend on itself.

The Dag definition contains tasks, dependencies, schedule information, and operational settings. It describes the workflow. It is not one execution of that workflow.

extract -> validate -> transform -> publish
               |                       |
               +------> quarantine     +-> notify

Each node is a task. Each edge says which task is upstream and which is downstream. By default, a task waits for its upstream tasks to succeed. Trigger rules can change that condition for branching, cleanup, and other control-flow needs.

Keep the graph focused on orchestration. The Dag should reveal the major units of work and their dependencies. Business computation belongs inside tasks or in the external systems that tasks invoke.

Airflow parses Dag files repeatedly. Top-level code therefore runs during parsing, not only during a scheduled run. Network calls, database queries, and expensive imports at the top level slow parsing and can overload dependencies. Put runtime work inside tasks.

A Dag run is one execution in time

A Dag run is one instance of a Dag. Several Dag runs can exist for the same Dag. Each run has its own task instances and state.

For a time-based schedule, a Dag run usually represents a data interval. A daily run might represent data from one midnight to the next. The scheduler normally creates the run after that interval ends, when the complete interval is available.

The run's logical date identifies the start of its data interval. It is not the wall-clock time when the run began. This distinction prevents a common scheduling mistake.

data interval:  July 13 00:00 ---------------- July 14 00:00
logical date:   July 13 00:00
scheduled run:                                      after interval closes

Use the data interval to select input and output partitions. Avoid selecting “the latest” data inside a scheduled task. A retry could then process a different input from the first attempt.

Airflow can create scheduled runs from cron expressions, fixed intervals, custom timetables, or asset events. You can also trigger a run manually or through the API.

Catchup creates missing scheduled Dag runs from the start date through the current interval. Disabling catchup means the scheduler normally creates only the latest interval. Backfill is an explicit operation that creates runs for a chosen historical date range.

Catchup and backfill do not make task code safe to repeat. Your task design must do that.

A task is a unit of work

A task is the basic unit of execution. A task instance is that task within one Dag run.

Airflow supports three common authoring forms:

an operator, which is a reusable task template;
a sensor, which waits for an external condition;
a TaskFlow function decorated with @task.

When you instantiate an operator in a Dag, you create a task. Operators and sensors often come from provider packages. A provider integrates Airflow with an external system such as a database, cloud platform, or messaging service.

TaskFlow turns a Python function into a task. Calling that decorated function while building the Dag does not run its body. It creates task relationships and returns an XComArg that represents the future result.

from airflow.sdk import dag, task

@dag(schedule="@daily", catchup=False)
def daily_orders():
    @task
    def extract():
        return {"partition": "orders"}

    @task
    def load(metadata):
        print(metadata["partition"])

    load(extract())

daily_orders()

The value passed from extract to load is represented through an XCom. TaskFlow creates the dependency automatically.

XComs are designed for small serializable values and metadata. They are not a data transport for dataframes or large files. Store large data in an object store, database, or another suitable system. Pass its location or identifier through XCom.

The scheduler decides what is ready

The scheduler examines Dag runs and task dependencies. When a task instance is eligible, the scheduler sends it to the configured executor.

The executor determines how task instances are dispatched. It is a scheduler configuration, not a separate Airflow component. A local executor can run work on one machine. Distributed executors can send work to remote workers or create pods.

The scheduler does not guarantee that related tasks run on the same worker. Tasks must not depend on one worker's local filesystem unless the deployment provides that contract.

Task instances move through states such as scheduled, queued, running, success, failed, skipped, and upstream failed. Retries add states between a failed attempt and the next attempt. The state model lets Airflow decide what may run and lets you diagnose what happened.

The final state of a Dag run is derived from its leaf tasks. A permissive trigger rule on a leaf can therefore hide an earlier failure. Review terminal tasks and trigger rules together.

The runtime is a set of cooperating components

A current Airflow deployment has these required parts:

The Dag processor parses Dag files from Dag bundles and serializes definitions.
The scheduler creates scheduled Dag runs and submits ready task instances.
The API server serves the user interface and REST API. Tasks also report state through its execution interface.
The metadata database stores Dag, run, task, connection, variable, and other control state.
A Dag bundle supplies versioned or local Dag files to processors and workers.

Workers execute tasks when the executor uses separate workers. A triggerer is optional. It runs the asynchronous triggers used by deferred tasks.

Dag bundle -> Dag processor -> metadata database <- scheduler
     |                                    |             |
     +----------------> workers <------ API server <----+
                              \
                               triggerer for deferred waits

The separation matters for scale and security. In Airflow 3, the Dag processor is a standalone component. Tasks and workers communicate through the API server instead of requiring direct metadata-database access.

Waiting should not occupy an expensive worker

A sensor checks whether an external condition is true. A sensor that holds a worker slot while sleeping can waste capacity.

Some sensors can run in reschedule mode. They release their worker slot between checks. Deferrable operators take this further. They hand the wait to the triggerer and resume on a worker when the trigger fires.

Deferral changes resource use, not business semantics. You still need timeouts, failure behavior, and a clear answer for an event that never arrives.

Retries require idempotent work

Airflow may run a task more than once because of retries, manual clearing, or historical reprocessing. Design each task so repetition produces the same intended result.

Treat a task like a database transaction:

write complete results or no result;
use an upsert or replace operation when an insert would duplicate rows;
read and write a specific data interval or partition;
avoid current time in critical computation;
make external side effects idempotent or protect them with a deduplication key.

Airflow tracks orchestration state. It cannot make an arbitrary external API call transactional. A successful task means the task process reported success. Your code must define what success means for the external system.

Concurrency is part of workflow design

More parallel tasks can reduce elapsed time. They can also overload a database, API, queue, or warehouse.

Airflow provides several control points:

task dependencies constrain order;
pools limit concurrent access to shared capacity;
Dag and task concurrency settings limit active work;
executor configuration controls dispatch capacity;
queues or executor-specific settings route work to suitable workers.

Choose limits from the downstream system's capacity. A large worker fleet does not increase the safe connection count of a small database.

Assets add data-aware scheduling

An asset is a logical grouping of data identified by a URI. A producer task can record an asset update. That update can contribute to scheduling a consumer Dag.

Asset-aware scheduling expresses that a workflow depends on data becoming available. Time-based scheduling expresses that a workflow should consider a time interval. You can use either model or combine them when the data contract requires both.

An asset identifies data for orchestration. Airflow does not inspect the content behind the URI. The producer and consumer still own schema, quality, and compatibility.

Observability starts with task history

The user interface shows Dag structure, run history, task states, and task logs. It supports manual triggers and operational actions such as clearing failed task instances.

Production observability also needs component health, metrics, durable logs, and alert routing. Local task logs are useful for development. Remote or external log storage is safer when workers are disposable.

Observe both the control plane and the work it invokes. A green task can submit an external job that later fails unless the task waits for and validates the external result.

Useful signals include:

scheduler and Dag processor health;
queued task age and pool saturation;
Dag run duration and failure rate;
task retry counts and duration;
metadata database capacity;
worker availability;
external job identifiers and outcomes.

Security follows the execution boundary

Dag files are Python code. A Dag author can cause code to run in the Dag processor and workers. Treat Dag authoring as a privileged capability.

Airflow distinguishes deployment managers, Dag authors, and operations users. Separate their permissions. Protect the API server and metadata database. Restrict workers to the credentials each workload needs.

Connections centralize access information for tasks, but connection storage is not a complete secrets strategy. Use a supported secrets backend where appropriate. Apply network policy and workload identity outside Airflow as well.

Do not pass secrets through XCom, task logs, Dag parameters, or source code. A user who can view task details may be able to see those values.

Deployment choices change the failure model

A local installation is useful for learning and development. Production needs an external PostgreSQL or MySQL metadata database. SQLite is not a production backend.

A distributed deployment adds workers, queues, shared or versioned Dag delivery, remote logs, and more failure boundaries. It also lets you isolate components and scale them independently.

Airflow provides official container images and a Helm chart. Managed services can own much of the control-plane operation. None of these choices removes the need to test Dag code, size dependencies, manage providers, secure credentials, and plan upgrades.

Provider packages have release cycles independent of Airflow core. Pin and test the complete environment. A Dag may depend on Airflow, the Task SDK, one or more providers, and client libraries for external services.

Where Airflow fits well

Airflow fits work that has explicit steps, dependency order, repeatable runs, and an operational history. Common examples include:

scheduled extract, transform, and load pipelines;
warehouse and lakehouse maintenance;
machine-learning training and evaluation workflows;
recurring reports and file delivery;
infrastructure or business processes with auditable steps;
workflows triggered by data asset updates.

Airflow is especially useful when Python-defined orchestration, retries, backfills, and a shared operating interface matter.

Limits and poor fits

Airflow is batch-oriented. It can orchestrate a streaming system, but it is not the stream processor carrying every event through a continuously running topology.

It is also not a data store, compute engine, message broker, or universal transaction coordinator. Tasks invoke those systems and track their orchestration state.

Airflow may be excessive for one small scheduled command with no dependencies or operational requirements. It is also a poor place to move large payloads through XCom or hide major business logic in Dag construction code.

Operating Airflow has a real cost. You must maintain the metadata database, component availability, Dag delivery, logs, metrics, secrets, providers, and upgrades. A managed service shifts some of that work. It does not remove workflow ownership.

A practical learning path

Run the official quick start and inspect the user interface.
Build one Dag with two TaskFlow tasks and one explicit dependency.
Compare a Dag definition, Dag run, task, and task instance in the interface.
Schedule a daily run and inspect its logical date and data interval.
Force a task failure, observe retry states, and clear the task instance.
Make the task idempotent for one fixed partition.
Pass small metadata with TaskFlow and store large output externally.
Add a provider operator and trace the external job through its final state.
Test Dag loading, task behavior, and a historical backfill in staging.
Study pools, executor choices, security boundaries, metrics, and production deployment.