Apache Airflow

The Heart of Orchestration

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. Workflows are represented as Directed Acyclic Graphs (DAGs).

DAG Orchestration Flow

Below is the DAG that the code above builds, drawn as it actually runs. Watch the connectors: the wire between two tasks carries a URI through XCom, not a dataframe — which is the whole reason the callout below matters.

The DAG this page's code builds — 3 tasks, 2 XCom hops

Idempotency is Key

Always design your DAGs so that if they run multiple times for the same logical date (e.g., due to retries), they produce the exact same outcome without duplicating data.

Core Concepts

Airflow utilizes several primitive concepts to build complex orchestration layers:

DAG (Directed Acyclic Graph): The blueprint of your workflow. It defines the execution order.
Operator: A template for a specific task (e.g., PythonOperator, BashOperator, BigQueryInsertJobOperator).
Task: An instantiated Operator inside a DAG.
Sensor: A special type of operator that waits for an external event to occur before succeeding.
XCom (Cross-Communication): A mechanism that allows tasks to talk to each other by sharing small amounts of metadata.

Note: Airflow is an orchestrator, not a processing engine. Avoid passing large datasets through XComs. Use XComs to pass URIs or metadata instead!

The TaskFlow API

Modern Airflow (2.0+) strongly recommends using the TaskFlow API which utilizes Python decorators (@dag, @task) to dramatically reduce boilerplate code.

from datetime import datetime
from airflow.decorators import dag, task

@dag(
    schedule="@daily",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    tags=["etl", "sales"]
)
def sales_processing_pipeline():

    @task
    def extract_sales_data():
        # logic to extract data
        return "gs://bucket/sales_2026_10_10.csv"

    @task
    def transform_data(file_uri: str):
        # logic to transform
        return "gs://bucket/sales_clean.csv"

    @task
    def load_to_warehouse(file_uri: str):
        # logic to load into BigQuery
        print(f"Loaded {file_uri} successfully!")

    # Setting dependencies is as easy as standard Python function calls
    raw_file = extract_sales_data()
    clean_file = transform_data(raw_file)
    load_to_warehouse(clean_file)

# Instantiate the DAG
sales_processing_pipeline()

Avoid Top-Level Code

Never write expensive database queries or API calls at the “top level” of your DAG file (outside of a task). The Airflow scheduler parses DAG files every 30 seconds, and top-level code will severely slow down the entire cluster.