Course Preview

Airbyte Fundamentals

Airbyte is an open-source data integration platform. It copies data out of APIs, databases, and files and loads it into a data warehouse, lake, or database using pre-built connectors, so teams do not hand-write and maintain a separate extraction script for every system they need data from.

itData engineering and analytics | OpenSkills.info

Course pathWalk it in order

Look it upDip in anytime

Go furtherLeaves this page

Don't Panic: Airbyte Fundamentals

Airbyte is an open-source tool that copies data out of one system and into another. That is the entire job. It pulls rows from a database, records from an API, files from a bucket, and lands all of it in a warehouse, then remembers where it stopped so the next run only moves what changed.

The problem it solves is tedium at scale. Before a tool like this, every system you wanted data from meant another small program: its own login handling, its own paging, its own logic for "what is new since last time." Multiply that by forty sources and you have a codebase whose only purpose is to break quietly whenever an upstream API changes. Airbyte replaces those programs with connectors that all speak one message format, so adding a source becomes filling in a form instead of starting a new project.

Three ideas carry the rest. A connector reads or writes exactly one system, and connectors never talk to each other; the platform sits in the middle passing messages. A connection is the pipeline you configure: one source, one destination, a list of streams, and a schedule. State is the bookmark: the source reports how far it got, the platform saves that, and the next sync starts from there. Clear the bookmark and the next sync re-reads everything, which is occasionally what you want and frequently a surprise.

The surprise worth front-loading: Airbyte does the E and the L of ELT, and stops. It will not model your data, join anything, or apply a business rule. It types the columns and removes duplicates in the destination, and that is the end of its ambition. Transformation is a separate tool's job, usually dbt, running after Airbyte. A second surprise, quieter and meaner: a sync can finish green while dropping values. Since a redesign in 2023, a value that does not fit its column type gets set to null and noted in a metadata column called _airbyte_meta, and the sync still reports success. When destination numbers look wrong, that column is where the story is.

One more thing that catches people. If you replicate a database with change data capture, the database keeps its transaction log until Airbyte reads it. Sync rarely, or pause the connection and walk away, and that log can grow until the source server runs out of disk. Sync often, and clean up after yourself.

Where to go next. The Cheatsheet is the reference for sync modes, the connector protocol, CDC metadata columns, and the platform services. The Practice Reference has the abctl commands to run Airbyte locally plus SQL to inspect what a sync produced. The Field Notes cover what teams get wrong: uneven connector quality, the Kubernetes commitment that self-hosting now carries, and why anything written about Airbyte before late 2023 describes tables that no longer exist. Start with the intro; it explains the whole machine end to end.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Sources