Build Systems

A build system transforms a declared set of inputs into outputs: binaries, libraries, packages, generated source, documentation, container layers, test results, or other artifacts. It decides what must run, in what dependency order, under which environment, and whether previous work can be reused safely.

The useful mental model is an executable dependency graph with a correctness contract. Nodes represent files, targets, or actions. Edges represent required inputs. Given a requested target, the build system finds the relevant transitive graph, schedules ready actions, and produces outputs. Speed comes from skipping or reusing work; correctness comes from knowing every input that can affect each action.

Why build systems exist

Direct compiler commands work for a small program. As a project grows, the real problem is coordination:

generated files must exist before compilation;
headers, modules, schemas, and resources create dependencies;
different platforms and configurations need different toolchains or flags;
many independent actions can run in parallel;
a small edit should not force an unrelated rebuild;
clean and incremental builds should agree;
developers and CI should invoke the same build definition;
release outputs need a traceable, reproducible path from source.

GNU Make expresses targets, prerequisites, and recipes, then rebuilds targets when prerequisites require it. Modern artifact-based systems model named targets and actions more explicitly, often adding sandboxing, toolchain resolution, local and remote caches, and distributed execution. Ninja deliberately concentrates on fast low-level execution and is commonly fed by a higher-level generator such as CMake or GN.

The graph model

parser.c ──> parser.o ──┐
parser.h ───────────────┤
                        ├─> application
main.c ────> main.o ────┤
config.h ───────────────┘

Each output should declare the inputs that can change it. A target's direct dependencies are adjacent inputs; its transitive dependencies are everything reachable through the graph. The graph should be acyclic for ordinary build execution: a cycle means no target in that cycle can become ready from its dependencies.

There are really two graphs:

the declared graph, which the build system can see;
the actual graph, which the tools and code truly depend on.

Correctness requires every actual dependency to be declared. A missing edge produces under-building: an input changes but the affected action is incorrectly reused. Excess edges produce over-building: unrelated changes invalidate work. The ideal graph is precise enough for correctness and narrow enough for useful incrementality.

Rules, targets, actions, and configurations

Terminology varies, but the recurring pieces are:

target — a requested logical or file output;
rule — a reusable definition of how a class of targets is built;
action — a concrete command or tool invocation with resolved inputs and outputs;
toolchain — compiler, linker, interpreter, SDK, and related tools selected for a platform;
configuration — choices such as platform, architecture, optimization, features, and debug mode;
workspace or package — a boundary for naming and organizing targets.

Configurations are inputs. If a compiler flag, environment variable, tool version, or target platform can alter output, changing it must invalidate the affected action.

Incremental builds

An incremental build reuses outputs whose relevant inputs and action definition have not changed. Systems determine freshness in different ways:

timestamps, as in traditional file-oriented workflows;
stored dependency and command metadata;
content digests of inputs, tools, environment, and action parameters;
discovered dependencies emitted by compilers or actions.

Incrementality is valid only when the cache key represents the real action. Forgetting an input can make a fast build wrong. Including irrelevant inputs makes a correct build unnecessarily slow. Deleting the output directory is a diagnostic tool for graph defects, not a sustainable build strategy.

Parallel and distributed execution

Independent ready actions can run concurrently. The graph determines safe ordering; the scheduler determines resource use. Hidden dependencies and undeclared shared state often appear as intermittent parallel-build failures because execution order changes.

Remote execution sends actions to workers; remote caching reuses outputs produced elsewhere. They are distinct capabilities. Both require portable action descriptions and controlled inputs. If an action reads an undeclared file from a developer's home directory, depends on the host PATH, or contacts an unrecorded network service, another worker cannot reliably reproduce it.

Hermeticity

A hermetic build is isolated from incidental host state and depends on declared, controlled inputs: source, dependencies, tools, configuration, and permitted environment data. Bazel describes hermetic builds as insensitive to libraries and software installed on the local or remote host, using specified tool and dependency versions instead.

Hermeticity improves cache correctness and portability, but it is not identical to reproducibility. A build may be hermetic yet nondeterministic because it embeds the current time or iterates over unordered inputs. A build may happen to reproduce on two similar hosts without being hermetic. The concepts reinforce each other but answer different questions:

hermetic — are all influential inputs controlled?
deterministic — does one input set always produce the same output?
reproducible — can independent builds of the declared inputs produce bit-for-bit identical specified outputs?

The SOURCE_DATE_EPOCH specification gives tools a deterministic timestamp derived from source instead of the current build time, addressing one common source of output variation.

Caches are correctness mechanisms

A cache maps an action key to outputs. A safe key includes everything that can influence the action: input contents, executable and toolchain, arguments, relevant environment, platform properties, and sometimes dependency-discovery results.

action key = hash(inputs + tools + arguments + relevant environment + platform)

A cache hit is a claim that recomputation would produce equivalent outputs. Poisoned, underspecified, or cross-trust cache entries can distribute incorrect outputs at impressive speed. Separate trust domains where necessary, authenticate cache writes, verify content, and retain enough metadata to explain why an entry was reused.

Build system versus adjacent tools

A compiler translates a language; the build system coordinates compilers and other actions.
A package manager resolves and installs dependencies; a build system may integrate one but still needs a declared dependency model.
A CI service decides when and where automation runs; it should invoke the project's build entry point rather than redefine the build in pipeline YAML.
An artifact repository stores outputs after or between builds; it does not decide how source becomes those outputs.
A meta-build system generates another build system's input, as CMake can generate Ninja files.

Choosing and evolving a build system

Evaluate the workload rather than tool popularity:

supported languages, platforms, and toolchains;
dependency precision and generated-code support;
clean, incremental, and no-op latency;
parallelism, local caching, remote caching, and remote execution;
hermeticity and sandbox diagnostics;
IDE and editor integration;
inspectability of commands, graphs, and invalidation decisions;
migration cost and the ability to run old and new definitions during transition.

Small projects may value transparent rules and low ceremony. Large monorepos often need precise target graphs, strong isolation, shared caching, and distributed scheduling. A fast engine cannot compensate for a vague graph; a sophisticated graph can still be unusable if ordinary failures are opaque.

Failure modes

under-build — an output is reused after an undeclared input changed;
over-build — unrelated work reruns because dependencies are too broad;
works on my machine — the action reads host state not present in CI;
clean-build dependence — incremental state masks graph errors;
cache poisoning — an incorrect or untrusted output is stored under a reusable key;
configuration drift — CI, IDE, and local commands do not describe the same build;
nondeterminism — time, randomness, locale, filesystem order, or network data changes output;
oversubscription — parallel actions compete for memory, I/O, licenses, or nested worker pools.

A practical adoption path

Define one canonical command for a clean build and its requested outputs.
Model targets and direct dependencies; include generated inputs and toolchains.
Compare clean and incremental results after representative edits.
Expose the actual commands, graph, timing, and invalidation reasons.
Isolate actions from undeclared host state and network access.
Add local caching, then shared caching, only after action keys are trustworthy.
Normalize nondeterministic inputs and verify reproducibility independently.
Optimize the critical path rather than merely counting actions.