Distributed Systems Fundamentals

A distributed system is a group of independent computers that cooperate through messages. To a user, the group often looks like one service. Inside, work and state cross machine and network boundaries.

That boundary changes the engineering problem. A function call either returns or raises an error in one process. A network request can succeed, fail, arrive late, arrive twice, or complete after the caller stops waiting. The caller cannot always tell which case occurred.

Distributed systems exist because one machine is not always enough. You may need more capacity, lower latency near users, continued service during a machine failure, or data in several locations. Distribution can provide those properties. It also introduces partial failure, concurrency, and uncertainty about time.

Start with messages and state

Each node has local state and observes its own events. Nodes communicate by sending messages. There is no instant, free view of the whole system.

This gives you the first useful mental model:

local state + messages + failures + time assumptions = system behavior

Before choosing a database or consensus algorithm, write down those assumptions. Ask which nodes can fail. Ask whether messages can be delayed, duplicated, reordered, or lost. Ask what the system promises while a network link is broken.

The answers define a failure model. An algorithm is correct only relative to its failure model. A design that tolerates crash failures does not automatically tolerate a node that sends conflicting or malicious messages.

Time does not give you one obvious order

Physical clocks on different machines are not perfectly synchronized. More fundamentally, two events on different nodes may have no causal relationship.

Lamport's happened-before relation gives you a precise way to reason about causality. An event happens before a later event in the same process. Sending a message happens before receiving that message. The relation is transitive. Two events are concurrent when neither happened before the other.

A logical clock assigns timestamps that respect this causal order. It does not prove that one unrelated event physically occurred first. This distinction matters when you order writes, reconstruct incidents, or resolve conflicts.

Replication and partitioning solve different problems

Replication keeps copies of data or computation on multiple nodes. It can improve fault tolerance, read capacity, and geographic proximity. The cost is coordination. Replicas can disagree while updates travel or while nodes cannot communicate.

Partitioning splits a data set or workload into pieces. Each piece has an owner or replica set. Partitioning can increase capacity because different nodes handle different pieces. The cost is skew, movement during rebalancing, and coordination for operations that cross partitions.

Many real systems combine both patterns. They partition the data, then replicate each partition. Keep the concepts separate when you diagnose behavior. A hot partition is not a replication failure. A stale replica is not a partitioning strategy.

Consistency is a contract

Consistency describes which results clients may observe. It is not a single on-or-off feature.

Linearizability makes each operation appear to take effect atomically between its request and response. All clients observe behavior consistent with one real-time order. Eventual consistency allows replicas to diverge, while promising that they converge when updates stop and communication continues.

These contracts shape application behavior. Stronger coordination can make reasoning easier, but it may add latency or prevent progress during some failures. Weaker coordination can keep more operations available, but the application must handle stale values or conflicts.

The CAP result concerns a specific failure condition. When a network partition prevents groups of nodes from communicating, an atomic read-and-write service cannot guarantee both that every request eventually receives a response and that all responses preserve the atomic consistency contract. CAP is not a rule that every system permanently chooses two letters. It tells you what must give during a partition under the theorem's definitions.

Consensus creates one durable decision

Consensus lets nodes agree on a value despite failures. Replicated state machines use a sequence of agreed commands. If each replica starts in the same state and applies the same deterministic commands in the same order, the replicas produce the same state.

Raft organizes this work around leader election, log replication, and safety. A leader accepts commands and replicates log entries. Majorities matter because any two majorities overlap. That overlap helps preserve knowledge across leader changes.

Consensus does not make a service infinitely available. A Raft group normally needs a majority to make progress. Losing communication with a majority stops new committed decisions, even though a minority may still answer non-authoritative local questions.

The FLP impossibility result sets another boundary. In a fully asynchronous model, no deterministic consensus protocol can guarantee termination in every admissible execution when even one process may crash. This result does not say consensus is useless. It explains why practical protocols add timing assumptions, randomized choices, failure detectors, or qualified liveness guarantees.

Delivery is not processing

A client retry creates ambiguity. The original request may have failed before execution, failed after execution, or succeeded while its response was lost. Retrying without a stable operation identity can apply the same intent twice.

At-most-once delivery may lose work. At-least-once delivery may repeat work. An exactly-once business outcome needs more than a transport label. It usually depends on idempotent operations, durable deduplication, or an atomic transaction around the effect and its record.

Design the API around that ambiguity. Give retryable mutations an idempotency key. Define how long the key remains valid. Make conflict handling part of the contract.

Availability is more than replica count

More replicas can survive more failures, but they also create more work. Each replica consumes storage and network capacity. Synchronous replication adds nodes to the critical path. Asynchronous replication can lose recent acknowledged updates if the primary fails before propagation.

Quorums make the tradeoff explicit. A Dynamo-style system describes a replication factor and separate read and write thresholds. Different settings change latency, durability, and the chance that reads encounter concurrent versions. They do not remove the need for repair and conflict resolution.

Measure tail latency, not only averages. One request may wait for several downstream calls. Slow outliers can dominate the user-visible response even when average node latency looks healthy.

Design from invariants

An invariant is a property that must remain true. Examples include “an order is charged no more than once” or “a committed log entry is never replaced.” Start with the invariant, then choose the consistency and failure behavior needed to protect it.

Use this review sequence:

Define the operation and its invariant.
Name every stateful component and message boundary.
State the failure model and timing assumptions.
Choose the consistency contract for each operation.
Define retry, deduplication, and conflict behavior.
Decide which failures preserve safety and which preserve progress.
Test delayed, duplicated, reordered, and dropped messages.

Distribution is useful when its benefits justify these costs. A single process or one database may be the better design for a small workload with modest availability needs. Add distribution to meet a concrete requirement, not to make the architecture look advanced.

Where to go next

First, become fluent in causality, failure models, replication, partitioning, and consistency contracts. Next, study one consensus protocol closely. Trace leader changes, quorum decisions, and recovery by hand. Then practice failure injection and invariant-based testing. Finally, read production system papers to see how real designs combine these ideas under workload and operational constraints.