Distributed Systems Fundamentals
Distributed systems are collections of independent computers that appear to users as a single coherent system. This topic covers the fundamental challenges they face: network partitions, partial failures, clock synchronization, consistency models, and the impossibility results that constrain their design.
itDistributed systems, messaging, and integration | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic — Distributed Systems Fundamentals
A distributed system is what happens when independent computers try to present one coherent service while the network keeps receipts in invisible ink. Each node has local state. Messages take time. A timeout tells the caller it stopped waiting, which is notably less useful than knowing what the server did.
The old plan was pleasantly compact: one process, or one database, made the decision and returned an answer. Distribution earns its keep only when a concrete requirement needs more capacity, lower latency near users, continued service after failures, or separate ownership. It also adds partial failure, concurrent events, and clocks that refuse to appoint themselves the universal narrator.
First keep partitioning and replication apart. Partitioning splits data or work so different nodes handle different pieces. Replication keeps copies so a failure or a distant reader is less disastrous. Most real storage systems do both, which is efficient until someone calls a hot partition a replication problem and the meeting becomes longer than the outage.
Then choose a consistency contract for the invariant that matters. Linearizability makes operations appear atomic in real-time order. Eventual consistency permits temporary disagreement and later convergence. Stronger coordination can make an invariant easier to protect, but it can add latency or pause progress during failures. Weaker coordination leaves more work running, while the application owns stale reads and conflicting versions.
Consensus is the arrangement for one durable shared decision. A replicated state machine works when replicas apply the same deterministic commands in the same order. Raft uses leader election, log replication, and majorities for that job. A majority side can commit new decisions; a minority side refuses to do so, which is frustrating only until it prevents two histories from becoming true at once.
Retries contain the surprise. At-most-once delivery can lose work. At-least-once delivery can repeat it. The useful outcome is often neither label but idempotency: a stable key lets repeated attempts represent one intended mutation, with durable duplicate handling behind it.
Read the intro for the full map of failures, causality, consistency, quorums, and recovery. Use the slides to see the relationships at a glance, and keep the cheatsheet nearby while reviewing a design. The Field Notes cover the operational costs that hide behind the tidy diagrams. The quiz is where the vocabulary has to choose a side.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://lamport.azurewebsites.net/pubs/time-clocks.pdf
Supports
- Distributed processes communicating through messages
- Happened-before as a partial order defined by process order, message send and receive, and transitivity
- Concurrent events and logical clocks
- Ordered commands as a basis for distributed state machines
- https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf
Supports
- Fully asynchronous process and message model
- Crash-failure assumptions and admissible executions
- Impossibility of guaranteed deterministic consensus termination with one possible faulty process
- Separation between consensus safety conditions and termination
- https://lamport.azurewebsites.net/pubs/byz.pdf
Supports
- Byzantine failures as malfunctioning components that can send conflicting information
- Agreement requirements in the presence of arbitrary faulty behavior
- https://groups.csail.mit.edu/tds/papers/Gilbert/Brewer2.pdf
Supports
- CAP definitions of atomic consistency, availability, and partition tolerance
- Impossibility of guaranteeing both atomic consistency and availability during a partition
- Practical limits of the permanent pick-two interpretation
- https://raft.github.io/raft.pdf
Supports
- Consensus for replicated logs and replicated state machines
- Leader election, log replication, terms, commitment, and safety
- Majority quorum overlap and progress requirements
- Behavior of majority and minority sides during a partition
- https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
Supports
- Partitioning with consistent hashing and replicated key-value data
- Tunable replica, read, and write thresholds
- Eventual consistency, concurrent versions, conflict resolution, hinted handoff, and anti-entropy repair
- Availability, durability, and latency tradeoffs in a production storage system
- https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
Supports
- Timeouts as a bound on client waiting rather than proof that side effects did not occur
- Retries as a response to partial and transient failures
- Idempotent APIs for safely retrying operations with side effects
- High-percentile downstream latency as an input to timeout selection
- Retry amplification, bounded retry attempts, exponential backoff, and jitter
- https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/
Supports
- Stable client request identifiers for duplicate recognition
- Semantic equivalence across retry responses
- Atomic recording of an idempotency token and the associated mutation
- Parameter validation when one request identifier is reused for different intent
- https://research.google/pubs/spanner-googles-globally-distributed-database-2/
Supports
- Synchronous replication at global scale
- Externally consistent distributed transactions
- Explicit clock uncertainty through the TrueTime API
- Partitioned data managed through consensus-backed replication
- https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems/
Supports
- Coarse-grained distributed locking and reliable low-volume storage
- Availability and reliability as design goals for a production coordination service
- Operational lessons from large client populations
- https://research.google/pubs/mapreduce-simplified-data-processing-on-large-clusters/
Supports
- Parallel data processing across clusters
- Runtime-managed partitioning, scheduling, communication, and machine-failure handling
- Distribution hidden behind a narrow map and reduce programming model
- https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf
Supports
- Paxos protocol for agreement among processors
- Quorum-based preservation of shared decisions
- https://research.google/pubs/the-google-file-system/
Supports
- Fault-tolerant distributed file storage on clusters of unreliable components
- Recovery and failure handling as storage-system design concerns
- https://research.google/pubs/bigtable-a-distributed-storage-system-for-structured-data/
Supports
- Distributed structured storage organized around tablets
- Data placement and cluster-scale storage architecture
- https://engineering.fb.com/2023/05/16/data-infrastructure/mysql-raft-meta/
Supports
- Integrating Raft with MySQL as a replicated state machine
- Control-plane membership and leadership decisions in the replicated log
- Operational complexity of external promotion and failover automation
- https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/
Supports
- Queue backlog growth when processing stops while arrivals continue
- Backlog age and stale work as availability risks
