Distributed consensus, replication, and fault tolerance / personal lessonsLesson 1 of 17
Distributed systems · Lesson 1

What a distributed system must agree on

Separate state, commands, replicas, and the guarantees a client needs.

By the end: Describe a system as state transitions and state an invariant before choosing a protocol.

This course assumes programming familiarity, not prior distributed-systems study. Read the lessons in order: later safety arguments depend on earlier models. The goal is to follow expert explanations and reason through new failures, not to memorize every implementation detail. Each exercise includes a hint and a complete illustrated answer.

Begin with a concrete rule

A seat-booking service starts with seat 12 unassigned. Reserve(12, Ada) must either assign that seat to Ada or report that somebody else owns it. Its invariant is a statement that must remain true after every allowed operation: at most one successful owner exists for that seat. Two copies of the seat table do not automatically preserve that rule. If both see an empty seat, they can independently report success to different customers.

A state machine consists of stored state and rules for changing it when a command arrives. One machine can check and update the seat atomically, without another command entering between the check and update. Replicating the state machine means several machines execute the same agreed commands in the same order. Starting with equal state, deterministic rules then produce equal new state. This is an induction: equal inputs to one transition give equal outputs, which become the next transition’s inputs.

Derive the reservation transition before distributing it

A state machine needs a precise definition of state, commands, and results. For the seat example, let the state be either empty or an owner's name. A command supplies a proposed owner. When the seat is empty, the transition records that name and returns success. When it already has an owner, the transition leaves the state unchanged and returns a rejection. The result belongs to the transition too. Two replicas that store the same final owner but previously returned success to different buyers have not implemented the same client contract.

Work through a separate paper case with one remaining parking permit. The state initially records one available permit and no allocation. Command P asks to allocate it to Chen. The machine checks availability, records Chen's allocation, reduces availability to zero, and returns success. Command Q then asks for an allocation to Dev. Its check sees zero and returns rejection without changing the state. Reverse P and Q and Dev gets the permit instead. Either execution can satisfy the rule. The requirement is that the service produces results consistent with one allowed execution, rather than combining the successes from both alternatives.

Atomic means that another operation cannot observe or modify an intermediate state of this logical transition. Suppose the program instead exposes a separate availability check and allocation write. P checks that one permit remains. Before P writes, Q makes the same check. Both then allocate. Each check used a true statement when it ran, but neither check reserved the right to act on that statement later. Running a correct function inside each process does not make the larger interaction atomic. A local lock can protect threads sharing that process; it cannot restrain another machine unless that machine participates in a shared protocol.

The invariant should include enough state to express the promise. A count that never becomes negative is insufficient if allocations can appear without reducing the count. For this permit, require that the number of successful, unreleased allocations never exceeds one. If the model includes releases, define which owner may release and how a later allocation relates to the earlier one. Otherwise an implementation might reject every valid release or let an unrelated client free the permit. Those are specification errors even if the original uniqueness condition still holds.

Deterministic execution also requires fixed inputs. A command that chooses its recipient by consulting each replica's local clock can give different results on equal state. Choose the relevant input once, include it in the command, and define how every replica interprets it. Agreement about inputs and order is a prerequisite for using deterministic transitions, not a consequence of calling the same function on several machines. Later lessons examine how a protocol establishes that agreement despite failures.

Copies serve different purposes

Replication stores copies of the same logical data. Partitioning, also called sharding here, divides data across groups. Three replicas of one seat partition protect that partition against some failures; three different partitions do not give three copies of each seat. A cache is another copy, but it may deliberately lag and may not be authoritative. A backup preserves a recovery point, often outside the live replication path.

Copies help only against the failures they do not share. Three processes on one host can all disappear with that host. Three machines can all faithfully replicate an accidental deletion. Fault tolerance therefore requires a failure model: a list of things that may fail and things the argument assumes remain trustworthy. Most of this course assumes crash/recovery failures, non-malicious participants, messages that can be delayed or lost, and durable storage that survives the stated process crashes. We change those assumptions explicitly near the end.

Choose the boundary of replicated authority

Start a deployment sketch by writing which data belongs to one authoritative decision. Suppose a venue has seats 12 and 13. Keeping three replicas of the table means that each replica can contain both seat records. Splitting seat 12 into one group and seat 13 into another is partitioning. Replicating each partition three times gives six stored copies of individual seat records across two groups, not six authorities for every seat. Replica count has meaning only after the data boundary and membership are known.

A request for either seat can be handled within its own group if the rule concerns only that seat. A request to reserve both seats together changes the boundary. Two independent groups might each decide correctly while one reservation succeeds and the other fails. If the client was promised both or neither, the groups need an additional coordination procedure. We will examine transactions later. For now, identify whether the invariant belongs to one state transition or relates transitions in separate groups. Correct replication inside each group cannot by itself establish a relationship between them.

A display cache has another role. It can copy the seat table to answer browsing requests without promising that every displayed seat remains free. Imagine a cache still showing seat 12 as available after the authoritative service has assigned it. That is acceptable only if the display contract permits stale information and purchase performs the authoritative check. If the cache itself returns a final reservation success, it has crossed into decision making. Calling a copy a cache does not excuse a broken ownership promise. The operation it exposes determines which guarantee matters.

Now compare two failure cases on paper. In the first, the machine storing one live replica stops, while two other replicas and their disks remain reachable. Those surviving copies can potentially support continued service under an appropriate protocol. In the second, an authorized deletion reaches all three replicas. The replicas agree perfectly, but the records are gone from the live state. A retained earlier backup addresses the second case if it actually contains the records and the recovery procedure can restore them. Its contents may be older than the latest acknowledged reservation, so restoration needs an explicit account of lost updates.

Physical placement changes the first case. Three processes sharing one disk do not give three independent durable histories against disk loss. Three machines in one building may share power and network equipment. Conversely, geographically separated replicas may still share a software defect that deletes every copy. State the failure being tolerated before counting machines. The course's crash model assumes participants follow their rules and retain required durable records through the stated crashes. It does not assume that copying data repairs incorrect commands, lost credentials, or arbitrary dishonest behavior.

Work through the model

Starting stateFirst commandSecond commandResult
Seat emptyReserve Ada succeedsReserve Bo rejectsOne owner: Ada
Seat emptyReserve Bo succeedsReserve Ada rejectsOne owner: Bo
Two independent empty copiesA accepts AdaB accepts BoTwo successes violate the rule
The first two orders are both valid. Replicas need one shared order, not a universally preferred customer.

Read the table as alternative executions

The model above compares three cases; its rows are not successive moments of one booking. In the first row, Ada's command changes empty to Ada, so Bo's later command must reject. In the second row, Bo changes empty to Bo, and Ada must reject. There is no fairness rule in this example that gives one customer priority. The service may choose either order when the requests compete. Once it reports final results, however, those results must fit the same order. The third row combines decisions made from separate empty copies and therefore has no such valid explanation.

On paper, add a third buyer, Esi, to the first row. After Ada succeeds, Esi and Bo can arrive in either order and both must reject. The induction behind replicated execution now has concrete steps. All replicas start empty. Applying Ada's command gives each replica state Ada and result success. Applying either remaining command leaves Ada unchanged and returns rejection. Equal state survives each step because the transition and its ordered input agree. If one replica skips Ada's command, the induction no longer applies, even though its program remains deterministic.

Next change only the observation point. A follower has not received Ada's command yet and still displays empty. That observation alone does not say whether the service's contract failed. A browsing interface may permit stale displays. A final reservation endpoint cannot independently accept Esi from that empty state while preserving the stated one-owner promise. This separates internal disagreement during message delivery from incompatible completed client results. Many protocols allow replicas to be temporarily behind while restricting which state can justify an authoritative response.

A different counterexample uses identical final storage. A first machine reports success to Ada. A second reports success to Bo. Later, a reconciliation job retains only Ada everywhere. The final database now has one owner, but Bo still possesses a successful response that the contract said was final. Counting owners after reconciliation would miss the earlier safety violation. When defining a distributed object, include the history of permitted outcomes, not merely the shape of its latest table. Repairing a record and preserving a promise are different tasks.

Use the table to locate missing evidence rather than invent a network algorithm for it. For each valid row, identify the state before each command, the tested condition, the mutation, and the returned result. For the invalid row, identify the exact pair of successes that cannot coexist. The figure supplies fixed evidence, not a scheduler that explores crashes or elects leaders. Later interactive models isolate selected mechanisms under narrower assumptions. None of these small examples establishes that a real booking service satisfies its contract across every possible execution.

Specify an observable contract

Safety means something forbidden never happens, such as two successful owners for seat 12. Liveness means useful progress eventually happens under stated conditions. Refusing all reservations preserves the ownership safety rule but provides no useful service. A good design must state both the forbidden result and the conditions under which it makes progress.

A client does not inspect every replica. It observes requests, responses, and timeouts. Ask what a successful response promises, what reads may return afterward, and what a timeout leaves unknown. This course derives those contracts before examining Raft and Paxos. You will reason about small executions, not build a production consensus engine. Timing diagrams and log tables are invented examples; their spacing and playback speed are not latency measurements.

State progress conditions and the meaning of a response

Safety can be refuted by one finite bad history. If Ada and Bo each receive a final success for the same unreleased seat, later repair cannot make that history valid. Liveness has a different form. A request waiting now may complete later, so a finite period of silence does not by itself refute an eventual-progress claim. To make the claim useful, name the conditions under which waiting must end. For example, a service might require enough functioning replicas, working storage, continued message delivery, and a sufficiently stable coordinating process. These are assumptions to examine, not hidden guarantees of the network.

Consider a separate paper policy that sends every reservation to one designated machine and never replaces it. While that machine runs, serial execution can preserve the seat invariant. If it stops permanently, every future request waits permanently. The policy has not necessarily violated uniqueness, but it has failed any progress promise covering that failure. Automatically letting another machine take over raises a safety question instead. What evidence tells the replacement which previous commands must be preserved, and what prevents the old machine from continuing to authorize conflicting reservations? Replication protocols must answer those questions together.

Success needs a durability boundary. If a server reports success after changing only volatile memory, a power loss may remove the reservation. If the advertised contract says successful reservations survive that failure, the response came too early. Writing one durable copy protects against a process restart under a suitable storage contract but not necessarily against destruction of that disk. Maintaining additional copies can broaden protection, provided the protocol preserves the acknowledged decision when authority changes. The later replication lessons distinguish local storage, agreement, and execution rather than treating every acknowledgment as equivalent.

A rejection and a timeout convey different information. A valid rejection says that this attempted reservation did not obtain the seat under the service's rules. A timeout says the client stopped waiting without a conclusive response. The reservation may already exist. The next lesson develops this uncertainty from what messages the client can actually observe. Do not assign a fresh operation identity merely because a timer expired. Safe retry behavior requires an explicit contract for identifying one logical request and recovering its result, which a later lesson derives.

Finally, separate a successful decision from a successful business action. A replicated machine can agree to process a request and then consistently reject it because the seat is occupied. That rejection is a correct execution, not failure of agreement. To evaluate a proposed design, write a successful reservation history, a permitted rejection history, and a failure history that the service promises to survive. Mark what each client knows after every response. This produces concrete obligations for the protocols that follow without assuming that every request must succeed or that unavailable machines can always be bypassed.

Exercises

Does deterministic code suffice?

A and B both run the same correct reservation function, but process Ada and Bo in opposite orders. Must they agree?

Hint

The state transition depends on the input order as well as the code.

Worked answer
ReplicaCommand orderOwner
AAda, then BoAda
BBo, then AdaBo
Identical code with different ordered inputs gives different owners.

No. A can keep Ada while B keeps Bo. Determinism gives equal outputs for equal state and equal ordered input. It does not select that order. Both clients cannot be told their reservations are final.

Identify the missing protection

Three live replicas immediately apply every deletion. Are they a substitute for a separate recovery history?

Hint

Distinguish machine failure from a valid but mistaken command.

Worked answer
CopyBefore mistakeAfter replication
ASeat recordsDeleted
BSeat recordsDeleted
Backup taken earlierSeat recordsStill retained if isolated from deletion
A recovery copy protects a different failure case from a live replica.

No. Replication can preserve availability after one machine fails while distributing a mistaken deletion to every live copy. A retained backup or version history can support recovery to a point before that command, subject to retention and recovery testing.

Sources and further reading

The explanations and worked answers above stand alone. These references provide the original algorithms, assumptions, and deeper proofs; they are not required downloads.