Build your own consensus algorithm.

This is not a recipe for production consensus. It is a learning exercise: build the smallest Raft-like replicated log that makes its safety assumptions visible. The goal is to discover why each rule exists by seeing which failure appears when you remove it.

Start with the contract, not the messages

A consensus protocol is easier to reason about when written as invariants—statements that must remain true through every interleaving, timeout, duplication, and crash.

Invariant	Plain meaning	Rule that protects it
Election safety	At most one leader exists in a term	Each node persists at most one vote per term
Log matching	If two logs share index and term, their prefixes agree	Append checks the preceding index and term
Leader completeness	A committed entry appears in later leaders	Voters reject candidates with less up-to-date logs
State-machine safety	No nodes apply different commands at one index	Apply only committed entries in index order

The minimum state

Every node has a volatile role—follower, candidate, or leader—and a monotonically increasing term. Three values must survive crashes: currentTerm, votedFor, and the log. If a node forgets its vote after reboot, it can vote twice in one term and create two leaders.

@dataclass
class Node:
    node_id: str
    role: str = "follower"
    current_term: int = 0       # persist before replying
    voted_for: str | None = None # persist before replying
    log: list[Entry] = field(default_factory=list) # persistent
    commit_index: int = -1      # volatile; reconstructed
    last_applied: int = -1

@dataclass(frozen=True)
class Entry:
    term: int
    command: str

One command, end to end

1. The client sends SET x=7 to the leader.
2. The leader appends (term=8, command=SET x=7) locally.
3. It sends AppendEntries with the previous index and term.
4. Follower B verifies the prefix, persists the entry, and acknowledges. C is delayed.
5. A and B form a majority, so the leader advances commitIndex.
6. The leader applies the command and replies. Future heartbeats tell C the committed index.

Election pseudocode

def on_election_timeout(node):
    node.role = "candidate"
    node.current_term += 1
    persist(node.current_term)
    node.voted_for = node.node_id
    persist(node.voted_for)
    votes = 1
    broadcast(RequestVote(
        term=node.current_term,
        candidate=node.node_id,
        last_index=len(node.log) - 1,
        last_term=node.log[-1].term if node.log else 0,
    ))

def may_vote(node, request):
    log_is_current = (request.last_term, request.last_index) >= last_log_id(node)
    vote_available = node.voted_for in (None, request.candidate)
    return request.term >= node.current_term and vote_available and log_is_current

The lexicographic last-log comparison is doing safety work: a candidate with a higher last term is newer; when terms tie, the longer log is newer. Without this check, a stale candidate can win and overwrite an entry a prior majority committed.

The prefix check is the repair mechanism

def append_entries(node, req):
    if req.term < node.current_term:
        return Reject(node.current_term)
    if req.prev_index >= 0:
        if req.prev_index >= len(node.log):
            return Reject(node.current_term)
        if node.log[req.prev_index].term != req.prev_term:
            return Reject(node.current_term)
    node.log = merge_from(node.log, req.prev_index + 1, req.entries)
    persist(node.log)
    node.commit_index = min(req.leader_commit, len(node.log) - 1)
    return Accept(node.current_term)

On rejection, the leader moves its nextIndex backward and retries until the follower finds a matching prefix. Then the leader overwrites only the divergent suffix. That is how a formerly isolated leader's uncommitted entries disappear without violating committed history.

Failure walkthrough: the leader crashes after one ACK

A stores entry 12, sends it to B, receives B's durable ACK, and commits. Before C receives it, A crashes. B's log is now the most up to date. Because any election majority overlaps the old commit majority, and because voters require an up-to-date candidate, a leader missing entry 12 cannot win. B becomes leader and repairs C.

Now change one fact: A sent the entry only to itself and crashed before any follower stored it. The entry was never committed. B or C may lead without it, and A must discard it when repaired. Client code therefore needs request IDs and retry semantics: a timeout does not tell the client whether a command committed just before the reply was lost.

Define the model boundary before writing code

Our teaching cluster has three fixed members, reliable local disks, authenticated point-to-point messages, and crash faults. Messages may be delayed, duplicated, reordered, or lost. Processes may stop and restart from persisted state. Nodes do not forge another node's identity or send intentionally contradictory messages. Timing eventually becomes favorable long enough to elect a leader and replicate entries, but no timeout proves that a peer failed.

This paragraph is part of the algorithm. Change authenticated membership to open membership and Sybil resistance appears. Change crash faults to Byzantine faults and signatures, larger quorums, and equivocation handling appear. Let disks acknowledge before durable write and a majority no longer means durable majority. A protocol proof is conditional on its model; code inherits those conditions.

Message contracts and idempotency

RequestVote carries candidate term, identity, last log index, and last log term. The response carries responder term and vote result. AppendEntries carries leader term, leader identity, preceding index and term, zero or more entries, and leader commit index. Its response carries current term and success, optionally a conflict hint for faster repair.

Every handler first compares terms. A lower-term request is rejected. A higher term makes the receiver persist the new term, clear its vote, and become follower before other processing. This common prelude prevents an old leader from continuing after learning a newer election exists.

Messages must be safe to repeat. A follower receiving the same append twice should not duplicate entries. A candidate receiving duplicate vote responses counts a voter once. Network libraries frequently retry after timeout, and a reply can be lost after the operation succeeded. Idempotency belongs in the protocol handler.

Timers create opportunities, not truth

Followers reset an election timer when they receive valid leader communication or grant a vote. A timeout makes a node a candidate; it does not prove the leader dead. Randomizing timeouts reduces repeated split votes because candidates are unlikely to start simultaneously. Heartbeats are empty AppendEntries messages and therefore exercise the same term and prefix authority as replication.

Choose timeouts from measured broadcast and storage latency, including tail behavior. The election timeout should comfortably exceed normal heartbeat processing. Too short produces needless elections during pauses; too long extends recovery. In production, stop-the-world runtime pauses, disk stalls, overloaded event loops, and virtual-machine suspension matter as much as network RTT.

Safety must not depend on the numeric timeout. Freeze a node for an hour, deliver an old heartbeat, or fire every timer together: the cluster may stop making progress, but it must not commit conflicting entries.

Persistence order is protocol order

A node receiving a higher term persists it before responding. A voter persists votedFor before granting the vote. A follower persists appended log entries before acknowledging success. If response precedes persistence, the network can carry evidence of a fact that disappears after crash.

Imagine B sends a successful append reply while data remains only in a write cache. Leader A counts A+B as a majority, commits, and replies to the client. Power loss removes B's entry, then A fails. C and rebooted B elect a leader without the supposedly committed command. The algorithm was correct; the storage contract was false.

Real durability depends on filesystem, database, hardware cache, barriers, cloud volume semantics, and error handling. write() may not mean stable media. An educational simulator can model persist() as an atomic event, but production code must specify what that event means.

Leader replication state

For every follower, a leader tracks nextIndex, the next log entry to send, and matchIndex, the highest known replicated index. On election, nextIndex starts after the leader's last entry. Rejections move it backward; successes move it forward. Conflict-term hints can skip entire incompatible terms instead of retrying one index at a time.

The leader advances commit index when a majority's matchIndex reaches an entry from the leader's current term. That current-term restriction is subtle. Counting an older-term entry directly can produce unsafe reasoning under particular election histories. Once a current-term entry commits, all preceding entries become committed through log order.

def maybe_advance_commit(leader):
    for index in range(len(leader.log) - 1, leader.commit_index, -1):
        replicated = 1 + sum(
            peer.match_index >= index for peer in leader.followers
        )
        if replicated >= majority(leader.cluster_size):
            if leader.log[index].term == leader.current_term:
                leader.commit_index = index
                apply_committed(leader)
                return

Reads are not automatically linearizable

A leader can be partitioned from the majority without immediately noticing. If it serves a local read, the answer may be stale while another leader commits newer writes. Linearizable reads require proving current leadership relative to a quorum or using a safe lease with explicit timing assumptions.

A read-index approach confirms leadership through quorum communication, then waits until the local state machine has applied through the confirmed commit index. A no-op entry committed in the current term can establish authority after election. Follower reads are stale unless routed through a protocol that establishes a safe read point.

Expose semantics in the API: linearizable read, bounded-stale read, or local stale read. Calling them all GET hides meaningful cost and behavior.

Snapshots and log compaction

An ever-growing log eventually exhausts disk and makes restart slow. A snapshot captures state machine state through an included index and term. After the snapshot is durable, earlier log entries can be discarded. The included term remains necessary for future prefix checks.

A lagging follower may need the snapshot because the leader no longer retains its required prefix. Snapshot installation must be chunked, checksummed, resumable, and atomic. The follower cannot expose half-installed state. After installation it keeps any suffix that legitimately follows the snapshot boundary and resumes normal append.

Application snapshot and consensus metadata must correspond to the same applied index. Taking a database backup while commands continue, then labeling it with a later index, creates state that never existed.

Membership changes need overlapping authority

Changing directly from configuration {A,B,C} to {C,D,E} is unsafe because A+B can form an old majority while D+E forms a new majority; the groups do not overlap. Joint consensus commits an intermediate configuration requiring majorities of old and new sets, carrying authority across the transition.

Simpler Raft variants add or remove one server at a time under constraints. Whichever mechanism is used, membership is a replicated state transition, not an edit to local configuration files. Learners or non-voting replicas can catch up before becoming voters.

Operationally, remove a failed node only after understanding whether it may return with stale credentials and state. Identity and transport authorization must follow membership so a removed process cannot continue impersonating a voter.

Client retries and exactly-once effects

A client sends request 42, the cluster commits it, and the leader crashes before replying. The client cannot distinguish that history from a request lost before commit. It must retry. Without deduplication, the command may execute twice.

Give each client a stable ID and monotonic request number. Replicate the latest processed number and cached result as state-machine data. A repeated request returns the prior result without re-executing. This provides exactly-once application of commands within the replicated state machine under the defined retention rules.

External side effects remain harder. If applying an entry sends email or charges a card, crash can occur between the external effect and recording completion. Use an outbox committed in replicated state, and let an idempotent worker deliver effects by stable ID. Consensus cannot atomically control an arbitrary outside system.

Separate protocol, transport, storage, and state machine

The protocol core should consume events and emit actions: persist state, send message, reset timer, or apply command. Transport handles serialization and authentication. Storage implements atomic durable transitions. The state machine interprets committed commands. A deterministic core is easier to simulate than callbacks mixing sockets, clocks, and disk.

Use a single-threaded logical event loop initially. Concurrency optimizations can come after invariants pass. Tag logs with node, term, role, index, peer, and message ID. Record state transitions rather than only errors.

Backpressure is mandatory. Bound pending proposals, per-peer replication windows, message sizes, and snapshot concurrency. A correct protocol can still crash from memory exhaustion when a client submits faster than followers persist.

Security is outside the proof but inside the product

Mutual authentication prevents an outsider from voting as a member. Authorize peer identity against the committed membership. Protect private keys, rotate certificates without losing quorum, and bind messages to cluster identity to prevent cross-cluster replay.

Raft does not tolerate a member lying. A compromised node can corrupt application requests or violate message rules, though a crash-fault majority may limit some damage accidentally. If malicious members are in scope, choose a BFT design rather than adding signatures and calling Raft Byzantine-safe.

Administrative endpoints can bypass consensus if they mutate local state. Restrict them to diagnostics or route changes through replicated commands. Audit membership and snapshot installation.

Observe the safety story

Expose current term, role, leader, commit index, applied index, last log index, per-follower match index, election count, proposal latency, append rejection reason, fsync latency, snapshot progress, and pending client requests. Alert on repeated elections, growing apply lag, a member unable to catch up, or quorum with no remaining redundancy.

A dashboard should distinguish availability from fault tolerance. A three-node cluster with two healthy nodes is serving but cannot tolerate another failure. Show commit latency by storage and network phase. Preserve structured event traces around elections for postmortems.

Learn by removing one rule

Remove	Counterexample
Persist vote before reply	Crash, forget, vote twice, elect two leaders in one term
Up-to-date log voting check	Stale candidate wins and overwrites committed entry
Previous index/term check	Follower accepts suffix on incompatible history
Majority before reply	Leader-only entry disappears after crash
Term comparison	Old leader continues influencing newer term
Request deduplication	Lost reply causes repeated business action

Encode each counterexample as a deterministic regression. The fastest way to understand a safety rule is to watch the smallest execution that fails without it.

From tests to formal reasoning

Unit tests cover handlers; simulation explores schedules; property tests generate command histories; model checking explores state transitions exhaustively within bounds. TLA+ or another specification language forces state, actions, and invariants into precise form. A model is smaller than production code but can reveal conceptual errors before implementation detail hides them.

Refinement is the next question: does code implement the modeled atomic actions despite threads, storage APIs, batching, and partial writes? Add assertions that connect runtime state to model concepts. Formal proof of an algorithm does not prove a particular implementation, and extensive tests do not prove all schedules. Use both.

Benchmark without hiding correctness

Measure throughput, median and tail latency, fsync cost, replication bytes, election recovery, snapshot catch-up, and behavior with one slow follower. State batch size, payload, durability mode, node count, and topology. A benchmark acknowledging before fsync is not comparable to one promising durable commit.

Load must include client retries and backpressure. Report steady state and failure intervals separately. A protocol sustaining high throughput while recovery queues grow without bound is borrowing from the future.

A disciplined build sequence

1. Write assumptions and four core invariants.
2. Build deterministic node state with no networking.
3. Implement term handling and persistent vote.
4. Add election with randomized logical timers.
5. Add append prefix checking and repair.
6. Add majority commit and ordered apply.
7. Add client IDs and deduplication.
8. Create a deterministic simulated network and crashable disk.
9. Add snapshots and membership only after core counterexamples pass.
10. Compare behavior against the Raft paper and a mature implementation.

Build a deterministic simulator

A simulator replaces sockets, threads, real clocks, and disks with an event queue. Each event has a logical time, unique sequence, destination, and payload. The scheduler selects the next event deterministically from a seed. Network rules may drop, duplicate, delay, or reorder messages. Disk persistence becomes an explicit event that can complete or be interrupted by crash.

Node code must not call the system clock or random generator directly. It asks an injected runtime to set timers or choose randomized durations. The simulator records every input and emitted action. When an invariant fails, print the seed and a minimized trace.

class Sim:
    def send(self, src, dst, message):
        if self.partitioned(src, dst): return
        delay = self.rng.randint(1, self.max_delay)
        self.schedule(delay, Deliver(dst, clone(message)))

    def crash(self, node_id):
        self.nodes[node_id].volatile = FreshVolatileState()
        self.nodes[node_id].running = False

    def restart(self, node_id):
        self.nodes[node_id] = Node.from_disk(self.disks[node_id])
        self.nodes[node_id].running = True

Model network partitions as directional sets because real failures can be asymmetric. Model message duplication and old messages delivered after healing. A protocol that works only on clean bidirectional partitions has not faced the interesting schedules.

Check invariants after every event

Do not wait until the end of a test. After every delivery, timeout, persistence completion, crash, and restart, run global assertions. Election safety scans leaders by term. State-machine safety compares applied commands at each index. Log matching compares any shared index/term and preceding prefixes. Committed entries must remain present in every later leader.

def check_state_machine_safety(cluster):
    decided = {}
    for node in cluster.nodes:
        for index, command in enumerate(node.applied):
            if index in decided:
                assert decided[index] == command, (
                    "different commands applied", index,
                    decided[index], command
                )
            decided[index] = command

Some properties require ghost state used only by the model, such as the set of entries ever acknowledged committed. Ghost state helps check claims without changing protocol behavior.

Check client histories, not only internal state

Record invocation and completion of every client operation. A linearizability checker asks whether completed operations can be placed in a sequential order consistent with real-time precedence and the state-machine specification. Internal indexes may look consistent while the public API serves a stale read incorrectly.

Include ambiguous operations whose client timed out. The checker may consider them either absent or taking effect at some legal point. Retries with the same request ID should return the same result. Use small key spaces to create conflicts and make exhaustive checking practical.

Failure lab: five traces to implement

Split vote

A and B become candidates in the same term; each receives one additional vote too late or messages divide so neither reaches majority. Randomized timers trigger a later term and one wins. Assert no two leaders in one term.

Old leader in minority

A leads A+B+C, then becomes isolated with one follower in a five-node cluster. The majority elects D. A may accept a client request locally but cannot commit. When healed it sees a higher term, steps down, and removes divergent suffix.

Committed entry on only a majority

A replicates entry x to B and commits, while C is offline. A crashes. Election rules must prevent C from defeating B with a stale log. B leads and repairs C.

Snapshot during catch-up

C is far behind. A compacts its needed entries and sends a snapshot. Interrupt transfer, retry chunks, install atomically, then resume appends. Assert applied state matches.

Membership transition under failure

Enter joint configuration, lose a member, and verify decisions require both old and new majorities. Complete transition only after the joint entry commits.

Make the disk adversarial

A simple in-memory dictionary is too kind. Model a write batch that can be absent or fully present after crash, then deliberately test weaker behavior if the intended storage API allows it. Inject fsync delay, full disk, checksum failure, and read error. Decide whether the process stops, retries, or marks itself unhealthy.

Never convert an I/O failure into a successful protocol response. If persistence cannot complete, the node cannot grant a vote or acknowledge an append that depends on it. Fail closed and expose health so orchestration does not keep routing leadership to broken storage.

Make the network realistic

TCP provides an ordered byte stream per connection, not exactly-once RPC. Connections reset, writes split, peers reconnect, and application messages repeat. Frame messages with length and limits. Authenticate peers. Reject oversized allocations before decoding.

Backpressure replication per follower. One slow follower should not block the leader's event loop or consume unbounded memory. Snapshot traffic should not starve heartbeats. Separate connection liveness from protocol leadership.

Design an honest API

A proposal API returns only after commit and apply if it claims linearizable command completion. It should return leader redirection or retryable status when the node is not leader. Include request ID and deadline. Deadline cancellation does not cancel a command that may already commit.

Expose read modes explicitly. Linearizable reads pay leadership confirmation. Stale reads expose applied index or age. Watch streams include revision so clients detect gaps and resume. Compaction errors tell clients to resnapshot rather than silently skipping events.

Snapshot format and validation

Include format version, cluster ID, membership or configuration reference, last included index and term, application checksum, chunk checksums, and creation metadata. Sign or authenticate snapshots when transported across trust boundaries. Validate before replacing current state.

Install to a temporary location, fsync data and directory metadata as required, then atomically switch. Preserve the old snapshot until new activation succeeds. Test upgrade and downgrade compatibility.

What the toy still lacks

Production systems batch proposals, pipeline replication, manage multiple disks or databases, encrypt transport, rotate keys, expose membership APIs, compact logs, stream snapshots, enforce quotas, prioritize internal traffic, and survive version skew. They implement pre-vote or check-quorum mechanisms to reduce disruptive elections and leader leases or read-index paths for reads.

They also need packaging, safe defaults, rolling upgrade matrices, backups, restore drills, metrics, tracing, alerting, capacity models, security response, and operator documentation. This gap is why writing a learning protocol should increase respect for mature libraries rather than confidence to replace them.

Protocol code review checklist

1. Does every message handler process higher terms first?
2. Are term, vote, and entries durable before dependent replies?
3. Can duplicate messages change state twice?
4. Does voting compare log term before index?
5. Can an old leader serve a linearizable read?
6. Does commit advancement enforce the current-term rule?
7. Are apply operations ordered and crash recoverable?
8. Can queues or messages grow without bound?
9. Are cluster and peer identities authenticated?
10. Can snapshot installation expose partial state?
11. Do retries use stable client IDs?
12. Does membership transition preserve quorum overlap?

Debugging guide by symptom

Election storms: inspect heartbeat handling, event-loop pauses, fsync tail latency, timer randomization, and term transitions. Do not begin by increasing timeouts. Follower never catches up: inspect nextIndex movement, conflict hints, snapshot boundary, and rejected prefix terms. Commit index stalls: inspect match indexes, quorum reachability, current-term rule, and persistence acknowledgements.

State machines differ: stop writes and preserve logs. Compare committed entries, apply order, snapshot index, and application determinism. Consensus agreement with different application outputs indicates nondeterministic state machine or snapshot corruption. Client sees duplicate action: trace request ID through commit, apply, result cache, and external outbox.

Two nodes claim leadership: terms may differ, which is normal during transition; two leaders in the same term violates election safety. Check durable votes, term persistence, identity duplication, and logs from every voter. A restored clone with the same node identity can appear as one voter in two places.

Extensions after the core model works

Add pre-vote so an isolated follower cannot inflate term immediately after reconnection. Add check-quorum so a leader steps down when it cannot contact a majority. Implement conflict-term optimization and measure repair messages. Add read-index and compare its latency with a committed no-op. Add bounded batching and observe throughput versus tail latency.

Add a non-voting learner, snapshot transfer, and single-node membership change. Add an outbox state machine with idempotent worker. Add metrics and render a timeline of terms, leaders, commit, and apply indexes. For each extension, write the invariant and counterexample before code.

Finally run two implementations against the same simulator interface and differential-test observable behavior. They need not elect the same leader, but completed client histories must satisfy the same specification.

Frequently asked implementation questions

Can the leader commit an entry after one follower acknowledges in a three-node cluster? Yes: leader plus follower is a majority, assuming both acknowledgements represent durable storage. Can a follower apply an uncommitted entry? No. It may store speculative suffix entries but applies only through leader commit index. Can a candidate vote for itself? Yes, after persisting its vote. Can two leaders exist temporarily? In different terms, yes; only the current majority-backed term can commit. In one term, election safety forbids it.

Why not delete an entry immediately when leadership changes? The new leader determines compatible history through prefix repair; local guesses can remove valid data. Why is cluster size odd? Even sizes usually do not increase tolerated failures over the preceding odd size but add coordination cost. Should a follower redirect clients? It may return known leader information, but clients must tolerate stale hints.

Worked trace with terms and indexes

Start with logs A=[(1,x),(1,y)], B=[(1,x),(1,y)], and C=[(1,x)]. A leads term 2 and appends z at index 2. Only A stores z before partition, so z is uncommitted. B and C form a majority, elect B in term 3, and append w at index 2. B replicates w to C and commits it.

When A returns, it sends an old term-2 heartbeat and is rejected with term 3. A steps down. B's append says previous index 1, term 1; A matches that prefix. At index 2 A has term-2 z while the leader has term-3 w, so merge deletes z and appends w. No committed entry was lost: z never had majority evidence, while w did.

Now change the first partition so A replicated z to B and committed before failing. B's log is more current and its vote rule prevents C from winning alone. Any election majority intersects A+B, and the up-to-date rule carries z forward. This pair of traces explains why local append is not commitment and why election compares log freshness.

More implementation questions

Why persist vote and term together? A crash between inconsistent updates can recreate illegal voting state; use an atomic durable transition. Why apply after commit? Speculative state might escape through reads or external effects and later be overwritten. Why keep term on each entry? Prefix compatibility and conflict repair need the epoch that created each position.

Can the leader delete its own suffix? After stepping down, the new legitimate leader may repair it. A node must not independently choose history. What if a majority has the entry but the leader does not know? The entry may become committed through a later leader; the client timeout remains ambiguous. Why are snapshots part of consensus? They replace log prefixes while preserving their included index, term, and state-machine result.

Keep the final implementation small enough that one engineer can print a complete event trace and explain every transition. Educational value falls when frameworks hide ordering. Add performance only after counterexamples remain impossible under millions of randomized schedules. The objective is not a GitHub star count; it is the ability to defend every acknowledgement, vote, truncation, and client response with an invariant.

Document every shortcut in the toy: fixed membership, in-memory transport, idealized atomic persistence, no Byzantine behavior, and simplified snapshots. A reader should never mistake omitted production machinery for unnecessary machinery. The model is successful when it clarifies why mature implementations contain features that first looked incidental.

Before finishing, demo one safe failure live: partition the leader, commit on the new majority, heal, and watch the old suffix repair. Then replay the exact seed. Repeatability turns a compelling animation into engineering evidence.

Keep the trace format stable enough to diff between versions; a protocol change should make its altered decisions visible rather than silently changing logs.

Test the model like an adversary

Fault	Assertion
Duplicate and reorder every RPC	No invariant depends on exactly-once delivery
Crash after every persistence boundary	Restart never creates two votes or loses an acknowledged entry
Partition old leader from majority	Minority leader cannot commit new entries
Heal divergent logs	Committed prefix remains; uncommitted suffix is repaired
Pause clocks and handlers	Timing changes liveness, never safety
Random history with reads/writes	Recorded behavior satisfies the claimed consistency model

Use deterministic seeds so a failure can be replayed. Then add model checking or property-based state-machine tests. Jepsen's consistency model map is a useful reminder that “consistent” is incomplete unless the exact model is named.

Takeaways

• Design from invariants outward.
• Persist term, vote, and log before acknowledging the messages that depend on them.
• A majority-stored entry and a client-visible committed result are related but not identical moments.
• Timeouts choose when to try; they must never determine what is safe.
• A teaching implementation is valuable precisely because it is small enough to break on purpose.

Further reading

• Ongaro and Ousterhout — In Search of an Understandable Consensus Algorithm
• Raft resources and visualizations
• Jepsen — distributed systems safety research