Article 10.1's second pitfall — treating a paper's evaluation as a neutral, universal benchmark — deserves its own full treatment, because nearly every performance number this series has cited since Phase 6 (HotStuff's O(n), QuePaxa's 10x, article 6.5's Narwhal throughput claims) was measured under specific, chosen conditions that determine the result at least as much as the algorithm itself does. This article names the specific variables that dominate consensus benchmarks, the metrics beyond raw throughput that actually matter for a real decision, and a concrete checklist for reading any vendor's or paper's performance claim without being misled by it — the practical, hands-on companion to article 10.1's more abstract reading framework.
The variables that dominate the result, more than the algorithm does
Batch size is the single most consequential, least-often-highlighted variable in any consensus throughput number. Every algorithm this series has covered benefits enormously from batching multiple client requests into a single consensus round (article 5.3's Multi-Paxos fast path, article 9.3's FoundationDB proxy batching, Meerkat's own batching strategy from article 10.3) — a benchmark run with aggressive batching and one run with none can show a 10x or greater throughput difference for the identical underlying algorithm, entirely independent of which specific consensus protocol is being tested. Any throughput number reported without stating the batch size used is, functionally, an incomplete number.
Payload size matters similarly: a benchmark using tiny, few-byte commands (common in academic papers, because it isolates the consensus protocol's own overhead from application-level serialization cost) will report very different numbers than one using realistic, kilobyte-scale application payloads — and the gap between the two isn't fixed across algorithms, so a comparison run at one payload size doesn't necessarily hold at another.
Network topology and replica placement — single-datacenter versus WAN-spanning, article 9.5's physical latency floor — dominates any comparison involving geo-distributed systems specifically. A benchmark run entirely within one datacenter tells you almost nothing about how an algorithm behaves under the WAN conditions QuePaxa (10.2) and Meerkat (10.3) were specifically built to handle, and vice versa: a benchmark run under artificial WAN latency tells you little about same-datacenter performance where most Phase 8 deployments actually operate.
Replica count interacts directly with the communication-complexity differences this series traced from article 6.2 through 6.3 — PBFT's O(n²) versus HotStuff's O(n) only diverges meaningfully at larger replica counts; a 4-node benchmark comparison between the two will show a far smaller gap than a 100-node one, because the quadratic term simply hasn't grown large enough yet to dominate.
Fig 1 — Before comparing two throughput numbers across papers, confirm all four variables actually match.
Metrics beyond raw throughput that actually matter
Tail latency (p99, p99.9) is often more operationally important than throughput or even median latency, and it's exactly where the mechanisms this series has covered diverge most sharply from their median-case behavior — article 4.2's split-vote livelock, article 5.6's occasional log-repair round, article 6.2's view-change cost all show up specifically in the tail, not the median, and a benchmark reporting only average or median numbers can hide exactly the pathological cases article 10.2's QuePaxa paper was built to address. Recovery time after a failure — how long a cluster is degraded or unavailable following a leader crash, a partition, or (article 9.6's subject) a reconfiguration — is a metric most raw-throughput benchmarks simply don't report at all, despite being the metric that most directly determines a real system's actual availability guarantees during the failures article 1.1 opened this entire series by insisting are not rare, exceptional events but a routine, expected part of distributed systems operation.
| Metric | What it hides if omitted | Where this series covered why it matters |
|---|---|---|
| Throughput alone | Batching, payload size, replica count sensitivity | Article 5.3 (Multi-Paxos batching) |
| Median latency alone | Tail behavior during elections, view changes, repairs | Article 4.2, 6.2, 5.6 |
| Steady-state performance alone | Recovery time after failure or reconfiguration | Article 9.6 (reconfiguration), 10.2 (QuePaxa's whole point) |
| Best-case network conditions | Behavior under adversarial jitter, partition, DoS | Article 1.1, 10.2, 10.3 |
A checklist for reading any performance claim
- Batch size and payload size stated? If not, treat the throughput number as incomplete, not comparable to any other paper's number.
- Network topology matches your deployment? A same-datacenter benchmark says little about WAN behavior and vice versa (article 9.5).
- Replica count matches your target scale? Communication-complexity gaps (article 6.2 vs 6.3) only show up at scale.
- Tail latency reported, not just median or average? Tail behavior is where most of this series' subtle failure-handling mechanisms actually get exercised.
- Failure/recovery scenarios included, or only steady-state? A number measured only under perfect conditions tells you nothing about article 1.1's routine failure reality.
- Is the comparison baseline the current best-in-class, or an easy target? Beating naive HuggingFace-style unoptimized baselines (a pattern this series flagged even outside the consensus domain, in the PagedAttention article) is a much lower bar than beating a well-tuned Multi-Paxos or Raft.
FAQ
Is it ever reasonable to trust a single benchmark number without re-verifying it yourself?
For directional confidence (this algorithm is generally in the right performance tier for my use case), yes; for capacity planning or a specific production commitment, no — this series has consistently recommended benchmarking your own actual workload (echoing the pgBackRest guide's identical advice for a completely different domain) rather than trusting any single external number for a decision with real stakes.
Why do so many papers report only favorable conditions?
Not necessarily bad faith — a paper's evaluation section exists specifically to demonstrate its stated contribution works, and authors reasonably focus experiments on the conditions that contribution targets (QuePaxa's adversarial-condition focus, article 10.2, is itself an example of a paper choosing evaluation conditions deliberately aligned with its actual contribution). The reader's job, per this article's checklist, is recognizing what conditions weren't tested, not assuming bad intent.
Do open-source benchmarking suites exist for comparing these algorithms directly?
Yes, in various forms — several of the systems covered in Phase 8-9 (etcd, TiKV, YugabyteDB) publish their own benchmark suites and results, and academic papers throughout Phase 7's coverage typically include comparison code; the checklist in this article applies equally to reading and trusting (or not) any of these existing suites' results.
Takeaways
- Batch size, payload size, network topology, and replica count each independently swing consensus benchmark numbers 5-10x — a number reported without all four isn't directly comparable to any other.
- Tail latency and recovery time after failure are frequently more operationally important than throughput or median latency, and are exactly where this series' failure-handling mechanisms (elections, view changes, log repair) actually get exercised.
- A concrete six-point checklist — batch/payload size, topology match, replica-count match, tail latency, failure scenarios, honest baseline — turns article 10.1's abstract "read trade-offs honestly" advice into an actionable review process.
- The right response to any single performance number is benchmarking your own actual workload, not trusting any external number for a decision with real production stakes.
References & further reading
- cvam.sight — Consensus 10.1: How to Read a Consensus Paper — the abstract reading framework this article's checklist makes concrete and actionable.
- cvam.sight — pgBackRest with S3 — the same "benchmark your own workload" discipline, applied in a completely different domain.
- cvam.sight — Consensus 9.5: WAN Consensus and Read Optimization — the topology sensitivity this article's checklist directly builds on.