When the Edge Can't Afford a Third Node — KubeCon Mumbai 2026 Day 2 Deep Dive

This is the dedicated follow-up that Day 1's Rook maintainers' talk (DD16) promised. Parth Arora (Software Engineer, IBM Storage) tackles a brutal edge constraint: Ceph wants three nodes for monitor quorum, but an edge site often has only two servers — and two monitors give you a quorum that tolerates zero failures. The answer is a floating monitor: a third Ceph mon (mon 'c') whose on-disk state is replicated in real time across both nodes with DRBD, so it can move to whichever node survives. To stop the failed node from waking up and corrupting state, Two-Node Fencing (TNF) uses Pacemaker STONITH to power-fence the dead node via its BMC over the RedFish API. The clever bit is the container workflow: an init container promotes DRBD to primary and a preStop hook demotes it — every other Ceph container runs unmodified.

It's the storage capstone of Day 2's edge thread, and it pairs with Day 1's Rook deep dive (DD16) and KubeEdge (DD15). If you read DD16, this is the chapter it pointed you to.

The edge constraint

Edge clusters on bare metal live under hard physical limits: space and power constraints, limited hardware availability, often only two servers — and yet the storage still needs to be highly available. A typical edge site is just Node-A and Node-B, side by side. You don't get to add a third rack unit because Ceph would like one.

Background — Rook and Ceph in one minute

To follow the fix you need the cast. Ceph is an open-source distributed storage platform that serves all three storage types from one cluster: block (RBD, ReadWriteOnce — like a virtual disk), file (CephFS, ReadWriteMany — a shared filesystem), and object (RGW, S3-compatible). Rook is the CNCF-graduated operator (created 2016, graduated 2020) that runs Ceph inside Kubernetes: it deploys, configures, and upgrades Ceph via an operator and CRDs, so apps consume storage through ordinary StorageClasses and PVCs and never touch Ceph directly.

Ceph itself is several daemons with distinct jobs, and knowing them makes the quorum problem concrete:

Daemon	Role
MON (monitor)	Holds the cluster map and consensus — the "brain." Needs quorum. This is the one the whole talk is about.
OSD (object storage daemon)	One per disk; stores the actual data and handles replication/recovery.
MGR (manager)	Metrics, dashboard, and orchestration helpers alongside the mons.
MDS / RGW	Metadata server for CephFS / the S3 object gateway — only if you use file/object.

Rook's architecture is three layers: the Rook operator deploys and manages Ceph; the Ceph-CSI driver dynamically provisions and mounts volumes into application pods; and Ceph itself is the data layer. The data path is notable — an app's RBD/CephFS mount talks to multiple OSDs directly via the kernel driver, so data throughput scales with disks. But every client first consults the MONs for the cluster map, which is why mon availability is the linchpin: lose mon quorum and even healthy OSDs become unreachable.

Why Ceph wants three nodes — the mon quorum

Ceph's monitors (MONs) are the brain of the cluster. Four properties matter:

Cluster state — MONs are the source of truth for the cluster map.
Quorum requirement — Ceph needs a majority of MONs to communicate and establish consensus.
Split-brain prevention — that majority rule is what keeps two halves of a partitioned cluster from both believing they're authoritative.
Persistence — a mon's critical metadata is written directly to local disk for durability.

Run the quorum maths and the problem is stark:

	3-node (standard)	2-node (edge)
MONs	3 unique (one per node)	2 unique only
Quorum needs	2 of 3 active	2 of 2 active
Failures tolerated	1 node	0 nodes

The trap with naive two-node Ceph. With two MONs, quorum is "both must be up." Lose one node and you lose quorum — the cluster goes read-only or down entirely. That's worse than useless for HA: you've doubled your hardware and still can't survive a single failure. The whole talk is about escaping this 0-failure box without a third physical node.

Why majority, and why odd numbers. The "majority" rule isn't arbitrary — it's the only way to guarantee that two halves of a network partition can't both think they're in charge. If you require strictly more than half, then at most one side can ever have it, so at most one side accepts writes. That's the entire defence against split-brain. It also explains the odd-number convention: 3 mons tolerate 1 loss (2 is a majority of 3), 5 tolerate 2, but 4 still only tolerate 1 (you need 3 of 4) — so the even count buys hardware without buying resilience. Two mons is the degenerate case: a "majority of 2" is 2, so any loss is fatal. The floating mon exists to turn a 2-node site into a genuine 3-mon majority system.

The idea — a floating monitor

The concept: run a third monitor that "floats" between the two nodes to provide a dynamic quorum. With mons a, b, and a floating c, you have three monitors and a real majority — but only two physical nodes. When a node dies, the floating mon relocates to the survivor, keeping quorum alive.

Fig 1 — the floating mon: a, b, and a floating c. When Node 2 crashes, mon c relocates to the survivor to hold quorum — while Node 2 is fenced so it can't rejoin and split-brain.

The failure scenario, step by step

The deck walked a non-graceful node shutdown:

Online — both nodes RUNNING; Node 1 holds mon a, Node 2 holds mon b plus the floating mon c. Three mons, quorum healthy.
Non-graceful shutdown — Node 2 crashes hard (power loss, kernel panic). Mons b and c on it go down; quorum is at risk.
I/O recovery — the floating mon migrates to Node 1 (its data already synced), restoring a 2-of-3 majority on the surviving node so applications keep reading and writing.
Fence the dead node — OCP/Pacemaker fences Node 2 to prevent split-brain before any relocation completes.

Fencing with Pacemaker (STONITH)

The dangerous moment isn't the crash — it's the recovery. If the failed node wakes back up and resumes operations without coordination, you get split-brain: two nodes both think they own the data. The deck was explicit about the three concerns:

Active continuity — the surviving node keeps serving I/O to applications during the outage.
Split-brain risk — arises if the failed node re-establishes connectivity and resumes without coordination.
Data consistency — OCP uses Pacemaker to fence the failed node via its Baseboard Management Controller (BMC) using the RedFish API.

STONITH, decoded. STONITH = "Shoot The Other Node In The Head." Rather than politely asking the failed node to stay down, Pacemaker reaches its out-of-band management controller (the BMC) over the RedFish API and powers it off — hard. Only once the dead node is provably dead does the survivor take over the floating mon. This is the load-bearing safety mechanism: you can relocate a monitor's identity safely only if you're certain the original can't come back and claim it too. This is what "Two-Node Fencing (TNF)" provides as an OpenShift feature.

Replicating the mon's disk — DRBD

For the floating mon to move, its on-disk state must already exist on both nodes. The floating-mon design requirements: it can run on either node, the mon metadata store on disk must be replicated between both, and it must work with hostNetworking: false. The key challenge the deck posed: how do you reliably replicate the data?

The answer is DRBD (Distributed Replicated Block Device) — open-source distributed replicated block storage for Linux, built for high performance and high availability. Think of it as RAID-1 over the network: a write to the local block device is synchronously mirrored to the peer node's device.

Fig 2 — DRBD mirrors the floating mon's data path (/var/lib/ceph/mon/c → /dev/drbd_c → LV) across both nodes in real time, so mon c can come up on whichever node survives.

In the full Rook/Ceph (ODF) 3-monitor "abc" configuration: mon a lives on Node 1 (its own LV_a), mon b on Node 2 (LV_b), and mon c is the highly-available floating mon whose /var/lib/ceph/mon/c sits on a DRBD device replicated between the nodes. Only mon c's state needs DRBD; a and b are ordinary node-local mons.

DRBD has its own split-brain — and that's why fencing comes first. DRBD allows exactly one node to be primary (writable) at a time; the other is secondary. If both nodes ever go primary on the same resource (each thinking the other is dead), they accept divergent writes and you get a DRBD split-brain that requires manual, lossy reconciliation. This is the deeper reason STONITH fencing is not optional and must complete before the floating mon relocates: the survivor can only safely promote DRBD to primary once it's certain the other node is powered off and cannot also be primary. Fencing isn't just protecting Ceph's quorum — it's protecting the block-replication layer underneath it. Two split-brain risks, one fence.

A subtlety worth calling out: the fence uses the node's BMC over RedFish specifically because in-band methods can't be trusted on a node that may be wedged. A crashed or partitioned node might not respond to SSH, an API call, or a graceful shutdown — but its baseboard management controller is a separate, always-on micro-controller with its own power and network, so Pacemaker can command a hard power-off out-of-band even when the host OS is unresponsive. That independence is exactly what makes the fence reliable: you're not asking the sick node to cooperate, you're cutting its power from outside. RedFish is just the standardized REST API for talking to that BMC.

The clever part — the container workflow

The elegance is that this is achieved with container lifecycle hooks, leaving Ceph itself untouched. The floating mon pod has a three-part workflow:

Stage	What it does
Init container	Unmount: handles leftovers from a failed pre-stop container. DRBD primary: promotes the DRBD resource to primary state on this node.
Main container	Mount DRBD disk: mounts the device to a hostpath and runs normal mon operations.
Floating-mon-shutdown	preStop hook: demotes DRBD to secondary and unmounts the hostpath.

Why this design is so clean. DRBD allows exactly one node to be "primary" (read-write) at a time. The init container claims primary before the mon starts; the preStop hook releases it on graceful shutdown. So the floating mon's data device is only ever writable on the node currently running it — which is precisely the invariant you need to move a stateful service safely. And the note that matters: all other containers run without modifications. The HA trick is bolted on at the pod boundary, not woven into Ceph. That's what makes it shippable in Rook.

Putting it together — TNF + Rook

The full picture: Rook runs a normal-looking 3-mon Ceph cluster on two nodes. Mons a and b are node-local; mon c floats, backed by DRBD-replicated storage. Pacemaker/TNF stands guard, ready to STONITH-fence a dead node via BMC/RedFish so the floating mon can relocate without split-brain. Applications see an ordinary, highly-available Ceph cluster — block, file, and object — on hardware that "should" only support a zero-failure-tolerant setup.

Limitations & honest trade-offs

This is a genuinely clever fit, but it's solving a constraint, not beating physics. What you're really buying is tolerance of one node failure on two nodes — and the caveats matter:

You still can't survive both nodes. Two nodes is two nodes; if the site loses power entirely, the cluster is down. This raises single-failure tolerance from zero to one — that's the win, no more.
Fencing must actually work. The whole safety model rests on STONITH succeeding. If the BMC is misconfigured, on the same power feed as the host, or unreachable, the fence fails and you're back to split-brain risk. The BMC/RedFish path is a hard dependency, not a nice-to-have.
DRBD is synchronous. Mirroring the mon's writes to the peer in real time adds latency to those writes and assumes a fast, reliable link between the two nodes. Across a flaky or high-latency interconnect, DRBD itself becomes the bottleneck (or risks its own split-brain).
Operational complexity goes up. You've added DRBD, Pacemaker, BMC fencing, and a lifecycle-hooked pod to what was "just Ceph." It ships in Rook, but there's more to understand and monitor when it misbehaves.
OSD data replication is separate. The floating mon solves quorum; your actual data still needs Ceph replication (size=2 across the two nodes) to survive a node loss. The mon trick keeps the cluster reachable; replication keeps the data present.

When this is the right tool. Reach for it when you genuinely cannot place a third node or even a lightweight third arbiter (a tie-breaker mon on a tiny box or a cloud VM is simpler if you can have one), and you need real HA at a two-server edge site — retail back-office, a cell site, a factory floor, a remote clinic. If you can add any third voting member, even a Raspberry-Pi-class arbiter, that's usually the lower-complexity answer. The floating-mon + TNF design is for the case where two boxes is a hard ceiling.

FAQ

Why can't I just run two monitors on two nodes?

Because Ceph quorum needs a majority. With two mons, quorum is 2-of-2, so losing either node loses quorum and the cluster stops serving. You tolerate zero failures — no HA at all. You need an odd number with a real majority, which is why the third (floating) mon exists.

What makes the third monitor "floating"?

Its on-disk state (/var/lib/ceph/mon/c) is continuously replicated to both nodes via DRBD, so it isn't tied to one machine. When a node fails, the floating mon can start on the survivor using the already-synced data, restoring a 2-of-3 majority.

What is STONITH / TNF and why is it required?

STONITH ("Shoot The Other Node In The Head") is Pacemaker power-fencing the failed node via its BMC over RedFish. Two-Node Fencing (TNF) packages this. It's required to prevent split-brain: only after the dead node is provably powered off can the survivor safely take over the floating mon — otherwise both nodes might claim the data.

What is DRBD doing here?

DRBD (Distributed Replicated Block Device) is network RAID-1 for Linux block devices. It synchronously mirrors writes to the floating mon's data device (/dev/drbd_c) across both nodes, so the mon's state exists on whichever node survives. Only one node holds the DRBD "primary" (writable) role at a time.

How much of Ceph had to change?

Almost none. The HA behaviour lives in the floating mon pod's lifecycle: an init container promotes DRBD to primary and mounts the device; a preStop hook demotes and unmounts. All other Ceph containers run unmodified — which is why it fits inside Rook.

Isn't a lightweight third "arbiter" mon simpler than all this?

Yes — if you can have one. A tie-breaker mon on a small third box or a cloud VM gives you an odd quorum with far less machinery. The floating-mon + DRBD + TNF design exists for the case where a third voting member genuinely isn't possible: no third device, no connectivity to a cloud arbiter, a hard two-server ceiling. Given that constraint, this is how you still get one-failure tolerance.

Does the floating mon protect my data, or just keep the cluster up?

Just the quorum — it keeps the cluster reachable through a node failure. Your actual data durability still comes from Ceph replicating objects across the two nodes' OSDs (replica size 2). The floating mon ensures clients can still reach a quorum and the surviving OSDs; replication ensures the data is physically present on the survivor.

What happens if fencing fails?

Then the survivor must not promote the floating mon, because it can't prove the other node is down — promoting anyway risks both nodes going DRBD-primary and Ceph-quorum-authoritative (split-brain). A correct setup treats a failed fence as a hard stop: stay degraded rather than risk divergence. That's why the BMC/RedFish path needs its own power and network and must be tested — the safety of the whole design depends on the fence being reliable.

Takeaways

Two-node Ceph naively tolerates zero failures — 2-of-2 quorum is the trap.
A floating third monitor restores a real majority on two physical nodes, relocating to whichever node survives.
DRBD makes the floating mon stateful-portable — its data device is mirrored across both nodes in real time.
STONITH fencing (TNF) is mandatory — power-fence the dead node via BMC/RedFish before relocating, or risk split-brain.
The HA is bolted on at the pod boundary — init promotes DRBD primary, preStop demotes; every other Ceph container is untouched, so it ships inside Rook.

Next in Day 2 — Scaling the Developer Frontier, the CNCF TAG DevEx roadmap for 2026.

References

KubeCon Mumbai 2026 — Day 2 index · the rest of Day 2
Day 1 DD16 — Rook: Storage for Kubernetes · the talk that pointed here
Rook · the operator running Ceph
DRBD · distributed replicated block device
Pacemaker · STONITH fencing

When the edge can't afford a third node — a two-node Ceph story.