Jie Ma, Ningyu He, and colleagues (Beihang University, PolyU, Amber Group, HUST, Zhongguancun Laboratory) point out a blind spot in Ethereum security research: about 58% of it studies smart contracts, the code users can see, while the client software that runs the network underneath, the "infra under the water", gets far less scrutiny, despite a shallow-copy bug once forking the chain and taking down Infura, MetaMask, and Binance in one incident. Ethereum deliberately runs on client diversity: 6+ independent execution-layer (EL) clients and 6+ consensus-layer (CL) clients, written in different languages, each implementing the same specification. That diversity is a resilience feature, and it's also a perfect setup for differential testing: send the same input to many independently-implemented clients and treat any disagreement as a potential bug. The team built two LLM-assisted differential testers, OpDiffer (EVM opcode-level semantics, ISSTA'25) and APIDiffer (JSON-RPC and Beacon-API responses, OOPSLA'26), and used them to find 98 real bugs across all 11 mainstream Ethereum clients, 86.7% already confirmed or fixed, including a node-crashing nil-pointer bug, a client returning the wrong JSON-RPC error code, and a specification error that could cause valid requests to be rejected network-wide.
Post 15 of my Black Hat Asia 2026 series. This is defensive, disclosed, peer-reviewed testing research (ISSTA'25, OOPSLA'26), so I go into more technical depth than the offensive talks in this series.
1. The part of Ethereum security nobody's watching
Ethereum is the second-largest blockchain platform, hosting 90M+ deployed smart contracts and $52.8B+ locked in DeFi. In 2025 alone there were 310 security incidents totaling $1.7B exploited, an average of $5.7M per incident. Nearly all of the research attention chasing that number looks at smart contracts. The team's framing: a smart-contract bug can drain one protocol; a client bug can take down the network those protocols run on. Two real incidents make the stakes concrete:
- The Geth shallow-copy bug (CVE-2020-26241). Geth, the most widely used execution-layer client, unintentionally used a shallow copy instead of a deep copy when isolating EVM execution contexts, causing unintended aliasing between contexts that should have been independent. The result was a chain fork that took down Infura (a major RPC service provider) and affected downstream platforms including MetaMask and Binance.
- A 2025 Prysm consensus-client bug, surfacing during the Fusaka mainnet upgrade, could trigger repeated state replay and reconstruction, a resource-exhaustion condition that dropped network participation to 75%, dangerously close to the 66.6% threshold Ethereum needs to reach finality (the point at which a block is treated as irreversible).
2. How an Ethereum node actually works
Since the Merge, an Ethereum node runs a dual-engine architecture, two separate client processes cooperating over a local API:
| Layer | Role | Client-facing API |
|---|---|---|
| Consensus Layer (CL) client | Runs consensus logic, orders blocks, finalizes them, handles block gossip between peers | Beacon-API |
| Execution Layer (EL) client | Executes transactions, updates world state, runs the EVM for smart-contract execution | JSON-RPC |
The CL and EL talk to each other via the engine-api. Crucially, the Ethereum ecosystem deliberately supports multiple independent implementations of each layer, written in different languages (Go, Rust, Java, C#, TypeScript) by different teams, all conforming to the same public specification. This client diversity exists specifically to avoid a single point of failure, if one client has a critical bug, the network keeps running on the others. It's a genuinely good design decision, and it creates the exact precondition differential testing needs: many independently-built implementations that are all supposed to produce identical results.
Fig. 1 — With 6 CL clients × 5 EL clients (30 node combinations), every test input is naturally cross-checked 30 ways for free.
The team split the surface into two research targets: OpDiffer attacks the EVM inside the EL client (opcode-level semantics and smart-contract handling consistency), and APIDiffer attacks the Beacon-API and JSON-RPC surfaces both layers expose (response consistency to identical requests).
3. OpDiffer: making an LLM write minimal, valid EVM test cases
The EVM executes smart-contract bytecode as a sequence of opcodes (PUSH1, ADD, EXP, and so on), each with its own rules for stack management, gas accounting, semantic behavior, and state updates, rules the specification states in prose, not in a machine-checkable form. Testing every opcode's edge cases needs a large, diverse set of bytecode snippets that each exercise one opcode thoroughly. Two obvious approaches both fail:
| Approach | Why it falls short |
|---|---|
| Use real, deployed contract source code as seeds | Only ~0.87% of deployed contracts have public source, and what is public skews heavily toward one contract type (endless ERC-20 clones), giving poor diversity and rarely triggering opcode corner cases |
| Randomly mutate raw bytecode | Produces bytecode that's syntactically well-formed but semantically broken, e.g. a JUMP instruction targeting an address that isn't a valid JUMPDEST, which just crashes uninterestingly instead of exercising real logic |
OpDiffer's fix: use an LLM to generate a minimal, semantically valid bytecode snippet that correctly invokes a specific target opcode, grounded in the opcode's specification. From that seed, the tool extracts an interprocedural control-flow graph (ICFG) per the spec and augments the seed with additional arguments to cover more paths, systematic coverage expansion around a seed the LLM guaranteed was valid to begin with. This flips the traditional fuzzing trade-off: instead of generating huge volumes of low-quality inputs and hoping some are valid, generate a small number of guaranteed-valid, spec-grounded inputs and expand from there.
The second challenge is automated triage: with 30 node combinations producing output for every test case, how do you go from "these two clients disagree" to "here's the specific buggy line of code"? OpDiffer's pipeline:
- Inconsistency → Bug. Compare return data, gas usage, and state changes (like storage) across all clients running the same bytecode; any divergence flags a candidate bug.
- Bug → Opcode. Use the EVM's own execution log to identify which opcode was executing when the divergence appeared.
- Opcode → Implementation function. Use an LLM to map the opcode to the specific function in that client's source code that implements it.
- Root-cause synthesis. Combine the bug symptom, the opcode, and the implementation function into a root-cause report a developer can act on directly.
4. APIDiffer: catching the API mismatches users actually feel
The Ethereum client API, JSON-RPC methods like eth_getBalance and Beacon-API endpoints, is what every wallet, block explorer, and Web3 app actually talks to. A production incident makes the stakes vivid: Etherscan, the most widely used Ethereum block explorer, once showed two different values for the same transaction depending on which API it queried, 0.1 ETH via debug_transaction, 0.01 ETH via trace_transaction, a discrepancy the Geth team traced back to a faulty client API implementation in Erigon (with Reth found to have a similar bug). Two APIs that are supposed to describe the same on-chain fact disagreed by 10x, in a tool millions of people use to verify their own transactions.
Building a systematic tester for this surface has its own two challenges:
peer_id, which is tied to each node's own public key). Some differences are cosmetic, semantically identical error messages worded differently ("Unable to decode data" vs. "could not decode request body into..."). APIDiffer filters these with heuristic rules first, then uses an LLM as a judge to classify remaining candidates into three buckets: must-identical (should always match, a divergence is a clear bug), may-divergent (legitimately allowed to differ), and must-divergent (expected to differ by design). Only the first bucket's failures get reported as bugs.5. The results: 98 bugs, 11 clients, all of them
| Category | Count |
|---|---|
| Total bugs found | 98 (26 EVM bugs via OpDiffer, 72 API bugs via APIDiffer) |
| Confirmed or fixed by maintainers | 85 of 98 (86.7%) |
| Client coverage | All 11 mainstream EL and CL clients running on the Ethereum mainnet were affected by at least one finding |
Three case studies show the range of severity, from "the node crashes" to "the specification itself is wrong":
5.1 A nil pointer takes down the node (PREVRANDAO)
Geth's implementation of the PREVRANDAO opcode failed to check whether its Random context value was initialized before dereferencing it:
func opRandom(pc *uint64, evm *EVM, scope *ScopeContext) ([]byte, error) {
v := new(uint256.Int).SetBytes(evm.Context.Random.Bytes()) // no nil check
scope.Stack.push(v)
return nil, nil
}
When Random is nil, the EVM panics and the node crashes, a denial-of-service triggerable simply by executing this opcode under the right conditions. The bug wasn't confined to Geth either: it propagated to a downstream fork, causing an unrelated chain (Neo X, built on a Geth fork) to experience node shutdowns.
5.2 One client, wrong error shape (Besu)
For a malformed execution request, the spec is explicit: if any parameter is out of order, too short, duplicated, or null, the client MUST return JSON-RPC error -32602 (Invalid params). Four of five clients did exactly that. Besu instead returned a payload status of "INVALID", a different response shape entirely:
// besu (wrong)
{"id":1,"jsonrpc":"2.0","result":{"latestValidHash":null,"status":"INVALID","validationError":"…"}}
// geth / reth / erigon / neth (per spec)
{"error":{"code":-32602,"message":"Invalid params"},"id":1,"jsonrpc":"2.0"}
The root cause was misrouted exception handling, a specific exception type was caught and mapped to the wrong response path, bypassing the correct JSON-RPC error branch entirely. The impact: a caller (potentially the consensus layer itself) that correctly checks for a JSON-RPC error object would silently miss Besu's differently-shaped failure, "consensus confusion." Because Besu was the sole outlier among all major EL clients, this was a clean, high-confidence finding, and the report was significant enough to be brought to Ethereum's own PM RPC Standard Meeting, driving further cross-client specification refinement.
5.3 The specification itself was wrong
The most systemic finding wasn't in any client at all, it was in the Beacon-API specification document itself. Two errors: a field (blob_gas_used) introduced into the consensus spec in March 2024 was still missing from the Beacon-API spec a full year later, and a proof-size field was defined as 33 in the consensus spec but 32 in the Beacon-API spec, a direct contradiction between two documents that are supposed to describe the same system consistently.
Because every client and every third-party tool implements against the spec, a spec-level bug like this can propagate to every downstream implementation: valid requests get rejected because a client faithfully implements a spec that's internally wrong, and tooling built from the spec becomes literally incapable of generating requests the real network will accept. This is the highest-leverage class of bug the research found, one error, multiplied across every client that trusts the document. To their credit, Ethereum's process handled it well: the spec maintainer confirmed the report within 15 minutes and pushed a fix PR within an hour, and a developer from a separate consensus-client team independently reviewed their own implementation against the correction.
6. Why this matters beyond Ethereum
- For blockchain developers: client security deserves the same priority as smart-contract security. A single client bug (the Geth shallow-copy incident) can fork the chain and take down major infrastructure providers, an impact many individual smart-contract bugs never reach.
- For anyone building on a spec-conformant, multi-implementation system (not just blockchains, think multiple database drivers implementing one wire protocol, or multiple SDKs implementing one API): differential testing across implementations is a cost-effective way to find both implementation bugs and specification bugs, the latter category is invisible to single-implementation testing by definition.
- For LLM-assisted testing generally: the pattern here, use an LLM to generate guaranteed-valid, spec-grounded seeds rather than raw random mutations, and use an LLM as a judge to triage which of many observed differences are actually bugs, generalizes well past Ethereum to any domain with a formal-ish specification and multiple implementations.
- The attack surface is bigger than EVM/API. The authors explicitly flag other under-tested client subsystems as future targets: the consensus mechanism itself, state storage, the transaction pool (mempool), the P2P networking layer, the gas mechanism, and the software supply chain.
Is this a vulnerability disclosure or academic research?
Both. OpDiffer was published at ISSTA 2025 and APIDiffer at OOPSLA 2026 (peer-reviewed software engineering venues), and the 98 discovered bugs were responsibly reported to the relevant client teams, with 86.7% already confirmed or fixed at the time of the talk. This is defensive infrastructure-hardening work, not an offensive exploit release.
Why use an LLM instead of a traditional fuzzer for the EVM?
A traditional mutation-based fuzzer starting from raw bytecode mostly produces syntactically valid but semantically meaningless inputs (like a JUMP with no valid destination), which crash uninterestingly without exercising real opcode logic. An LLM grounded in the opcode specification can generate a minimal snippet that is guaranteed both syntactically and semantically valid, then be augmented, a much higher hit rate for interesting test cases per unit of test-generation effort.
How do they avoid false positives from legitimately different responses?
A layered filter: known-allowed-to-differ fields (like a node's own peer ID) are excluded by heuristic rules first, and an LLM-as-judge classifies remaining candidates as must-identical, may-divergent, or must-divergent, only must-identical mismatches are reported as bugs. This mirrors how a human reviewer would triage the same data, at scale.
What's the single most important finding here?
Arguably the Beacon-API specification bug: it shows that differential testing across implementations can find errors that no single implementation's own testing ever could, because every implementation was faithfully building to the same flawed source document. That's a class of bug invisible until you have multiple independent implementations to cross-check against each other and, implicitly, against the spec's own internal consistency.
Takeaways
- Ethereum client (infra) security is understudied relative to smart-contract security (~58% of research), despite client bugs having forked the chain before.
- Client diversity (6+ EL, 6+ CL implementations) is a resilience feature that doubles as a testing opportunity: differential testing compares outputs across independently-built, spec-conformant clients.
- OpDiffer uses an LLM to generate minimal, spec-valid EVM bytecode seeds (solving the diversity/validity trade-off that blocks both source-based and mutation-based seeding), then an LLM-assisted pipeline to localize root causes from observed inconsistencies.
- APIDiffer combines syntax- and semantics-aware request generation with an LLM-as-judge triage step to separate real API bugs from benign, allowed differences.
- 98 bugs found across all 11 mainstream clients, 86.7% confirmed/fixed, ranging from a node-crashing nil-pointer bug to a wrong error-response shape to a specification-level bug that could have propagated to every client that trusted it.
- The methodology, LLM-grounded valid-input generation + LLM-as-judge triage, applied to differential testing across spec-conformant implementations, generalizes well beyond blockchain.
References & further reading
- Black Hat Asia 2026 — this talk's official page — abstract, speaker bios, and slides/video when posted.
- Black Hat Asia 2026 — full Briefings schedule — every talk in the conference.
- ISSTA 2025 — the OpDiffer paper and artifact venue.
- OOPSLA 2026 — the APIDiffer paper and artifact venue.
- go-ethereum #30593 — the Etherscan debug_transaction / trace_transaction discrepancy.
- go-ethereum #29799 — the PREVRANDAO nil-pointer fix.
- Besu #8640 · beacon-APIs #514 — the response-shape and spec-field case studies.
- clientdiversity.org — live tracking of Ethereum's EL/CL client diversity.