Detailed explanation of MEV-Boost working principle and Ethereum fork selection rules

Author: Georgios Konstantopoulos, Mike Neuder

Compilation: Kxp, BlockBeats

Introduction

On April 2, malicious Ethereum network actors exploited a vulnerability in the MEV-Boost relay to steal $20 million from a single MEV seeker (see Flashbots report). Over the next few days, the developers addressed the vulnerability by releasing five patches. These patches, combined with network delays and validator policies, caused a brief fluctuation in the Ethereum network on April 6. Reorganizing blocks can be detrimental to the health of the network as they slow down the rate of block production and reduce settlement assurances.

In this post, with Seekers under attack and the network temporarily unstable, we explore the interplay between MEV-Boost and consensus, analyze the subtleties of Ethereum's proof-of-stake mechanism, and enumerate some possible ways forward path.

MEV-Boost and why it matters

MEV-Boost is a protocol designed by Flashbots and the community to mitigate the negative impact of Maximum Extractable Value (MEV) on the Ethereum network.

There are 3 actors in MEV-Boost:

1. Relay - Auctioneers who trust each other, connecting block producers and block builders

2. Constructors - complex entities that construct blocks to maximize MEV for themselves and the block creators

3. Block producers - Ethereum's proof-of-stake verifiers

The approximate sequence of events for each block is:

1. Builders create a block by receiving transactions from users, searchers, or other (private or public) order streams

2. The builder submits the block to the relay

3. The relay verifies the validity of the block and calculates the amount it pays to the block producer

4. The relayer sends a blank title and payment value to the block producer of the current slot

5. Block Producers evaluate all bids they have received and sign the blank header associated with the highest payment

6. The block producer sends this signed header back to the relay

7. Relays publish blocks using their native beacon nodes and return them to the block giver. Rewards are distributed to builders and proposers through transactions within blocks and block rewards.

A relay is a trusted third party that facilitates a fair exchange of block space by block producers and transaction ordering by builders for MEV extraction. Relays protect builders by protecting builders from MEV theft, preventing block producers from copying builder transactions to take MEV away without distributing it to the seekers/builders who found it. Relays protect block producers by confirming the validity of their blocks, processing hundreds of blocks per slot on their behalf, and ensuring the accuracy of block producer payments.

MEV-Boost is key protocol infrastructure as it enables all block producers to democratically access MEV without requiring a trust relationship with builders or searchers, which contributes to the long-term decentralization of Ethereum.

Ethereum's fork selection rules and MEV-Boost

Before discussing the attack and response, let's take a look at Ethereum's proof-of-stake mechanism and related fork selection rules. The fork choice rule allows the network to reach consensus on the chain head, as defined in the article "Ethereum post-merger reorganization":

"The fork choice rule is a function evaluated by the client, which takes as input the seen blocks and other information, and outputs the "canonical chain" to the client. The fork choice rule is important because there may be multiple valid chains to choose from (for example, if two competing blocks with the same parent block are published at the same time). "

Fork selection rules are also time-dependent, which has a significant impact on block generation.

slot and sub-slot cycle

In Ethereum's proof-of-stake mechanism, time is divided into slots every 12 seconds. The proof-of-stake algorithm randomly assigns a validator a license to propose a block within that slot; this validator is called a block producer. Within the same slot, other validators are assigned the task of validating (voting) for the chain-head applying the fork-choice rules according to their local views. This 12-second slot is subdivided into three phases, each costing 4 seconds.

The events that occur in the slot are as follows, where t=0 indicates the beginning of the slot.


During the slot, the most critical moment is the authentication deadline at t=4. If attesting validators do not see the block by the attestation deadline, they will vote for the previously accepted head of the chain (according to the fork choice rule). The earlier a block is proposed, the more time it has to propagate and thus accumulate more approvals (because more validators see it before the approval deadline).

From a network health perspective, the optimal time for a block release is t=0 (according to the specification). However, since block value increases monotonically over time, block producers have an incentive to delay the release of their blocks so that more MEV can accumulate. See Timing games in Proof-of-Stake and this discussion for details.

Previously, a block producer could publish a block after the certification deadline (even close to the end of the slot), as long as the next validator observed the block before building the block for its subsequent slot. This is due to the child block inheriting the weight of the parent block, and the fork selection rule terminates at the leaf node. Therefore, postponing block releases has no side effects. To help shift rational behavior (postponing block releases) to honest behavior (publishing on time), an "honest reorg" was implemented.

Block Producer Reward Mechanism and Honest Reorganization

Two new concepts are introduced into the consensus client that have a critical impact on the authentication deadline.

1. Block Producer Reward Mechanism - Aims to minimize rebalancing attacks by awarding block producers a fork selection "reward" equal to 40% of the full authentication weight. Importantly, this reward only lasts for the entire slot.

2. Honest reorganization - take advantage of block producer acceleration, allowing honest block producers to force reorganization of blocks with authentication weights below 20%. This is already implemented in Lighthouse and Prysm (starting with v4.0 - Capella release). This change is optional because it is a decision made by block producers and does not affect the behavior of authenticating validators. So we don't apply it to all clients at the same time, nor is it tied to any particular hard fork.

It should be noted that honest reorganization is avoided in some special cases:

1. During epoch boundary blocks

2. If the chain is not finalized

3. If the head of the chain is not the head of the slot before reorganization

Case 3 ensures that honest reorgs only remove one block from the chain, which acts as a circuit breaker, allowing the chain to continue producing blocks during periods of extreme network latency. It also reflects that proposers have less confidence in the perception of the network, as they cannot be sure whether their proposed enhanced blocks will be considered the norm.

The figure below shows how honest behavior can be changed to implement a restructuring strategy.


In this case, let b1 represent a late block. Due to the delay, b1 only has 19% of the proof weight of the nth slot. The remaining 81% of the proof weight is assigned to the parent block HEAD because many provers did not see b1 before the proof deadline.

If there is no honest reorganization, the proposer of slot n+1 will consider b1 as the head of the chain and build sub-block b2. The proposer made no effort to reorganize b1, even though it only has 19% proof weight. At slot n+1, b2 has the proposer's enhancement, assuming it is delivered on time, b2 will become the norm by accumulating a majority of proofs for that slot.

With Honest Reorganization, things are very different. Now, the proposers of slot n+1 see that 19% of b1's proof weights are below the reorganization threshold, so they build a block with HEAD as a parent and force b1 to reorganize. When we reach the proof deadline for slot n+1, the honest prover will compare the relative weights of b1 (19%) to b2 (40% from the proposer's boost). All clients implement the proposer enhancement, so b2 will be considered the head of the chain and will accumulate proofs for slot n+1.

Relay and Beacon node fixes for unbundling attacks

In the unbundling attack on April 2, the proposer exploited a relay vulnerability to send an invalid signature header to the relay. Over the next few days, the relay and core development teams released numerous software patches to mitigate the risk of repeated attacks. The five main changes are as follows:

Relay Changes: Check the database for known malicious proposers (used in production by Ultrasonic Relay only and has been removed). Checks if the relay has delivered the full block to a slot in the P2P network. Introduces a uniform random delay in the range 0-500ms before a block is published (removed from all relays).

Beacon node change (relay beacon nodes only): Validate beacon blocks before broadcasting. Check the network for false confirmations before publishing a block. The combination of these changes led to consensus instability, a problem exacerbated by the fact that a majority of validators now use the honest reorganization strategy described above.

UNINTENDED CONSEQUENCES

The five changes above all introduce delays in the hot path of relay block issuance, which increases the probability that relay blocks will be broadcast after the confirmation deadline. The diagram below shows the sequence of these five checks and how the delay introduced causes the block publication to exceed the confirmation deadline.

A large number of signed headers with a lag of t = 0 (e.g. t = 3) usually does not cause problems until these checks are implemented. Since the relay has very low overhead, it will publish the block before t = 4 without waiting for the confirmation deadline.

However, due to the delayed introduction of these five patches, the relay may now be partially responsible for the delayed broadcast. Let's look at the hypothetical block publishing process below.


The relay receives the signed header from the block producer at t = 3. By t = 4, the relay is still performing checks, so the broadcast will occur after the confirmation deadline. In this case, the combination of the delayed signed header sent by the block producer and some additional delay introduced by the relay caused the failure to broadcast before the confirmation deadline. If there hadn't been an honest reorganization, these blocks would likely have made it onto the chain. As shown in Figure 2, the honest block producers of the next slot will not deliberately reorganize because these blocks are late. However, if the confirmation deadline is missed, the blocks will be reorganized by the next block producer.

Consequently, the number of forked blocks increased dramatically in the days following the attack.


Two weeks of data from Metrika showed that in a worst-case scenario, 13 blocks (4.3%) could be reorganized in an hour, which is about 5 times the normal rate. The dramatic increase in the number of forked blocks became apparent as relayers rolled out various changes. Thanks to the great community efforts of the relay operators and core developers, once the impact was known, many changes were rolled back and the network was restored to a healthy state.

As of today, the most useful changes are beacon node block validation and repudiation checks before issuance. Malicious block producers can no longer perform attacks by sending invalid headers to relays and ensuring relay beacon nodes do not see repudiated blocks before publishing them. Nonetheless, relays still face the more general denial-of-attacks proposed in MEV-Boost and ePBS.

Next Action

In this post, we highlight how MEV-Boost works and how critical it is to Ethereum consensus. We also provide a detailed analysis of some lesser known aspects of the timing-related Ethereum fork choice rules. Using the "unbundling" attack and developer responses as a case study, we highlight the potential vulnerability of the time-dependent aspects of the fork choice rule and its impact on network stability.

With this in mind, the research community should evaluate what is an "acceptable" number of reorganizations, taking into account the more general exposure of denial attacks, to determine whether mitigations should be implemented.

Additionally, several future directions are currently being actively explored:

1. Implement a headlock mechanism to protect MEV-boost from equivalence error attacks. This would also require changes to the consensus client software and possibly specification changes to extend the proof submission deadline.

2. Increase the number and distribution of bug bounty programs for MEV-Boost software.

3. Extend the simulation software to explore how sub-slot timing affects network stability, which can be used to evaluate how to adjust proof submission deadlines to reduce reorganization.

4. Optimize the block release path on the relay to reduce unnecessary delays - this is already being explored.

5. Acknowledge MEV-boost as a core protocol feature and incorporate it into the consensus client, ie "enshrined-PBS (ePBS)". Two-slot ePBS is vulnerable to obvious attacks, so implementing a "headlock mechanism" is still an option.

6. By adding more hive and/or spec tests around issues around latency and certification deadlines.

7. Encourage relay client diversity by building additional implementations of the relay specification.

8. Consider adjusting penalties for obvious attacks, but keep in mind that even a full 32 ETH penalty may not deter malicious behavior when the extreme MEV opportunity exists.

9. Revisit sub-slot timing and consider adjusting the block propagation phase (for example, adjusting the certification deadline from t=4 to t=6).

Overall, we are excited about the resurgence of MEV and the mev-boost ecosystem. Through unbundling attacks and mitigations, we have learned about the critical relationship between latency, MEV-boost, and consensus mechanisms; we hope the protocol will continue to be hardened to handle this.