GM,

Last week, Ethereum experienced a crisis, with approximately 8% of Ethereum nodes unexpectedly disconnecting for 4 hours due to a program bug. The severity of node disconnection is comparable to a malfunction in TSMC machinery. Not only will staked ETH on the nodes be cut as a penalty, but in severe cases, it could lead to a fork in Ethereum, rendering all transactions sent during that time invalid. Surprisingly, after this node software glitch, the popularity of the affected software, Nethermind, soared. Within a week, its market share jumped from 8% to 14%. What's going on?

Node Disconnection

The incident occurred early on the 21st Taiwan time when a user noticed an error in the Ethereum node software Nethermind. He reported it on the code collaboration platform GitHub, warning other users of potential issues. A minute later, this error was promptly tagged as critical, indicating a serious situation. Engineers who had plans for the weekend had to quickly go online to fix the bug.

The urgency stemmed from the fact that blockchain operates as a decentralized network maintained by nodes globally. While the disconnection of some nodes won't halt Ethereum, simultaneous disconnection of a large number of nodes can affect the stable operation of the blockchain. The following image illustrates the global distribution of Ethereum nodes. Currently, Ethereum is maintained by a total of 7,259 nodes distributed across different countries. The majority are in the United States, followed by Germany, the United Kingdom, and Canada in sequence.

From the perspective of network resilience alone, this data doesn't look healthy. In the event of a major power outage in the United States, Ethereum could potentially lose almost half of its nodes in one fell swoop. This represents a geographical centralization risk. The distribution of nodes among different countries should ideally be more balanced. Therefore, some are willing to go to great lengths, even sending nodes into space. The recent paralysis of Nethermind nodes has drawn attention to another, more severe risk than geographical location — the imbalance in software market share.

Market Share Imbalance

Node software serves as the communication bridge between computers and the blockchain. Anyone running node software on their computer becomes a part of the blockchain. Essentially, blockchain upgrades involve upgrading node software. When node software on people's computers is all upgraded to the latest version, the blockchain upgrade is considered complete. However, since all programs have bugs, there are currently several interoperable Ethereum node software options available on the market to prevent the entire blockchain from collapsing due to a minor bug.

The chart below illustrates the current market share of node software, with consensus layer software on the left and execution layer software on the right. The consensus layer's role is to ensure that global nodes maintain the same state, while the execution layer processes transactions and executes smart contracts. The situation with node software is somewhat analogous to email inboxes. Although different inboxes can communicate with each other and have similar functions, most people still choose to use Gmail.

From the chart, you can see that the market share on the left, the consensus layer, is relatively balanced, but on the right, the execution layer is clearly imbalanced. Currently, Geth, developed by the Ethereum Foundation, holds the highest market share. Its position is akin to Google Pixel phones in the Android ecosystem. Geth not only has operated stably for a long time but is also most capable of keeping up with the latest developments. Therefore, Geth's market share is as high as 78%.

It is precisely because Geth performs so well that it has become the biggest risk for Ethereum. Geth is the industry's model student, but even the best has its flaws, and it's impossible to never make mistakes. After the Nethermind incident, two different opinions emerged online. Some argue that using Geth is the most secure choice. Others believe that if one day Geth encounters a problem, Ethereum will be in big trouble!

In the past, there was a well-known saying in the IT industry: "Nobody Gets Fired For Buying IBM." The implication was that if even IBM could make mistakes, what could be considered absolutely safe? This is exactly the sentiment of Geth supporters, especially since the recent issue occurred with Nethermind. In theory, Geth supporters should be more convinced that their choice is the best, but the results are completely the opposite of what everyone expected.

Many people only realized the problem of software market share imbalance due to the Nethermind incident. Additionally, Ethereum considered human nature in its mechanism design, incorporating a "punishment" for conforming behaviors. In this case, Nethermind was penalized by having ETH staked on its nodes reduced as a consequence of a 4-hour disconnection due to a programming error. However, the short duration of the disconnection and the fact that Nethermind is not a mainstream node software meant the impact on the blockchain was not significant, resulting in a relatively mild penalty.

The chart below illustrates the changes in ETH quantity on Nethermind nodes, showing that the ETH burned by the mechanism was recovered in just a few hours. It can be said that there was almost no impact.

But what if the problem lies with Geth? According to calculations, if a node is disconnected for 2 days, it will burn two months' worth of staking rewards. If disconnected for 5 days, it will burn an entire year's worth of rewards. The longer the disconnection, the heavier the penalty. For instance, with a 40-day disconnection, using Nethermind would only burn 0.4% of the total staked ETH, while using Geth would result in a 90% burn. If 80% of the network uses Geth, it could lead to the complete depletion of circulating ETH in the market.

This "punishment" mechanism is actually designed to balance the distribution and quickly reduce the influence of nodes with issues on the blockchain. However, from the results, it almost equates to the higher the market share, the heavier the penalty. This punishment mechanism is a major factor driving the increase in market share for Nethermind after the incident, as people dislike being fined. If issues are unavoidable, then minimizing the penalty is naturally preferable.

User Pressure

Many people now stake their ETH with centralized exchanges to earn staking rewards, but few read the terms of use. Ironically, well-known exchanges like Coinbase, Binance, Kraken, and Bitfinex are currently users of Geth node software. Take Coinbase, which is no longer accessible to Taiwanese users, as an example. They clearly inform users of the risk of being penalized for staking ETH and under what circumstances they will take responsibility:

To ensure the normal operation of nodes, some blockchains penalize validators who violate protocol rules. In the event of a penalty, Coinbase will decide whether to compensate for the loss based on the cause. If the penalty is due to an error on Coinbase's part, we promise to compensate users for those losses. However, if the loss is due to a hacker attack, individual behavior, or errors in the protocol itself, we will not compensate for the loss. While the probability of such events is extremely low, Coinbase has not experienced such losses, but it is not impossible.

Coinbase is already one of the exchanges that provides the most comprehensive information about risks and response methods. However, they still haven't clarified whether they will compensate for losses caused by Geth errors. ETH staking services provided by exchanges are not free; for example, Coinbase charges a 25% revenue-sharing fee from users. Exchanges must pay attention to user feedback, as users can choose other service providers if dissatisfied.

Recently, MetaMask has introduced a highly competitive individual staking service. Not only is it easy to operate, with users able to become independent nodes with just 32 ETH by opening the Portfolio page, but it also charges users only a 10% service fee based on their earnings. For users, this service not only offers higher returns but also eliminates the hassle of KYC from centralized exchanges, and it may even exempt users from tax reporting.

MetaMask emphasizes the use of diverse node software to reduce the risk of a single point of failure. This has prompted Coinbase and Kraken to publicly commit to diversifying their node software, and even the largest staking service on the internet, Lido, has started to reduce its reliance on Geth.

The question arises: if not Geth, then what to use? Nethermind, which just encountered issues, is a preferred alternative to Geth, and some may opt for the third-ranked Besu or the fourth-ranked Erigon. This results in the peculiar phenomenon where, despite the programming error occurring in Nethermind, the exemplary Geth seems to be "caught in the crossfire."

Recently, I finally finished watching the Netflix documentary series "Life on Our Planet." I discovered a fascinating pattern over the Earth's 4 billion years – species are better equipped to resist attacks when they gather together. However, to survive the mass extinctions occurring every few hundred million years, dependence on the dispersion and diversity of life forms is crucial. Faced with threats, human instinct tells us to come together for safety, but nature teaches us that longevity comes from spreading out. Ethereum staking operates in a similar way. Despite its mechanism going against human instinct, it enables the blockchain to endure for a longer duration.

Blocktrend is an independent media sustained by reader subscriptions. If you find Blocktrend articles valuable, feel free to share this piece. You can also join discussions on the member-established Discord or save this Writing NFT to add this article to your Web3 records.

Furthermore, please recommend Blocktrend to your friends and family. If you successfully refer a friend who subscribes, you'll receive a one-month membership extension for free. You can check the article list for past publications. In response to frequent inquiries about referral codes, I have compiled them on one page. Feel free to use them.