Holešky debrief call notes:

• EthMag: https://ethereum-magicians.org/t/holesky-incident-debrief-february-26-2025/22998
• In Twitter form: https://x.com/TimBeiko/status/1894773111578562856

What's happening?

The Pectra fork went live on the Holešky testnet but a contract address that gets incorporated into a hash was incorrectly specified in three execution clients (because mainnet operates differently – this wouldn't have happened on mainnet). A majority of clients attested to an invalid block and then many validators were immediately shut down to avoid finalizing the wrong chain. The bug was fixed by execution layer client releases but now the consensus layer client devs are trying to get the chain stable, which has proven difficult since ~90% of the testnet validators voted for the fork. CL devs are trying to save Holešky but it's not existential that they do so: this is turning out to be a great exercise in both incident response and consensus disaster recovery.

The testing team is now spinning up a separate million-validator devnet-7 so that consolidations can be thoroughly tested for the Pectra upgrade. They're coordinating with entities that need to test consolidations (staking pools, DV operators, etc). The Pectra fork on the Sepolia testnet will likely go ahead next Wednesday as planned.

If you are already running Holešky validators:

• The consensus is: turn your Holešky validators back on (if you shut them down), attempt to sync
• If you're failing to sync, do not run to CL devs for support. They're busy!
• How to check if you're on the right chain: https://gist.github.com/samcm/e2da294dab77e93ad0ee0e815580294f
• DO NOT delete slashing DBs. Run normally. If you attested to the invalid block, your slashing protection will prevent you from attesting but you'll still produce blocks
• Once the missed slots are <25%, core devs will start coordinating slashing among their validators. They may be able to absorb most of the slashings in their validators
• Finalization will likely take weeks, but the goal rn is just a stable network
• If you run non-validating nodes on the correct chain, this will help the network for peers

Keep up with updates

If you want to keep up with updates to see how it goes or know how continued Pectra testing on devnet-7 is going, tune into the ACD call tomorrow!: https://www.youtube.com/watch?v=tlezpGztpi8