Solana Network faced a significant slowdown in block production after a planned upgrade, negatively impacting transactions and forcing validators to downgrade their software to restore performance.
The network freeze continued over the weekend as validators prepared a second network restart.
Solana’s last blackout
The Solana Network faced another network slowdown following an upgrade initiated in the validator software. The incident, which took place on February 25, resulted in a significant disruption to transactions on the network. As a result, validators rushed to downgrade the validator software in an attempt to restore performance on the network. The problem started at 6:00 am UTC when the network updated the validator software to 1.14. With the network stuttering, validators were forced to downgrade the network to version 1.13 to restore network performance. Solana’s Compass website noted,
“The network experienced a significant slowdown in block production that coincided with a validation software update. Engineers are still performing a root cause analysis.”
However, the downgrade failed to restore normal operations on the Solana network, forcing validators to attempt to restart the network at 1.13.6. The network announced the plans for a reboot on Twitter, stating,
“The Solana network is currently restarting after an issue during the upgrade from 1.13 to 1.14 that slowed down the block completion. Once the validators with 80% stake have restarted, the network will resume.”
The root of the problem
According to available information, the issue is related to an upgrade from version 1.13 to version 1.14, which caused a slowdown in block validation. To restart the network, 80% of active validators are required to resume operations.
“As more validators complete their reset, this number will increase according to the amount of participation they have delegated – this means that larger validators like CEX have a huge impact on reset times.”
Solana validators remained in talks about the incident over the weekend, with infrastructure provider Chorus One noting that the incident demonstrated the level of decentralization in the network.
“Without all these discussions, we would be back in an hour. Yet every decision along the way is debated: whether to downgrade, whether to reboot, when to switch from the downgrade approach to the reboot approach. Voting happens. We ended up taking 8-10 hours to recover instead of 1”.
The network freeze in Solana continued over the weekend as it continued on Saturday, with validators preparing a second restart attempt to restore services to users. At night, the validators concluded that the best way forward would be to time a reboot and fork the chain. However, the first attempt had to be abandoned when the validators discovered that they had chosen the wrong starting point for the reset, further prolonging the outage.
An almost complete closure
The issue, which started as a simple slowdown in transaction processing, evolved into a near complete shutdown of any activity on Solana, with block production interrupted, resulting in transactions not being processed or validated. . As a result, on-chain crypto assets have become immovable until the network comes back online. Key players in the Solana ecosystem were still looking to identify the problem over the weekend, with one theory suggesting that a “fat block” messed up the blockchain mechanics.
Not the first network outage
This isn’t the first time Solana has had a problem with network outages. In September 2021, the network was hit by intermittent instability due to what the protocol’s Twitter handle described as “resource exhaustion.” He grid it made headlines again in December 2021 after it emerged that technical difficulties and network congestion caused another network slowdown. Just a month later, in January 2022, a 48-hour blackout wreaked havoc on the Solarium network, forcing some users who had taken out loans to liquidate their holdings.
Disclaimer: This article is provided for informational purposes only. It is not offered or intended to be used as legal, tax, investment, financial or other advice.