Mainnet Incident 22 March 2023 update 2023/03/22 19:32 (GMT).

22 Mar 2023, 22:41
Mainnet Incident 22 March 2023 update 2023/03/22 19:32 (GMT) The project team investigation has identified the network outage to have happened at the time that a new validator was being promoted by the protocol to become a consensus validator. The issue is believed to have happened due to insufficient validation on the Tendermint public keys specified in the announce node command. The Tendermint public key supplied was incorrect but this was not detected by the code. The protocol was then unable to parse the key, resulting in the node crashing while promoting the node to consensus validator status. A fix has been prepared and is currently being tested in conjunction with preparing an updated genesis file in readiness for a possible checkpoint restart to deploy a patch fix.

Same news in other sources

3
Vega Protocol
Vega ProtocolVEGA #924
Telegram
23 Mar 2023, 22:51
Mainnet Incident 22 March 2023 update 2023/03/23 18:45 (GMT) @everyone The network has been processing blocks throughout the day without issue, and the epoch change occurred at 17:40 without incident. We are now confident that the original cause of the network outage has been successfully resolved. However, the network resumed block production, some nodes have not been forwarding Ethereum events. This means that the Vega chain is not currently aware of actions taking place on the Ethereum network, which impacts some aspects of staking. Actions that happen entirely on Vega, including moving stake delegations between validators are not affected, but associating or disassociating tokens to/from a Vega key will not complete until this is resolved. This is under investigation and this post will be updated as soon as the Vega project team knows more. There remains no risk to user funds or tokens due to this issue.
Mainnet Incident 22 March 2023 update 2023/03/23 18:45 (GMT) @everyone.
Mainnet Incident 22 March 2023 update 2023/03/23 18:45 (GMT) @everyone The network has been processing blocks throughout the day without issue, and the epoch change occurred at 17:40 without incident. We are now confident that the original cause of the network outage has been successfully resolved. However, the network resumed block production, some nodes have not been forwarding Ethereum events. This means that the Vega chain is not currently aware of actions taking place on the Ethereum network, which impacts some aspects of staking. Actions that happen entirely on Vega, including moving stake delegations between validators are not affected, but associating or disassociating tokens to/from a Vega key will not complete until this is resolved. This is under investigation and this post will be updated as soon as the Vega project team knows more. There remains no risk to user funds or tokens due to this issue.
Vega Protocol
Vega ProtocolVEGA #924
Telegram
23 Mar 2023, 05:26
Mainnet Incident 22 March 2023 update 2023/03/23 00:45 (GMT) The validators have completed the deployment and the Vega network is operational and producing blocks. The project team will monitor the network to ensure the fix has been successful and the network is operating normally. Confirmation the fix is resolved will be communicated after ensuring future epoch transitions and validator status changes are handled correctly.
Mainnet Incident 22 March 2023 update 2023/03/23 00:45 (GMT).
Mainnet Incident 22 March 2023 update 2023/03/23 00:45 (GMT) The validators have completed the deployment and the Vega network is operational and producing blocks. The project team will monitor the network to ensure the fix has been successful and the network is operating normally. Confirmation the fix is resolved will be communicated after ensuring future epoch transitions and validator status changes are handled correctly.
Vega Protocol
Vega ProtocolVEGA #924
Telegram
22 Mar 2023, 22:43
Mainnet Incident 22 March 2023 update 2023/03/22 20:42 (GMT) The Vega project team has prepared a patch fix made available in version 0.53.1; this version has been made available to the validators to deploy via a checkpoint restart. It is confirmed that all funds on the Vega network are safe. The Vega project team will monitor the network to ensure the fix has been successful during future validator status changes.
Mainnet Incident 22 March 2023 update 2023/03/22 20:42 (GMT).
Mainnet Incident 22 March 2023 update 2023/03/22 20:42 (GMT) The Vega project team has prepared a patch fix made available in version 0.53.1; this version has been made available to the validators to deploy via a checkpoint restart. It is confirmed that all funds on the Vega network are safe. The Vega project team will monitor the network to ensure the fix has been successful during future validator status changes.