A bug related to Native IOTX staking was reported to IoTeX Foundation on January 27. This thread outlines the incident details, as well as the resolution plan that will be enacted to resolve any impacts to IoTeX voters.
Incident Report
Abstract
Since epoch #6593 (01/23/2020, 11:53pm UTC), Native IOTX staking buckets were not included in total votes – only IOTX-E (ERC20) buckets were counted. As a result, Delegates and voters were not rewarded accurately for the affected epochs. A fix has been deployed, which will be activated on February 10.
Root Cause
Native staking buckets are stored in a smart contract on IoTeX Mainnet. The gas limit to read buckets data from the contract was originally set to 1,000,000. This gas limit was sufficient for a smaller number of total buckets, but not enough after a certain number of total buckets. After epoch #6593 when the number of total native staking buckets exceeded 120, the 1,000,000 gas limit was not enough to read the buckets data. As a result, Native IOTX staking buckets were not counted appropriately after epoch #6593.
Mitigation
After being notified about this bug, IoTeX Foundation investigated and developed a patch to fix the issue, which has been deployed in Mainnet v0.10.2. In this new version, the gas limit to read native staking buckets has been increased to the block gas limit. End-to-end test cases were used to cover this case, and we verified this new fix reads staking buckets correctly. For more details on this issue, please see our postmortem on Github.
IoTeX Delegates have been informed, and have until February 10 to upgrade their nodes to v0.10.2.
Resolution
From discussions with the community, this bug has affected not only those staking with Native IOTX, but also those staking with IOTX-E (ERC20). This is because there are multiple components of rewards (block rewards, Foundation bonus, epoch bonus rewards), which rely on accurate total votes counts. Note that the total amount of IOTX distributed to all Delegates was correct; however, the allocation of the total amount of IOTX was incorrect. Delegates without native staking buckets received more rewards than they should have, while Delegates with native staking buckets received less rewards than they should have.
To resolve this issue, the IoTeX Foundation will ensure the appropriate amount of rewards from January 23 - February 10 are distributed to all voters after Mainnet v0.10.2 is activated:
- All rewards distributed between January 23 - February 10 will be kept by Delegates/voters.
- Any Delegates/voters that received fewer rewards than they should have will receive the amount they are missing from IoTeX Foundation funds. Note: we will post the delta calculations in this thread once finalized.
This resolution approach ensures that ALL Delegates/voters will at the minimum receive the accurate amount of rewards, with some voters receiving extra rewards on behalf of IoTeX Foundation.
Questions?
Thank you to our community that helped identify and resolve this issue. IoTeX Foundation takes these issues seriously and we have developed a resolution to ensure this bug does not affect community tokens/rewards in any negative way. Bugs happen – thanks for your patience while we resolve this issue.
If you have any questions, please feel free to PM an Admin on Telegram.
