Unexpected removal of DRep delegation leading to an issue with withdrawal

[Scitz0]

An error was noticed when trying to submit a transaction including a withdrawal when the stake key has a vote delegation to a registered but inactive DRep. Multiple tests were performed. And it was not always the case when withdrawing rewards with a vote delegation to an inactive DRep that the error was given.

Describing the flow of events to make it a bit clearer.

1. Issue noticed and multiple re-delegations and transactions where performed. Rewards was withdrawn in one of the tests.
2. To re-construct the issue in a more formal way, the wallet was vote delegated to DRep with hex 2274519ac5359c003a7ace69c475ae55a86eb8f9fec6cf6feaada1debf. This DRep was registered in epoch 724, updated in 734 and marked as inactive in 754. Rewards was successfully withdrawn in epoch 758, tx hash de2342e28817d3117e8fd09e01dd948668cb15a61d1cfcdf914eb8861c9d404f. This confirmed how it should work, that inactive state should not prevent withdrawal.
3. Not convinced that the result in (2) was the full story we found the DRep delegation we originally had when we got the error and re-delegated vote power to this in epoch 758 after test in (2), hex 22fa6a8dc2635dddcf9af495cb144f7eb4ff845866fe48695ad7cb65d3. This DRep is from own account but doesn't matter. Registered in epoch 664 and vote cast in 684 as the only action. Thus expired in epoch 704.
4. We waited until epoch 759 to receive new rewards to withdraw as it was easier with the tools at hand (I know you can withdraw 0 lovelace). Now we get the error message again as we originally found. Tried both w/ and w/o 258 tagged sets to be sure it wasn't connected to that.

Network: Preview Testnet
Abs slot for submission test: 65610177
Node version: cardano-node 10.1.1 - linux-x86_64 - ghc-8.10 git rev 4184f9297bf7306713bfbb8f39eef61c056198cc
CLI version: cardano-cli 10.1.1.0 - linux-x86_64 - ghc-9.6 git rev 0000000000000000000000000000000000000000
Ogmios version: v6.9.0 (a07c154e)

Wallet address:

addr_test1qpfepft9zs3y8ejcv84tq6tkp00wdm46fr6h3am02leunk8dc55q34v2ggxw9hea4rr3rry933a2zdh60v43h237s8ks7t2dja

CBOR (w/o 258 tag)

84a60081825820efc7370f054274109615a8c0d1545b035ed8920df3965b688431a36b8e3839fa010182825839005390a565142243e65861eab069760bdee6eeba48f578f76f57f3c9d8edc52808d58a420ce2df3da8c7118c858c7aa136fa7b2b1baa3e81ed1a4bb25055825839005390a565142243e65861eab069760bdee6eeba48f578f76f57f3c9d8edc52808d58a420ce2df3da8c7118c858c7aa136fa7b2b1baa3e81ed1a004c4b40021a0002a9b9031a03e9468405a1581de0edc52808d58a420ce2df3da8c7118c858c7aa136fa7b2b1baa3e81ed1a0042018e0800a100828258209b5bd6d6a8142c69586493d367a3a561e26d76037ac3928939fdc374a110bbdf5840c238651cc86a703a3db55d9eebb31a4364a31efdec6bf16bf165ee2e2a9e6328ecaa04fec30411efe9c22ace476d5c558f49d0b05ed8bd1a82374e5edf3dd60c8258201cf1e6972f59c345fa295fab41261a1e2058d49b7441741dc711a9745f70b9df5840fdb77b2181578dfb55661546b1b7bde4b0c1bd8ef40c45516a9e5eb4132595d5bb79906bfc74bb95b3af403e9e800c83ea89744738d4b9ac98f2bf4c91760e05f5f6

CBOR (w/ 258 tag)

84a600d9010281825820efc7370f054274109615a8c0d1545b035ed8920df3965b688431a36b8e3839fa010182825839005390a565142243e65861eab069760bdee6eeba48f578f76f57f3c9d8edc52808d58a420ce2df3da8c7118c858c7aa136fa7b2b1baa3e81ed1a4bb24f4d825839005390a565142243e65861eab069760bdee6eeba48f578f76f57f3c9d8edc52808d58a420ce2df3da8c7118c858c7aa136fa7b2b1baa3e81ed1a004c4b40021a0002aac1031a03e94a6805a1581de0edc52808d58a420ce2df3da8c7118c858c7aa136fa7b2b1baa3e81ed1a0042018e0800a100d90102828258209b5bd6d6a8142c69586493d367a3a561e26d76037ac3928939fdc374a110bbdf5840802c5255c6c7e508378d6ca83484d0bf06c0f8ce6c08df0b10515af0e4f3caff728b84ee6edebcd3796ce7b4ae17012c8b92e9e8a23b72bc020799edb5cae70a8258201cf1e6972f59c345fa295fab41261a1e2058d49b7441741dc711a9745f70b9df58405074a4ccc1c0b8d5c5264b8501e3dfb9886998eba26857cc32b08bcf6f9f6bc294ef92e78d5456ddef8ab8268d8fdb79317d6015f993bdf3f30944c194e91c0df5f6

Ogmios error:

{"jsonrpc":"2.0","method":"submitTransaction","error":{"code":3150,"message":"The transaction is attempting to withdraw rewards from stake credentials that do not engage in on-chain governance. Credentials must be associated with a delegate representative (registered, abstain or noConfidence) before associated rewards can be withdrawn. The field 'data.marginalizedCredentials' lists all the affected credentials.","data":{"marginalizedCredentials":["edc52808d58a420ce2df3da8c7118c858c7aa136fa7b2b1baa3e81ed"]}},"id":null}

Node error:

Command failed: transaction submit  Error: Error while submitting tx: ShelleyTxValidationError ShelleyBasedEraConway (ApplyTxError (ConwayWdrlNotDelegatedToDRep (KeyHash {unKeyHash = "edc52808d58a420ce2df3da8c7118c858c7aa136fa7b2b1baa3e81ed"} :| []) :| []))

[Scitz0]

It was put to my attention by Martin that it could maybe be related to the v10 HF, that DRep was marked as inactive long before the HF was enacted in epoch 743. Just mentioning it if it helps in figuring out whats going on.

[lehins]

So, I was able to get to the bottom of this earlier today. It is indeed a bug in the ledger code. It was introduced in #4598 and the actual reproducer of this bug requires a bit of a setup and it was really hard to spot. For the curious here would be the bugfix, if it was spotted before we made an actual release: c3059b0
Now we have to do a bit more care, since this code is running on mainnet, albeit not at PV 10 yet.

In order to demystify this bug here is how it depicted itself on preview:

1. At slot number 65453414 in tx 70a44a963b8fdd34fe901f5e1b9476883fac5747a0df51edef7375cb1b78fa4b#1 (block hash + tx ix):
   • staking key hash edc52808d58a420ce2df3da8c7118c858c7aa136fa7b2b1baa3e81ed
   • delegated to a to a DRep with key hash 7611337d51b4dd3b568706758f0689afab6afc262c2fe842fd6db34d
2. At slot number 65521313 in tx cda5fee6405e78fe86dd44614de7480234762e0a272218741f7ccfc665c56297#0:
   • same staking key hash edc52808d58a420ce2df3da8c7118c858c7aa136fa7b2b1baa3e81ed
   • delegated to a to a different DRep with key hash fa6a8dc2635dddcf9af495cb144f7eb4ff845866fe48695ad7cb65d3
3. At slot number 65603506 in tx 69441ff717cfd324467a2230be0e40dae5e93ea92ef79cc97ec89dae31dbe06c#2:
   • staking key hash edc52808d58a420ce2df3da8c7118c858c7aa136fa7b2b1baa3e81ed
   • delegated to a to a DRep with key hash 7611337d51b4dd3b568706758f0689afab6afc262c2fe842fd6db34d

Normally what should have happened is DRep 7611337d51b4dd3b568706758f0689afab6afc262c2fe842fd6db34d should have forgotten about that delegation from step 1 during the step 2. However, the bug cause that delegation to be retained in the DRep's state. Which lead to stake credential's delegation to a totally different DRep be cleared out during step 3.

Once there was no delegation to any DRep it was impossible to withdraw rewards, so that part behaved as expected and inactivity of DReps played no role in this bug, it was just a coincidence.

This is a very unfortunate bug, but it is a blessing that it was discovered before the hard fork to protocol version 10.0 on mainnet! So, @Scitz0 thank very much for this bug report!

The fix is implemented in #4773 and we'll be working out the details on how to proceed with deploying this fix on preview/preprod/mainnet. Clearly cardano-node-10.1.2 and prior cannot be used for the hard fork out of bootstrap on mainnet, so there will be a new release for that purpose. We'll be discussing the details during the Hard Fork Working Group meeting tomorrow.