How to generate IN_RECOVERY state ledgers

[hangc0276]

How to generate IN_RECOVERY state ledgers

Ensemble: 2-2-2

Case 1

• Ledger1 (0={bk1, bk2}, 150={bk2, bk3}) OPEN state
• Shutdown bk2 and bk3
• Use command bin/bookkeeper shell readledger -fe 1 -le 1 -l 1 -r -m to force recover the ledger
• Force recover will fail and the Ledger keeps in IN_RECOVERY state, and can't be recovered

Case 2

• Ledger1 (0={bk1, bk2}) OPEN state
• Shutdown bk1 and bk2
• Use command bin/bookkeeper shell readledger -fe 1 -le 1 -l 1 -r -m to force recover the ledger
• Force recover will fail and the Ledger keeps in IN_RECOVERY state, and can't be recovered

Case3

• Ledger 1 (0={bk1, bk2}, 150={bk2, bk3})
• Shutdown bk2 and bk3
• Use command bin/bookkeeper shell readledger -fe 1 -le 1 -l 1 -r -m to force recover the ledger
• Force recover will fail and the Ledger keeps in IN_RECOVERY state, and can't be recovered
• Unload the topic
• Ledger 1 will be marked as CLOSED state and lastEntry will be set (for example, lastEntry will be set to 220) -> The Ledger's CLOSED state is update by BookKeeper client when the topic unloaded
• Load the topic failed even though we enabled SkipUnRecoverableLedger
• Use command bin/bookkeeper shell readledger -fe 1 -le 1 -l 1 -r -m to read messages and recover the ledger, the message can be read out and ledger keeps in CLOSED state
• Use command bin/bookkeeper shell readledger -fe 160 -le 160 -l 1 -r -m to read messages and recover the ledger, the message will throw read failed exception and ledger keeps in CLOSED state. It means the ledger's last segment can't be replicated.

How to deal with IN_RECOVERY state ledgers in decommission

We can't deal with it except delete this ledger. When the ledger runs into IN_RECOVERY state and can't recover to CLOSED state, it means part of the ledger's data has been lost.

The only issue is that Pulsar's SkipUnRecoverableLedger flag can't cover this case and the topic load into the Pulsar broker failed.

Pulsar SkipUnRecoverableLedger doesn't work

• Ledger 1 (0={bk1, bk2}, 150={bk2, bk3})
• Shutdown bk2 and bk3
• Unload the topic
• Ledger 1 will be marked as CLOSED state
• Load the topic failed even though we enabled SkipUnRecoverableLedger