Handle reorgs

[casey]

Currently we detect and error on reorgs, but we should actually handle them.

We should implement this by adding checkpoints and rollbacks to redb, which would make this trivial:

cberner/redb#341

So this issue actually requires adding checkpoints and rollbacks to redb.

[apezord]

Think will be fixed soon @casey ? It impacts ord-dogecoin more because dogecoin has shorter blocks. I could give this a shot if you haven't started and want to leave a few notes.

[decentraliser]

ord-litecoin just got stuck on reorgs as well
// error: reorg detected at or before 2438403

[apezord]

Eventually it will affect BTC too, and the impact will be wider. Best to fix before that happens.

[DanielHook15]

I got the following error message (on BTC):
ERROR ord::index::updater Values channel closed unexpectedly
error: reorg detected at or before 78XXXX

Now ord no longer works. When I try to inscribe something, it says
error: reorg detected at or before 78XXXX

What can I do?

[Magicking]

Not sure of the exact configuration of your setup but could setting your bitcoin nodes confirmation block to a higher limit could prevent you to be too much affected by this issue @apezord @decentraliser ?

[DanielHook15]

I am just using Bitcoin Core in its standard form, no virtual machine or other fancy things.
Thank you very much for the fast help, but what do you mean by "setting your bitcoin nodes confirmation block to a higher limit"?

[DanielHook15]

Do I have to delete the index file and reconstruct it?
If yes, can I somehow delete the information about the last few blocks from index.redb so that ord only reconstructs the last entries? I want to avoid that it has to restart from block 1, which would take several days.

[apezord]

So reorg just happened on BTC

https://twitter.com/CynicusUnum/status/1636913796714299392

and as expected everything breaks

https://twitter.com/ordswap/status/1636942287602081792

Guys, is this a serious project? You did know that reorgs were going to happen eventually right? And you said nothing in any documentation about making manual backups?

Not a good look.

[apezord]

If yes, can I somehow delete the information about the last few blocks from index.redb so that ord only reconstructs the last entries?

It should be possible to write code to roll back the chain, which would be faster than a reindex, but someone needs to write that code.

[apezord]

This might even lead to users losing artifacts depending on how wallets are implemented.

Raising priority on this PR https://github.com/casey/ord/pull/1938

[DanielHook15]

Can this loss also happen with the normal ord wallets that one has if following the ord handbook or only with other wallets like sparrow?

[0attack]

wow wow wow this is really bad. so everything is completely broken with no solution idea or even anyone saying work is being done. not to be rude at all but given how unresponsive the repo owner and collaborators are, can anyone quickly release a fork with a fix? otherwise who knows how long it will take. that could be the best way to resolve this mess...

[0attack]

so the stated solution is for everyone to reindex from scratch and to expect to need to manually do that multiple times a year... https://github.com/casey/ord/issues/1945#issuecomment-1474887671

[DanielHook15]

Recreating the index would be a bit waste of time, but not a big issue. However, is this really the full solution? apezord mentioned further above that we might loose artifacts. Is it really safe to use ord further (with a new index.redb file) or should I wait for a new version of ord that is safe?

[DanielHook15]

@apezord: Thank you for your warning! It would be great if you could explain in more detail how we could lose our nfts and what we can do to avoid that.

[0attack]

Recreating the index would be a bit waste of time, but not a big issue. However, is this really the full solution? apezord mentioned further above that we might loose artifacts. Is it really safe to use ord further (with a new index.redb file) or should I wait for a new version of ord that is safe?

yes, a waste of time but also huge waste of limited writes on a storage device since the software has an unaddressed bug causing extreme disk usage. being forced to accept needless reindexing multiple times a year will add up quickly. a noticeable and significant decrease in disk life expectancy will be the result.

@apezord: Thank you for your warning! It would be great if you could explain in more detail how we could lose our nfts and what we can do to avoid that.

iiuc, it would be a possible issue with third-party wallets but additional clarification would be nice to get.

[rot13maxi]

started a discussion on changes to make the indexer handle re-orgs: https://github.com/casey/ord/discussions/1951

[ordinalsbot]

so the stated solution is for everyone to reindex from scratch and to expect to need to manually do that multiple times a year... #1945 (comment)

in our case 1 out of 2 VMs recovered by re-indexing but looks like even re-indexing is not a sure fix.

[lglove]

how to reindex

[nammaki]

how to reindex

https://github.com/casey/ord/issues/1946#issuecomment-1480408163

[yangmingming]

ord-litecoin也被卡在重组中 // 错误：在2438403或之前检测到重组

Litecoin is not ok now

[77656233]

Is there any update @casey?

https://github.com/hirosystems/gh-action-testing/blob/e06e254620d903a9709a1ccbdcd9ba141d73ff6f/src/burnchains/bitcoin/indexer.rs

They somehow handle reorg, can't you implement that also?

[0attack]

another un-handled reorg occurred at 783830... https://github.com/casey/ord/issues/1945#issuecomment-1495341510

[V01dZer0]

How's this issue going on?
Still got an error today.

[onbermejo]

same here. any solutions?

[dylan1951]

same

[0attack]

ord doesn't handle reorgs so it breaks if there's a reorg which there always are. you need to delete the index.redb file then run ord index again which will reindex from scratch.

[dylan1951]

where tf is the index.redb file???

[0attack]

where tf is the index.redb file???

linux should be ~/.local/share/ord/index.redb, mac should be ~/Library/Application\ Support/ord/ and windows should be ~\AppData\Roaming\ord

https://github.com/casey/ord/issues/1623, https://github.com/casey/ord/discussions/1627 for more info

[donnlee]

well thank goodness i made a backup of index.redb after the database format change introduced by 0.5.0. I really did not want to go through that again.
I hope i remember to make another backup after this new sync finishes.

[dannybabbev]

Hi, this is a serious issue and should be handled with top priority. Indexing the redb takes days and reorgs happen very often.

[V01dZer0]

Since ord only cache sat ranges of the last tx, it's not possible for redb to rewind the the -2 block when bitcoin main net reorg.

This issue should be closed with [WONT FIX]

[raphjaph]

We finally have reorg resistance: #2320