Restoring/migrating/importing from older Signal backups (db v105)

[arc-des]

Hi, thanks again for this wonderful tool!

I'm not sure if this is an issue you can/are motivated to help with, but I thought it'd be good to ask.

Newer versions of Signal cannot restore older backup files (the app crashes at the last block) and older versions of Signal that do seem to successfully restore cannot pass the registration step (returning "Unable to connect to service" when trying to verify phone number). Updating the app post-restore but pre-registration doesn't work (causes it to crash loop at the welcome page), so I'm a bit stuck trying to restore my backups from 2021/2020 (my newer ones are corrupt).

I'm looking to migrate (or import) older db versions (v105 or v44) to a newer backup file - I've tried to use --migrate_to_191 which informs me that the db version is too old and stops (I can see from #233 that the earliest supported version is v132), which makes sense.

I then tried exporting the v44 backup to HTML, which states:

[Error]: Currently unsupported database version (44).
         Please upgrade your database or append the `--migratedb' option to attempt to
         migrate this database to a supported version.

Running it with the --migratedb flag then sucessfully exports to HTML. I then ran a regular backup-to-new-backup file operation with --migratedb and it seemed to work but didn't change the backup db version - I'm unsure if it's meant to or if this is just for the export options?

Are there any plans to extend the migration functionality to older backups? Or is there a way to export the backup and re-import it (i'm aware the JSON import is for Telegram only, but I was thinking something along those lines/ importing from an HTML export as that seems able to use the --migratedb flag on v44 or v105). I don't know enough about how the database file works - opening the dumped database.sqlite in n++ looks to my inexperienced eyes as if there's something up with the encoding, however if I could get that working/get my head around it I was thinking of attempting to manually copy over the history of select threads to a more recent backup (derived from a Desktop backup).

I've attempted exporting to XML and importing through the --importplaintextbackup function from #227 - of course not designed for this at all - which gave [Error]: Foreign key constraint violated. This will not end well, aborting.

Thanks again!

[bepaald]

Hi!

Newer versions of Signal cannot restore older backup files (the app crashes at the last block)

Do you have a crash log of this? Usually, the backup is first restored (a database is created and filled with your data), then the 'migrations' run, to get the database up to a current version. I'm curious if the crash actually happens during the data import, or during the migration phase. I am suspecting the latter, there is even a known problem during the migration to v191 (which is actually the reason I have re-implemented that particular one). A debug log of the restore process will show what's happening.

It is important to note, that Signal should be able to restore older backups (at least the v105 one I'd say), so really you should open a bug report over on Signal's github.

I'm looking to migrate (or import) older db versions (v105 or v44) to a newer backup file - I've tried to use --migrate_to_191 which informs me that the db version is too old and stops (I can see from #233 that the earliest supported version is v132), which makes sense.

Correct, but I am certainly willing to try to bring that down to 105. There is no guarantee it will work though, while working my way down the database versions, at some point I will find a migration which requires access to the Android runtime or hooks into remote Signal servers, at which point this tool will not be able to do anything. But 105->132 is not that much, I will look into it and give it a shot.

Running it with the --migratedb flag then sucessfully exports to HTML. I then ran a regular backup-to-new-backup file operation with --migratedb and it seemed to work but didn't change the backup db version - I'm unsure if it's meant to or if this is just for the export options?

Yes this is just for a few of the export functions, it does not do a full migration of all data, just the bare minimum to get the exports to work.

Are there any plans to extend the migration functionality to older backups? Or is there a way to export the backup and re-import it

Extending the --migrate_to_191 option would be the first thing I will try, that shouldn't be too hard (depending on the contents of the migrations). I don't think many other options are very promising, but if extending the migration fails, I might attempt to hack together something to make a direct backup-to-backup import possible between specific database version (using the --importthreads option). That will be a lot more difficult though, let's hope it won't be necessary.

opening the dumped database.sqlite in n++ looks to my inexperienced eyes as if there's something up with the encoding

Sqlite databases are binary files which can not be generally opened/edited with plain text editors (though I know nothing about n++). Common ways to open sqlite databases are with the sqlite shell provided by sqlite itself (look for sqlite-tools here), or some third party application like sqlitebrowser. But, what you are thinking about doing is not trivial to say the least, and I think you'll have a hard time.

I've attempted exporting to XML and importing through the --importplaintextbackup function from #227 - of course not designed for this at all - which gave [Error]: Foreign key constraint violated. This will not end well, aborting.

Indeed, not designed for this, though I'm not immediately sure why it would give a Foreign key constraint error. The XML exported by --exportxml is however in a format (compatible with SMS Backup & Restore), that is not supported by the --importfromplaintextbackup function (which expects Signal's xml format).

Anyway, I'll let you know when I have started on expanding the migration option and you can test.

Thanks!

[bepaald]

From the log you sent me, you are indeed experiencing signalapp/Signal-Android#13034, as we expected.

I have started implementing migrating lower database versions (I think I'm at 121 now, not sure). Working my way down, the first two were immediately somewhat questionable, which is probably why I stopped there initially, but I gave it a shot. I will let you know when I'm down to 105, then you can test.

[arc-des]

Heya! I've sent you the logs (and believe it is the v191 migration it fails on) - I did get it working through incrementally updating until I could connect to the registration service, though!

I didn't raise my own issue with Signal as there are quite a few duplicates or near duplicates - most relevant is #13034, but also possibly 13506, #13485, #13470.

In #13034, nh2 and ichthyosaurus wrote about upgrading Molly progressively (Molly because they have their previous releases available), which I did try independently as mentioned, however I skipped a few versions. I tried exactly what ichthyosaurus suggested, starting from an earlier version for my v44 db, and managed to get it working on Molly 6.47.4 - I didn't test registering on 6.33.1, so it's possible that would have worked. I then exported my backup successfully, updated the app to the latest version and exported again. Running it through the tool shows the db version has updated (to 219 then 247), and I've successfully restored that into Signal proper too.

I managed to combine it with an older (2022) Desktop backup fine (though my 2024 one is returning Foreign key constraint violated. This will not end well, aborting.)

I'm still happy to test extending those migration functions though! It'd be great if this tool could do that as it was quite a bit of effort (and slightly tricky to track down that process). As mentioned in #250, I can test in either Molly or Signal (depending on which I end up using as my main messenger) or I could switch into a different profile on my device to do so.

Thank you for the info on Sqlite - I did a bit more research and agree with you that it looks quite tricky!

I have started implementing migrating lower database versions

Thank you so much!!

[bepaald]

Heya! I've sent you the logs (and believe it is the v191 migration it fails on) - I did get it working through incrementally updating until I could connect to the registration service, though!

I didn't raise my own issue with Signal as there are quite a few duplicates or near duplicates - most relevant is #13034, but also possibly 13506, #13485, #13470.

In #13034, nh2 and ichthyosaurus wrote about upgrading Molly progressively (Molly because they have their previous releases available), which I did try independently as mentioned, however I skipped a few versions. I tried exactly what ichthyosaurus suggested, starting from an earlier version for my v44 db, and managed to get it working on Molly 6.47.4 - I didn't test registering on 6.33.1, so it's possible that would have worked. I then exported my backup successfully, updated the app to the latest version and exported again. Running it through the tool shows the db version has updated (to 219 then 247), and I've successfully restored that into Signal proper too.

Yes, I thought it would be #13034 (there is actually a message from me in that issue about the --migrate_to_191 option). Very good you managed to get your database updated! Especially the v44 one, I don't expect I can support migrating from such a low version.

I managed to combine it with an older (2022) Desktop backup fine (though my 2024 one is returning Foreign key constraint violated. This will not end well, aborting.)

Ok, I think, since the backup is now an 'official' one, the foreign key violation is probably a bug. Is that using the same Desktop data as in #250? If you want to try to solve that, I probably need logs and at least the details on what foreign key exactly was violated.

I'm still happy to test extending those migration functions though! It'd be great if this tool could do that as it was quite a bit of effort (and slightly tricky to track down that process). As mentioned in #250, I can test in either Molly or Signal (depending on which I end up using as my main messenger) or I could switch into a different profile on my device to do so.

I have brought the minimum version down to 98 now, so at least your v105 should hopefully work. If you could test that, that would be great but don't feel obligated if you don't actually need that backup to be upgraded. I can imagine it would be too much trouble.

What is the end-game for all these backups? You have some old Android backups and Signal Desktop backups that need to all be merged together into a working, active Signal backup? Am I understanding that correctly?

Thanks!

[arc-des]

(there is actually a message from me in that issue about the --migrate_to_191 option). Very good you managed to get your database updated! Especially the v44 one, I don't expect I can support migrating from such a low version.

Ah yes, I saw the comment! and yeah, after I failed with my first incremental update test (I did probably 5-7 versions) I assumed it wouldn't work. I couldn't sleep the other night and decided to go through with the full method as detailed by ichthyosaurus plus the earlier ones needed to go back to v44 - about 20 apks/ update-open-checks in total till it worked.

Ok, I think, since the backup is now an 'official' one, the foreign key violation is probably a bug. Is that using the same Desktop data as in #250? If you want to try to solve that, I probably need logs and at least the details on what foreign key exactly was violated.

It was the same data as before - however this was when adding the 2024 Desktop data after the 2022 Desktop data - Combining just the 2024 data with the updated molly backup seemed to import correctly although still misses the newer data from some contacts for some reason.

More than happy to provide the logs; in fact I did generate some while testing so I'll send those over! Let me know if there are any specific commands to run (sounds like I should move this part of the chat over to #250?)

I have brought the minimum version down to 98 now, so at least your v105 should hopefully work. If you could test that, that would be great

Wonderful! Yes, I'll do so - until my backup's properly working (as below) it's not much effort for me to try :)

What is the end-game for all these backups? You have some old Android backups and Signal Desktop backups that need to all be merged together into a working, active Signal backup? Am I understanding that correctly?

Yep! So, I have a 2020 Signal Backup (and 2021, but there's some corruption) that gets my earliest messages to 2020, then a 2022 Signal Desktop backup that started in 2019, then my most recent Signal Desktop backup, which is from 2022-2024.

[bepaald]

It was the same data as before - however this was when adding the 2024 Desktop data after the 2022 Desktop data - Combining just the 2024 data with the updated molly backup seemed to import correctly although still misses the newer data from some contacts for some reason.

More than happy to provide the logs; in fact I did generate some while testing so I'll send those over!

Thanks for those. There were actually a few things in there that I attempted to fix today.

So, I have some ideas about these problems, but I'm not completely sure on any of them. Especially that imported the same desktop data can cause issues depending on whether other data was imported before...

• the foreign key errors: I think this is because in a way, the code took a small step back a few days ago. Initially, there were also foreign key errors, because the createRecipient() function failed to update existing recipients (which only matched by phone number and not by UUID). We got rid of these by allowing these updates, but this could cause Signal to crash if the identity_key was not found. So then I disabled new/updated recipients if this key could not be inserted. This caused many recipients to again not be created, bringing back the foreign key errors from before. Now when the key is not found, a fake one will be inserted. This will hopefully make the foreign key errors disappear, and also allow Signal to not crash (fingers crossed).
• Looking at the log I think maybe the one with the missing messages was run without --importdesktopcontacts? This would cause it to skip many conversations, I would guess the UUIDs of the contacts in those new conversations may just have changed.

Note, maybe a good tip: you can check the contents of a backup by adding --exporthtml [directory] to your commands (or running that on the resulting output backup). This function does not in any way guarantee the validity of the backup, it is much more forgiving of errors and missing data, but it will quickly show you what threads/messages are in a backup file without having to go through the restore process on a phone (the HTML does take up some disk space though (well, the attachments mostly)). This can also be useful going forward as I suspect there is some overlap between the to-be-merged data? You might want to use the --limittodates or --autolimittodates functions (also see the readme) to prevent doubling of messages, and the HTML export to check the results.

Let me know if there are any specific commands to run (sounds like I should move this part of the chat over to #250?)

Yes, it may be a good idea to keep the importing-bugs together in the other issue, since (I hope) they will mostly be the same bugs (or at least very related). Than we can keep this issue to just "migrating older signal versions".

Wonderful! Yes, I'll do so - until my backup's properly working (as below) it's not much effort for me to try :)

Excellent. Whenever you have time of course, no real hurry. If it turns out it's working, I think this issue could probably be closed since I don't think I will be going much lower than v98. Looking at some of the earlier migrations, I don't think I can get down to v44 anyway, though if needed I could probably always try a couple more.

Thanks!

[arc-des]

Hey!

I tried the migration again - it seems to get quite far but then fails, possibly because of the corruption on the 2021 backup (?). Running it through the tool to fix it, then attempting migration gives the same error:

To 185
Checking foreign key constraints... ok
Checking database integrity (full)... ok
To 186
To 187
To 188
To 189
To 190
To 191
[Error]: After sqlite3_step(): database disk image is malformed
         -> Query: "WITH needs_delete AS (  SELECT _id FROM message M WHERE   _id >   (    SELECT min(_id) FROM message INDEXED BY message_date_sent_from_to_thread_index WHERE       date_sent = M.date_sent AND       from_recipient_id = M.from_recipient_id AND       thread_id = M.thread_id AND       (         COALESCE(body, '') = COALESCE(M.body, '') OR         type & 0x10000 != 0      )  )) DELETE FROM message WHERE _id IN needs_delete"
[Error]: Migration failed

So if this is indeed due to corruption I may not be much help testing with my backup. Otherwise happy to give it another go - especially if I can import into Molly to test.

the foreign key errors:

Wonderful! I'll try this again and report back on the other thread.

the one with the missing messages was run without --importdesktopcontacts

This is true - thanks for catching it. I glanced over the command and misread --importfromdesktop.

The HTML export is a great tip; I've been doing so to some extent in testing but find it's often quicker to test on my phone. I used --autolimitdates for the 2022 desktop backup, but don't need to for 2024 as it starts a little after the 2022 one ends, plus using --autolimitdates causes it to not import, presumably because the update goes to more or less today as I was starting from the blank recent backup.

[bepaald]

I tried the migration again - it seems to get quite far but then fails, possibly because of the corruption on the 2021 backup (?). Running it through the tool to fix it, then attempting migration gives the same error:

So if this is indeed due to corruption I may not be much help testing with my backup. Otherwise happy to give it another go - especially if I can import into Molly to test.

I don't think this has anything to do with any possible corruption, the error you are running into is the exact same one Signal crashes at. Which actually makes sense, since I (almost) literally copied Signal's migration code. It was actually more surprising that migration has seemed to work for a few people using my code while it failed on Signal.

Good news is, I'm almost confident I have finally found the cause of this issue. I have just pushed a fix for this, so it would be great if you could test. If it works I believe I may have a fix for signalapp/Signal-Android#13034 as well.

Thanks!

[arc-des]

Hi - just tested it and it appeared to work! Importing into Molly went fine :D
Is there anything else you want me to send regarding it?

[bepaald]

Thank you, that's great to hear. I don't think I need to see any output, I'm almost 100% I've found the root cause of the Signal bug. I've already compiled and tested a patched Signal apk and confirmed at least #13034 and #13506 are solved (and possibly others). As soon as I find some time I will be opening a pull request on Signal's github.

Unless you have any more pressing matters concerning migrating old versions, I think this issue can be closed. I'm not planning on supporting lower functions, and assuming Signal gets fixed this entire functionality can probably be deprecated.

Thanks for testing, let me know if there's anything else!

[arc-des]

Wonderful, I'm really glad! I don't have anything else regarding the old migrations as far as I can tell - it all appears to work pretty smoothly :)

Thank you so much with the support on this - so to recap, this tool should work for v98 and up, and for earlier versions (at least down to v44 from c. 2020) you should be able to follow the ichthyosaurus process from above/ the 13034 thread:

Downloading from https://github.com/mollyim/mollyim-android/releases

Download Molly-v6.14.5-2-FOSS.apk - or earlier if needed (start at the latest version that seems to restore/get to the registration step and update sequentially from there) and restore the backup. Then update the app and open for each of the following in order:
Molly-v6.17.3-1-FOSS.apk
Molly-v6.18.4-1-FOSS.apk
Molly-v6.19.9-1-FOSS.apk
Molly-v6.20.5-1-FOSS.apk
Molly-v6.23.5-1-FOSS.apk
Molly-v6.24.4-1-FOSS.apk
Molly-v6.31.2-2-FOSS.apk
Molly-v6.38.2-1-FOSS.apk
Molly-v6.41.3-1-FOSS.apk
Molly-v6.47.4-1-FOSS.apk - this version worked for me for registration (follow the usual steps to restore the backup - it shouldn't ask you for a passphrase, once you point it at the backup you restored in step 1 it should jump straight to registration) but it's possible Signal will break that in the future. If you get 'Unable to connect to service,' try updating version by version until it lets you register.

Finally thank you so much for continuing to work on this, especially for such old backup versions!

[bepaald]

Thanks for the recap, glad everything is working now.

Just for any future readers of this issue: the PR I sent in was accepted and should be in the upcoming 7.27 version of Signal. So for anybody on a version equal or higher than that, upgrading Signal should just work, no need for incremental Molly versions, or this tool.

Thanks!