--importfromdesktop doesn't seem to recognize contacts present in android backup file #250
Comments
|
Hi!
This does not look right, but I'm sure that's a copy-paste error (or you're leaving out the actual backup passphrase and desktopkey)? Either way, the correct command line:
Hm, I think he progress indicator is probably showing 2700 messages processed, but not necessarily imported. In case the input backup file was empty, it is no wonder only the note-to-self thread is imported, as 'self' would be the only contact present in the backup.
As far as I know — but I could be wrong — there are three ways of getting contacts in Signal (and subsequently in a backup).
This is where it gets strange. If you have exchanged messages with someone, they must be in the backup. Maybe you could check manually if the contacts are in both the Android backup and the Desktop database: I hope this will list the contacts in the two databases. Do you see anything strange? Is the contact you exchanged messages with present in both lists? And do the If the contact is in both databases, and the id matches, Also, while we're testing, so you don't have to continuously restore the backup on your phone, you could add
Not sure yet, but I don't believe the feature is broken (I just tried it a minute ago, and it seemed fine). We'll figure it out. Thanks! EDIT Also, maybe unneeded, but please make sure the backup you are importing into is the latest (with the newly started conversation in it). It has happened before that people use the wrong backup as input file. When in doubt, again, |
|
Hey, thanks so much for the extensive reply! I've tried it again with my most recent backup, and it seems to work now. Maybe it's because the contact actually opened/saw the new message I sent before I made the backup on my phone (although they didn't reply). This maybe a noob question, but I'm still curious: why isn't it possible to just make a new backup file containing all messages in signal desktop and restore that completely? In this case my contacts are synced through the google account, so you would assume all the information the Signal app needs is there. I don't understand why signal needs to have already existing conversations in the backup, since it doesn't work that way with a normal backup. I was able to make a nice export to html with the full message archive (including profile names/pictures). It's just so tantalizing knowing all the data is there, but impossible to properly restore to my phone unless I start new conversations with all those people first ;-). |
Ok, good, that may have been it, not sure how that works honestly
That's a lot of questions, not sure if I follow them all precisely, but here goes: Your Google contacts ≠ your Signal contacts (even if there's overlap). Signal is a privacy oriented messaging app, that's the whole point of it, as such it does not share its contacts with Google (personally, I think that's a good thing). The other way around does happen: Signal reads your system contact list, but this is just to fill in data for your contacts (profile picture, phone number, preferred name, etc) but not enough to chat with them: most contacts in your system's list probably do not have even Signal installed, and if they did Google doesn't know so this information is not in the system list. This is why the Signal PIN exists, specifically to recover your contacts (among other things) in case your phone is lost without a proper backup having been made. In any case, this is not an issue for this project but for Signal, so you should take it up with them if you want. This tool can't just insert all the messages into the backup because messages reference contacts. All messages that are inserted into the database must have a 'from' and 'to' recipient (indicated by a simple number). If these numbers don't correspond to entries in the recipient-table of the database, Signal will crash. Lastly, this tool is not creating the new contacts because, well... I don't know how. Currently I'm strongly leaning towards the idea that it's is impossible: the required data is simply not present in the Desktop database. This may sound weird, but it should be noted that Signal Desktop is not a standalone application, it always needs a registered phone linked. Inserting a contact is more than just adding a name and some identifiers in the correct table in the database, there are also identity-keys (so you know you are chatting with the correct person) and encryption keys. As far as I can tell these are not present in the Desktop database (or are unique among the various Signal products). It's also possible that some sort of syncing with Signal's servers must be done in order to create valid contacts. Truthfully, I'm not 100% sure it can not be done, but every attempt I've ever made just results in crashes. (I am going to give it another shot though, let me know if you want to test it out when I have something. In your current situation I think you'd be the perfect test subject) |
…(almost done I think). Fix sorting of HTML search results. Update emoji list. Shrink searchidx.js some. Some refactoring (fileSize()). Warn when Desktop data directory is not found. Adjustments to CSS.
|
I have implemented another attempt at importing contacts from the Signal Desktop data. To my surprise, I have for the first time actually managed to do it without crashing Signal. My testing was (very) limited, I just used my three testing phones. I started with a fresh install of Signal on phone A. Linked it to a Signal Desktop instance and started two conversations, one with phone B, one with a newly made group (with members A, B and C). Then I wiped all the data again and started Signal Android on A again, not restoring any backups or entering any PIN. I then exported a backup from this phone, which was completely empty (contained only itself as a contact) and imported the data from Signal Desktop. After a few attempts, it seems to have worked. Both the private conversation and the group chat are imported and working. If you want to give this a try:
If things do appear to work, please be aware things could always break in the future. I don't think it's very likely, but it's always possible some bit of data that is not set during import is allowed to be missing today, but not in some update to Signal years from now. Two things to expect (if the import works at all):
Lastly, there are many situations that can occur that may have an effect on the success of this function. As I mentioned, my testing was limited. Some things which may cause problems (but not necessarily), which I was unable to test:
Let me know if you plan to test this feature sometime. If not (no problem of course) I think this issue can be closed as the described behavior was all expected. Thanks! |
bepaald
added
enhancement
|
Hi! Thanks so much for this tool, it's a lifesaver! This behaviour is the same on 2 desktop backups (one from 2022-06 and one from 2024-03 - the numbers are different but the error is the same); both work without the using --importdesktopcontacts, but need me to have sent a message on the "blank" backup to create an initial thread; it doesn't merge them without that. Unsure if relevant but it fails to find the key in config.json on it's own - I have to use the --desktopkey flag. Let me know if I should raise this as a bug outside of this thread! |
|
Thanks for trying this option, it would be great if I could get this working somewhat reliably. Looking at the code I did actually spot one issue, but from what I can tell it wouldn't have caused the error you are seeing (it would cause problems later, when restoring, if you'd run into it). I am currently not sure what is causing this, but maybe I can find out. Would you please update the program (I just pushed some changes), and run again the import again, this time
Not sure if I can figure anything out from the log, but hopefully this will lead me in the right direction. I might very well have more commands and queries for you to run later, I hope that's ok. Also, is the Signal account used for creating the empty backup the same as the one the Desktop client was linked to? I notice from the
I don't think it's related to the error you are seeing, but it is unexpected. You have your Signal Desktop data in some non-standard folder I'm assuming (since it's a backup from a Mac)? Are
Either way is fine by me. Thanks again! |
|
Hi! Thanks so much for this! I've just run it with those flags and it has the same error (more or less) - will send the log. I've just used the one number for Signal though I noticed just now I got a lot of [Warning]: Failed to find recipient for uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Yes, I'm using --desktopdir to point it at the Signal Desktop folder I copied from my Mac. The folder contains both The error is [Error]: Failed to read (encrypted) key from config.json Edit: Is there a way to send the log privately? Seems like GitHub no longer has a dm option If you're comfortable with email if you contact me at [redacted] (and reply here to verify) I can attach the logs. They expose some messages and contacts - or I can try to sanitise them to post here? |
No worries. Note though that this tool is available on Mac as well, through
To be honest I'm not completely sure either. When reinstalling Signal, you always need to register (enter phone number, wait for sms), even when restoring a backup. I thought doing this with an already registered phone number would give you the same (existing) account, but this is probably not the case unless also entering your Signal PIN.
OK, that's curious. Looks like a separate issue, I'll try to replicate this tomorrow. Is that error the only error printed? What happens after that, does the program exit? I ask because it should, after failing to read the encrypted key (which is the current default), fall back to attempting to find a plaintext key. Which is what you want, since you're reading an old Desktop clients data, from before encryption. I would expect it to either additionally print (note, this is missing "(encrypted)"), or to continue successfully.
My email address is in the README (search for 'mail'), and on every commit in this repo (for example the latest). But feel free to sanitise anyway, at least actual message contents and such. It would be good to be able to match things like names and other identifiers across different places though, so maybe leave them be or make sure to sanitise them the same everywhere. I have been trying to think of how these invalid messages can be inserted by this function, but haven't been able to come up with anything. I'm getting really curious, I do hope we can get to the bottom of it. Not sure how much free time I'll be having the coming days, probably Sunday should work. It may happen that I don't reply for a few days, either because I'm busy or because I need to think about the problem, but I'll get back as soon as I have some news (or more questions). Thanks! |
|
Excellent - I'll give homebrew a go when I'm at my mac. The program exits immediately after the Failed to read (encrypted) key error - I don't think there are other errors but will retry tomorrow and report back. Thanks so much for looking into this! |
|
Thank you for the log file, it has arrived safely. It must have been a bit of work redacting it all, I apologise, I hadn't realized there would be so much actual message contents and phone numbers in the output. On first glance there is a lot I can do with this, so obvious in hindsight. The program currently only matches Desktop recipients to valid Signal contacts in the Android backup. When it doesn't find one it attempts to create it. However, in your case, the Android database appears to be filled with contacts (likely from granting Signal access to your phone system address list) without the valid Signal id/keys. In this case a valid recipient is not found, but one can not be created either (because the phone/pni must be unique in the database). Instead, I must update existing contacts so that they are valid. I believe all the
Ok, that definitely sounds like a bug then. I will try to reproduce this tomorrow. Thanks again! |
|
Excellent, thank you again. That makes a lot of sense - a friend pointed out the UNIQUE contstraint failed suggests there's a duplication issue - and my contact list is by and large the same as it was then (with a few extra contacts, some name changes etc)
Thanks, sounds great! I tested it again and it just gives that error and exits: Though of course it's easy enough to work around by adding the --desktopkey manually. |
|
I had a go at this. Now the tool should hopefully update existing recipients if one with one of the unique identifiers (phone number, pni) is already present in the database:
I think this should take care of most of the issues I saw in the output.log. A particularly confusing one I noticed was Given that in these cases, both the group and the recipient were only just created by this program, I'm surprised some memberships are already present. I can not think of why this would happen, unless some group members are actually listed twice in the group's member list (in the Desktop data), causing the tool to attempt to add them twice. I added a check for this, which may produce some output. If something else is going on the existing error will probably persist and we need to do some investigating. Now it is not unlikely I made some mistakes in the new code, or that the state of the data you are feeding it is different from what I think based on the log, but I think a new test is in order. The desktop key not falling back to the old style is fixed now (it was a Windows only problem), so you could leave out Please do add the Thanks! |
|
Thanks so much! I won't be able to test this until Thursday/Friday; I'll update this then :) The signal file i tested with is either blank, or may have 1 conversation/contact (no groups) - could that have caused it? I can wipe/reregister and get a guaranteed "blank" backup if that's better? |
That's perfectly fine, of course, whenever it suits you.
I'd say don't bother, let's try to get things working as they are. A fully empty backup will not be possible anyway, since there will always be 'you' in there (and 'you' is probably one of the problem cases, because you have the same phone number still, but a new uuid, so that needs to be dealt with anyway). |
|
Thank you very much for this feature! I stumbled upon your tool while trying to fix an emergency yesterday. A lifesaver indeed <3 We had a de-coupled windows client to populate a fresh android backup, and it worked well, also importing contacts from desktop (should be on by default imho). We still had the Signal PIN if that matters. Didn't have time yet to scroll around much, but much of the old history is saved (around 4 years, 6GB or so). Thanks for the effort you put into working around Signal's design flaws! |
|
Thank you once again - I've tested the function and it appears to work in that it sucessfully imports the desktop data, including contacts, though when restoring Signal crashes on the 'Messages' screen - it sucessfully restores and lets me verify the number, and I can see a brief moment of the messages page with the threads I expect, which is great! I sent the output log and have just sent my crash log now too if that helps? I assume it's related to signalbackup-tools but I'm not sure. |
Glad to hear it. I think having the Signal PIN does really make a difference, maybe even all your contacts were already in the backup file. In that case, the import from desktop function should be working quite well by now. Please do try to scroll around as much as you can though, better to find out things are broken sooner than later, the
Thanks for testing, it seems we have made progress. But, as somewhat expected, the real debugging will start now. The crash log should definitely help, thanks for sending that. I'll take a look at it later and update here if I have news. Thanks! |
|
So, from the crash log it looks like a number of recipients were created with empty Could you, just to confirm, run the following on both the original (almost) empty Android backup, as well as on the imported backup (sigDt24_log.backup): I am expecting it to say Then, I'm curious why the function could not find the keys for these contacts in the desktop data, as far as I can tell the keys can not be NULL in the Desktop client either. Could you run: This should output all entries in the Note, the order of the fields in the json line can vary. This program is looking for the I have attempted to adjust the function now to refuse to insert empty I also plan — when I find some time — to test what happens if I just insert a fake, made-up, key into the database. En empty key causes a crash, but maybe an invalid one will only cause a "your safety number with [xxx] has changed" message, and then Signal automatically updates the key. That would be very nice as a fall-back method. This will require me to prepare some custom backup file and test on an actual phone, which always is a bit time consuming, but I plan to do that sometime in the near future. |
|
Running it on the blank signal backup returns 0 as expected, and 3244 on sigDt24_log.backup (or it's equivalent - i ran it through the tool again to decorrupt it incase that was causing crashes). Returns no results: Thanks again for this. I'm going to be away from this computer for a while now, so I'll try to get the Homebrew version working as I'll be with a Mac. If we're unable to get the identity keys for my desktop contacts I could potentially try to insert a fake key myself if you give me some pointers, then I can repackage the backup and test it again? Alternately, I could test (probably when I'm back at this machine, so a few weeks) even if we get my identity keys working - I could get a burner SIM and do so using Molly or on a separate profile if that would be of benefit; i'd like to help out a bit if I can! |
|
Hm, 3244 is a lot, I did not expect that. Could you show me the output of and I'm not yet entirely sure what is going on to be honest. When I have some theories, I will likely make some adjustments to the function and ask you to eventually redo the import and send the log again.
Yeah, maybe that would be nice at some point, but let's focus on this first. Don't worry about helping out, you're already doing enough by answering questions and running commands for me. And I'm afraid that may continue for some time, you might get sick of it eventually. |
|
On the Mac using homebrew now by the way. The output of the first is: The second is: Sounds great! Happy to run commands and answer Qs as required, though my availability will vary of course :) |
Good!
Thanks! I was actually just about to log in with another query for you to run, but you beat me to it. No hurries by the way, I won't have time to look at it anymore today anyway. Could you try this one: From what I can tell, it seems that a (very) large portion of the contacts in your Desktop database do not have any keys associated with them. In the Android database this is not possible, even for unregistered contacts, but I think in the Desktop client unregistered contacts don't need a key (this is my theory at least). Maybe that's what is happening. If you had to guess would it be possible that 3933 out of the 4020 contacts in your Desktop data are not (no longer) on Signal? I did manage to test inserting fake keys into a database, and am happy to report it just worked! I replaced the keys for two contacts with random data in an otherwise known working backup, and after restoring I just received a few "safety number changed" messages for these contacts, but as far as I can tell everything else is working. So that will be an option if we find there are indeed no keys present in the Desktop database for those contacts.
That's good to hear. Always take any time you need, of course. I will probably not be available every day either. Thanks! |
|
Yea!
It's perhaps possible - there are only perhaps 1-200 contacts I would have contacted individually - the rest are likely to be part of group chats, some of which had probably hundreds of members, if not more, and many are likely to be transient, or people I sent SMS to when Signal supported that (though would this affect desktop data?). 87 existing registered users seems quite a bit too low for the GCs, but if I'm no longer part of those groups and didn't message them individually would it affect the key situation? In which case, 87 people I've messaged individually who are still on Signal seems a tad low but within bounds.
Amazing, thank you! Definitely worth trying :) |
|
Thanks again. So I keep asking you to run these queries just because I'm trying to be sure these contact shave no keys associated with them. When I'm sure about that (or as sure as I can be), I'll just insert fake keys for those recipients now I know that works. From the above output I can tell the So, I have two (hopefully) last queries to run. First a simple one: And then, as a last ditch effort to look for a possible identity_key, or to see if there is any indication of the registration status of these contacts: This will list the full data on 3 random contacts whose identity_key is not found. There is certainly going to be personal info in there (at least phone numbers, but also names and possibly the latest message contents), so redact it as thoroughly as you like and send it by email. I am not sure how SMS-only contacts would have affected the Signal Desktop data. As far as I know Signal Desktop never supported SMS, and messages sent and received as plain SMS would never show up in a linked Desktop client. But I never used Signal in that way so I'm not 100% sure. Also maybe the contacts did get synced even if messages did not.
That is a good question, I have no idea. But it be a possible explanation, maybe keys for contacts that only exist in groups you are not a member of are deleted in the Desktop client... Anyway, I want to do a few more tests with the fake key insertion, and then I hope I have some time tomorrow to implement using those where necessary. If that's done, it's time to test the import and restore again... I'll keep you posted. Thanks! |
|
Here's the simple one: The second one I'm redacting now and will email over shortly :) |
|
From what you sent me, I'm giving up looking for those So I think it's time to test again. Again, with the Thanks! |
|
I think it mostly worked!!! Things look a bit strange with all the safety number changed notifications, but it seemed to import all my threads as expected, at least with the most important ones I've audited so far. I cannot express my gratitude enough! My old Note to Self appears as a chat with a user - possibly with the Signal name I was using before? With an 'SMS messaging is no longer supported / invite to Signal' banner at the bottom. Perhaps this is because I was deregistered? A new note to self appears if I initiate a new message to my own contact (I have not yet sent anything in this). Another minor/temporary issue I found is that I'm no longer in any group chat - most come up as Unknown, 0 members with the history still there. Legacy groups appear to have me as a member but it's the old note to self version of me. However when I'm re-added it seems to sort itself out! I have the logfile which I'll send over now, and I'm happy to do more testing for you where needed. |
That is great to hear! This was the 04/2024 import into the (almost) empty backup?
Yes, the safety number changes were expected. If they annoy you and there are too many to do in the app manually, I think they can be quite easily deleted with this program as well.
I think this is also somewhat expected. Since you created a new account, as far as Signal is concerned 'you' are no longer 'you' :) The new account is now 'you' which means the old note-to-self messages are someone else (and this old 'you' is no longer registered, which is why there is an 'invite' banner). If you want, I think we can manage to actually move those messages to the new note-to-self thread (not fully thought this out, but I think it's just a small number of
Right, that could have been annoying. And I hadn't thought of that, but again, this is to do with you having a new account. The old 'you' is a member of those groups, but the current account was of course never added. I don't think it's possible to just add the new account to those groups by editing a backup file. I haven't tested of course, but I think group-info needs to be synced with the server and across the group members. I imagine it could be a big security hole if I can add just anyone to any group by issuing an SQL statement on one local phone.
I have received it, but not yet looked at it. I will do this later of course, thank you very much for sending it. Also many thanks for your generous donation, you didn't have to, but it is greatly appreciated. Please do make sure to test as much as possible. Some possible errors would only show themselves when Signal actually loads the data (when things come into view). Depending on the size of the backup there are of course limits to what one can do, but better to find a serious problem now than in a year or more. Have you managed to send and receive messages with newly imported contacts in newly imported threads? Does that all work as expected as far as you can tell? Thanks! |
Personally I don't mind too much! The main annoyance is that it puts my contacts out of order (as all contacts refresh with the safety number stuff)
I'd like that if possible, but I can fairly easily get back to the contact as and when needed if not! Yep I guessed as much with the groups; I'll just ask to be readded to the important ones. You're welcome for the donation, feels like the least I can do for helping me with something so important to me. I'll test as much as I can and let you know! One of the 'Unknown' groups has a Message Request notification that, when I open the group, invariably crashes the app (sometimes I need to scroll down for it to trigger the crash). I know what the group is and it isn't a problem that it's broken for me - there would be no history as disappearing messages are on, and the contents that I can see is just many people's name change notifications. It probably had a few hundred members, which might be part of the issue. I can send the log if desired but otherwise I'll just delete the chat.
I believe so - sending/receiving messages seems to work! Both for new contacts, Desktop-era contacts and pre-desktop era contacts |
No problem (I think :)). I will write up some instructions when I find the time (hopefully later today, if the PR on Signal doesn't take up all my time after work).
I wouldn't mind seeing the (crash part of) the log, just to see if it's something simple, if you don't mind grabbing it. I'm not too familiar with these message request notifications, but I can imagine it is personal to the old 'you' and doesn't like to be opened by the new 'you'? Anyway, if it's not too much trouble I'd like the crash log, but feel free to delete the chat right after (meaning: don't wait for me to investigate and fix that one).
Good, that's the most important part. So, any idea what the next step is? I believe you would now want to import more data into this backup? From two Android backups and one more Desktop backup, is that correct? |
|
To move the messages from the old note-to-self to the new note-to-self:
If my brains are still working that should be it. Let me know if you encounter any problems. |
|
Hi! Thanks - for some reason running the second command results in: Unsure if this is supposed to work but running just the first part of the query before the second --runprettysqlquery hangs on a >. |
Your closing quotes aren't normal ASCII quotes, see Thanks for all the crash logs. The first one (b06e422a5715.txt) has no attachment though. The other two appear to be the same error (which is good). I will investigate, but perhaps you could help out:
If you know around which message it crashes, maybe you could
Do I understand correctly this group does not have its original title, and is listed as 'unknown'? I would like to investigate that as well then. Thanks! |
|
For the two crashes maybe you could show the output of |
|
Hi! Of course - I pasted it out and the program changed my quotes! Ran it again: thread id 450 is the new note to self, but I've not sent any messages in it. There's also another note to self - I think because I'm combining: 2017-12 to 2020-02 a 2020 backup file (updated to the current db version)
Will do when I have time! Likely tomorrow or early next week.
Ah yes, so almost all the groups started as 'UNKNOWN' as the title with full history and 0 members (including me, so I can't send messages). When re-added to the group it fixes itself - the group updates with the correct name and members and keeps the history. I haven't tried re-restoring after being readded; my guess is this 'UNKNOWN' issue is to do with me changing ID.
Sent over email as i'm unsure how much it exposes - there were approx. 60 lines, so not too many hopefully? |
Right, I see you're a few steps ahead of me already with more imports done, that's very good. In order to move the messages from the old thread to the new one, the new one must actually exist. But it will not be (properly) created until you send a message in it, so I suggest you do that (you could always delete it later). Then maybe paste the output again, the situation with multiple recipients in a note-to-self thread is a bit unusual, but shouldn't be a problem. I'm curious if the new note-to-self will have a recipient matching one of the ones in the old thread.
No need, thank you. Thanks to the email you sent over, I managed to find the problem. I normally skip over all "group-v2-update" type messages, they are complicated and it would be too difficult to get working (if it's even possible at all). However, when doing the import only to export directly to HTML, I add some of them in a way that is not compatible with Signal, but is compatible with the If you run the import again, the messages should be skipped and the errors should be gone. Alternatively, just running
Right, that makes perfect sense. Because of your new identity, you were not a member, and because you were not a member, you couldn't see any group-info (like title and other members). I guess that can't be helped then.
Yes, very helpful, thanks! |
|
Hi!
So these are the results from the pretty SQL query with what are definitely the right chats and get the following result (possibly due to the ID change?): So it appears you were right! Attempting to add
Excellent, thanks so much! I did re-do all of the importing etc after updating the tool, so hopefully that fixes it! I haven't fully tested it yet though, but will report back (probably after the note to self is sorted or given up on). |
|
Hi! Maybe a stupid question, but when you did the Also, now that I think of it, it is probably smart to mark the old thread as empty as well. Signal will probably do this at some point automatically, but maybe not. So just to prevent the old empty thread hanging around in your thread-overview, I'd say: (using the same numbers as an example, recheck them if you made a lot of other changes in the backup in the meantime) If that does not work as expected (or that was what you already did (apart from the second --runsqlquery)), let me know (I'll probably need to see the tools output then). Thanks! |
|
Hi! Not a stupid question as I'm liable to miss things like that, however in this case I did put an output - same result. I've now used the command as above and it also doesn't work - it seems to delete the old thread while retaining the new note to self, however this was after importing the 2022 desktop backup. In the next day or so I'll try just the updated 2020 Signal backup, which did also seem to separate the old and new notes to self, then if that works proceed with the imports. I also have a sort-of note to self thread that I used possibly before the note to self function existed - I messaged a separate number that I owned but never replied from (I may have done at the very beginning but not later on). Do you think it'd work to import that into the Note to Self too, or would it cause conflicts? Thanks so much again! |
That is so curious, not sure what is going on.
Wait, isn't that the idea? Move all messages to the new (real) note-to-self, and clear out the old nts thread? Or are you saying the old nts is deleted, but none of its messages are moved?
I really don't think this is necessary, it shouldn't make a difference for this procedure. Maybe you could show me the output of: This is just the same command as before, with a slightly modified version of the initial select-query before and after it, to show changes. In this example I used the thread- and recipient-ids from one of my own databases, so replace Here you can clearly see the new note-to-self thread has 5 messages initially, the old one has 262. Then the two update-statements modify 262 and 1 row of data respectively (moving all 262 message to the new nts thread, then marking 1 thread as empty). Lastly, it shows the new nts thread now has 267 message (262+5), the old one has zero. I would love to see the output of this command in your case. Maybe throw
I think the same procedure should work, but it's hard to say that confidently since it doesn't seem to work in the other case either currently... Thanks! |
Yep - I think so at least, but have now wiped signal so can't double check - I don't think I could find the original note to self, and the new one had no history except the one message I'd sent to it. I ran the command, replacing your numbers with the ones from mine as above (44 -> 250, 71 -> 260 and 51 -> 3) and it had another unique constraint failed, and didn't appear to merge the columns: I did try restoring just the updated db 2020 Signal backup with no desktop import, sending an nts message, exporting and running these operations and got the same results (with fewer messages in the original thread of course). I hope this helps! |
|
Ok, well the error message explains what you are seeing. The actual moving of the messages fails, because of the UNIQUE constraint, but marking the old thread as inactive succeeds. So by the end of these queries, no messages were moved, but the old thread is hidden. Now the big question, why is the UNIQUE constraint violated? The actual constraint is that in any one thread, no two messages can exist that were sent by the same recipient at the same time (in milliseconds). Now, if this constraint is violated during the move, that can only happen if either
Since the constraint is already in place, as far as I know (1) can't happen. But, since the single message that exists in the new thread is new, while the messages in the old thread are old, (2) also seems unlikely. But obviously something is going on. The following will check any already doubled messages in the old thread: This will check if any of the messages in the old thread has the same timestamp as the message in the new thread: Given the error you reported, I would expect one of these queries to return at least one result. Hopefully things may become more clear when we look at these doubled message. Perhaps one can simply be deleted, alternatively the |
|
I just tried this with my most recent test and strangely both came back with no results. I'll try it on some of the others tomorrow or Saturday to confirm! |
|
Ok, well I'm probably losing my mind then :-) Seriously, I'm not really sure what's going on, I need to think about it. One note: do make sure that whenever you try any commands (both the actual thread-merge, as well as the queries for investigating the problem), you run them all on the same backup file. Or, if you do use a different backup file (with more or less or other data imported into it), recheck the thread- and recipient-ids again (like here), because they may change. |
|
Very strange! I did go back and double check with multiple backup files, running --listthreads first, with the same result: and: Is it possible to limit the function to specific dates? I'd be interested in trying to merge the old one from the beginning until mid 2024 into the new one, or possibly limiting the new one to November this year and see if that changes anything? Or if that doesn't work importing a single day of the old one into the new one to see if that has any effect. Or just importing certain types of messages (messages, replies and media/attachments) in case it's a similar issue as before with weirder message types causing conflicts? Ultimately merging these would be a nice quality of life improvement (mostly so I can annotate old notes within the thread which is something I used to do) however if it isn't possible that's fine - now that Signal has chat folders I can put my various old notes to self in a folder and access them easier that way. |
Yes certainly. To get the oldest and newest message from the old thread, use [~] $ # remove the last 3 digits, to go from milliseconds to seconds:
[~] $ date -d @369655200
vr 18 sep 1981 12:00:00 CEST # note this is localized dutch, it should look different for you
[~] $ # and the other way around:
[~] $ date -d "Sep 18, 1981 12:00:00" +%s
369655200 # to use this number in a query, add 3 zeros to convert from seconds to milliseconds.Then you can limit the moving of messages to certain dates by appending the I plan to actually make a (temporary) custom function for you to move the messages, which will do it one at a time, and hopefully be able to provide some feedback on when and why it fails. Maybe later today, but probably tomorrow.
I understand. But honestly I'm just so curious myself now. I actually had a dream last night that I solved this, but when I woke up it was just dream-nonsense unfortunately :-) Thanks! |
|
I've added a function for moving messages from one thread to the note-to-self thread. It basically does the same checks and procedure as the commands I posted earlier, but prints some more info when it fails. Also, it moves the messages one by one, so if only a few messages cause the UNIQUE error, all others should still be moved over. Please try it out, and let me know the output. Thanks! |
|
Hi! Thank you so, so much for this. I've now tested this and it appears to work well! It skipped over 44 messages, 43 of which are still in my old Note to Self, plus a '[old_user] reset the session' message at the top. They are all from March 2019 or earlier, around the time/before the Note to Self function was officially introduced, so these are messages that appear to be both sent and received by me at the same time - almost certainly the reason for the issues before. Perhaps this is why the queries didn't find them - were they looking for duplicates that were either received or sent at the same time, or duplicates with the same timestamp regardless of sender/recipient? The new Note to Self appears to have all, or nearly all, of the messages from the old thread, just with the sender and recipient flipped, which makes sense. There are a few duplicates that were imported, but these have slightly different timestamps - they appear to originate from the desktop client which may explain the delay. There are also some test SMS' I sent to myself which also appear as duplicates but with slightly different timestamps for the received time (sent time appears the same). I expect this is an issue that would only apply to pre-official Note to Self introduction messages. I'll send you the actual output when I'm back at my machine! I'm intending to do a few more things - I found another old Note to Self (I believe an artefact of the Desktop imports) and there's a friend's chat who changed number that I'm planning on consolidating. Finally (I think), is it possible for me to rename the 'UNKNOWN' chats that I won't be added back into? There are a few with people I'm no longer in contact with and it would be helpful for archival purposes to have the name reflect the contents. Please let me know if there's anything else you'd like me to test, aside from sending the output. Edit: sorry for infecting your dreams; it's sad it wasn't a Mendeleev moment! |
Right! I was mistakenly assuming the old note-to-self contained only 'real' note-to-self type messages. But if the thread started as just messages sent to yourself (I forgot that used to be a thing), some of my assumptions do not hold. Combined with the Frankenstein-like nature of your database now, with parts imported from all sorts of origins (and consequently some messages being matched to the new 'you', and some to the old), I think everything we're seeing can probably be explained. Are you mostly happy with the note-to-self as it is now, or are there any more changes you want to make? Since normal (current) note-to-self threads only contain outgoing messages, I didn't bother to adjust that, thinking any moved messages would also already be outgoing. But in your case, if you are seeing incoming messages in your note-to-self and you want them switched, let me know, it shouldn't bee too difficult (yes, I remember saying that before :-)). In fact, I think this should do it:
Let me know if you need help with that. Depending on how you do it, merging threads could become complicated. I did have a function in this program specifically for this case (number changes):
For individual (non-group) contacts, the best way to do this is by simply giving these contacts a 'nickname', which is an official Signal feature and can be done in-app. For groups, I think it's not too difficult, but small disclaimer: I expect the group-titles were already set during import from desktop and the titles were changed to 'UNKNOWN' after the backup was restored and Signal asserted you (the new 'you') were no longer a member. It is possible the same thing will happen again (but I think not). To set a groups title you would run:
Will do, I think I'm good at the moment, but thank you.
No worries, I should really learn to relax a little sometimes, that's on me. I hope we can get your database in a satisfactory state sometime in the future. Let me know how it goes with the suggestions in this post, and of course, if there anything else. Thanks! |
Because of a freshly reported bug in the To check, please run: If the output shows If you do see results from this query, I expect the If you see anything else (an error, or there are results, but they don't look as expected), please let me know. Thanks! |
|
Hi! Thanks for this :) Running I also ran I also sucessfully integrated the other chat by using the usual set thread_id function. I haven't yet tried renaming the groups - it's fairly low priority for me at the moment so I'll probably do it in the next few weeks. Once again thanks so much for all your help! |
|
Great to hear, only good news it seems!
Perfectly fine of course, take your time.
And to you as well. Your help in developing the importcontacts-option, and with identifying and fixing bugs has been invaluable! Cheers! |
Hi, I've tried using your marvelous tool today to restore data from Signal Desktop to an android backup. My phone died completely so there's no other way to restore the data.
I've tried using the syntax described here:
#57
In my case: signalbackup-tools_win.exe signal-2024-10-10-13-48-26.backup --desktopkey --importfromdesktop --output new.backup
However, when I import the resulting file in android only my 'note to self' messages are present. This is despite the app's progress indicator showing ~2700 messages being imported.
I'm not completely sure how I can get the contacts that are present in the desktop data to be present in the android backup file as well. I got a bit confused with the wording in that part of the readme.
However, I've tried this with both a completely empty backup and a backup where I've started a new conversation with a person I have a huge message history with. But it didn't work in either case. The history still doesn't show up after importing.
I hope you have an idea about what I'm doing wrong, or maybe the feature is indeed broken.
The text was updated successfully, but these errors were encountered: