john on freelock.com had HQ disappear, and then dropped messages on rejoin

[ara4n]

See https://matrix.to/#/!cURbafjkfsMDVwdRDQ:matrix.org/$14874453102710gzqix:matrix.freelock.com. Have asked for a rageshake too. Sounds like a bad combination of yet more federation corruption (@richvdh and @erikjohnston) and possibly a regression on the IndexedDB stuff causing local echo to break (@kegsay)

[kegsay]

Local echo isn't preserved between restarts. Other than that, it's completely different code to the indexeddb stuff, so I don't know why you think this has anything to do with it.

[ara4n]

Quoting the problem description here to avoid having to follow the permalink:

John (@john:matrix.freelock.com)
whoa, that was weird
(where did my message go?)

John (@john:matrix.freelock.com)
Ok, I guess riot.im/develop must be broken right now...
I was reading through my unread rooms, and thought, hey, where's Matrix HQ? It was gone, not in my room list anywhere
I went to the directory and tried to find it, got a CORS error when searching matrix.org...
tried joining #matrix:matrix.org, and it showed me a room preview, with a message asking if I wanted to join "Coffee"
a few seconds later the join message disappeared, the room appeared on my list, and it appeared to go back to normal -- but when I wrote the "whoa that was weird" message, it completely disappeared, didn't show in the chat
(but it all looks fine in Android, and here in riot.im/app all seems normal)

So I don't know why you're talking about local echo being preserved over restarts.

The issue here is firstly that HQ disappeared (presumably a federation problem with freelock.com), and then secondly that a message sent into the room ("whoa, that was weird") disappeared entirely from the timeline. This is surely a js-sdk problem of some kind, and I genuinely don't know whether there's anything in the fairly invasive IndexedDB changes which could make this more likely. If not, let's consider it 'just' a js-sdk bug.

@freelock, please can you ping here when you've rageshaked?

[freelock]

@ara4n I did use the "submit bug" link in riot, approximately 20:18 UTC...

[kegsay]

fairly invasive IndexedDB changes

This is the problem: the indexeddb stuff was designed to NOT be invasive - if it's something which didn't get persisted (or has changed) between restarts then yes that is likely to be indexeddb. Other than that, the indexeddb stuff operates in its own world: it accumulates raw json and periodically saves it, it doesn't ever touch the objects used by Riot, including timeline entries.

This sounds like the Room is being replaced which then drops the pending messages. iirc @dbkr and I looked into the peeking code and it looks like that will blindly clobber any stored room of the same id (because we assume you would never peek into a room you are already a part of), so I would look there.

[ara4n]

(this is being tracked on our internal 'broken homeservers' dashboard now)

[freelock]

Ah...
FWIW, this only happened in Matrix HQ, so it's not like this is a problem with any "normal" rooms to my knowledge. And I haven't seen this behavior in the past few days, might already be fixed...

I don't think it was really a local echo issue -- my guess is that it's something to do with the DAG, that my message got posted to some extremity that was not at the bottom of the scroll window, and "whisked away" to somewhere else in the timeline. I think this has something to do with the inconsistent state of that particular room...

What might be interesting about this is that the window showed one "branch" but added message events to some other branch? Regardless, it all resolved fine on /app, and haven't seen the problem recently.

Cheers,
John

[richvdh]

sounds like matrix-org/synapse#1935 or similar to me

[richvdh]

could be matrix-org/synapse#1935 or matrix-org/synapse#1953. I'm not investigating this any further, and it seems unlikely anyone ever will

[freelock]

Issue has not recurred. Thanks!