Events recieved out of order occasionally (days apart)

[catfromplan9]

Description

There is no trend to it as far as which server this occurs on, it just happens periodically in random rooms. Could be because a large HS deleted all accounts and this built up a large federated event queue? Unsure. Message date says today but it is listed where 3 day old messages are, that sort of thing

Steps to reproduce

• unsure

Homeserver

Synapse

Synapse Version

1.83.0

Installation Method

Debian packages from packages.matrix.org

Database

Postgres

Workers

Single process

Platform

Debian sid, bare metal

Configuration

No response

Relevant log output

N/A

Anything else that would be useful to know?

No response

[catfromplan9]

I'd like to give you logs but I'm not sure how I'd find them considering how much logs there is to sift through

[reivilibre]

This isn't entirely unexpected if a homeserver is having e.g. outages or network connection problems.
You also might see this sort of thing if e.g. your homeserver is IPv6 or IPv4 only but some other homeservers in the room are the opposite.

I appreciate it's hard to fish out any details to investigate this on your end but it is also hard to say anything interesting without any details :)

[catfromplan9]

This isn't entirely unexpected if a homeserver is having e.g. outages or network connection problems. You also might see this sort of thing if e.g. your homeserver is IPv6 or IPv4 only but some other homeservers in the room are the opposite.

I appreciate it's hard to fish out any details to investigate this on your end but it is also hard to say anything interesting without any details :)

Seems like now its only happening for matrix.org and most other HS in room have the issue

[catfromplan9]

Ok, now its not having issues for some messages. Its hard to describe this, just buggy behaviour regarding event federation in general I suppose?

[catfromplan9]

Did some talking to server admin and this might be a state reset bug

[catfromplan9]

Please fix state resets someone, this makes matrix unusable for anything large scale if randomly some bans/room topic setting can just undo themselves, or some federation breaks between random servers

And yes, new issue discovered is that an arbitrary list of servers cannot see messages from all the other servers that are not those. Its nothing specific with the servers, nothing remarkable. Seemingly just complete random odds on if server will break in room

[H-Shay]

I'm a little unclear on what's being reported here - it sounds like you believe the buggy behavior to be related to state resets? If so we should close this issue in favor of #8629

[catfromplan9]

I'm a little unclear on what's being reported here - it sounds like you believe the buggy behavior to be related to state resets? If so we should close this issue in favor of #8629

State reset is an issue, but also events do not federate to some servers from some other servers. Maybe this is a part of the state reset problem? I'm not sure

[H-Shay]

Right - given that there isn't anything specific we can investigate here, I am going to close this.