Federated v4 room with concerning power level loss/reset event

[th0mcat]

First event

Sometime between 2021-09-04 and 2021-09-15, nearly the entire moderation and administration team of the Cybersecurity-General channel (#cybersecurity:matrix.org) experienced a simultaneous loss of their power levels. The team first realized there was an issue around 19:22 GMT on 2021-09-15, when thomcat's Mjolnir instance started throwing a permissions error. Many of the admin team then tried to use the /op command in-room, and none were able to. We then looked at the room permissions, and discovered that all mutes (PL -1) were gone, as well as all mods and almost all admins. I used a backup account that was still at PL 100 to re-admin/mod the team. Whatever happened with the room propagated to every mod/admin's homeserver, including matrix.org.

Looking through the database on thomcat.rocks, whatever happened with the room happened between the two m.room.power_levels events listed below. As far as we can tell, no admin/mod user for #cybersecurity:matrix.org changed the power levels manually.

Before/After event	event_id	origin_server_ts
Before	$thoHjLQlH1Kb2egT7w75eJnwMYQd-44zgNveeAOcL9g	1630796234276
After	$biE9qvuOCLJrsIUIBmkdL5nRZ19HkmsKyH02FuCpGKk	1631745071346

Second event

After I muted a user yesterday, at some point last night (2021-09-20) or this morning we experienced the exact same event in reverse, meaning all the mutes/mods/admins were restored to their PLs from around 2021-09-04.

Before/After event	event_id	origin_server_ts
Before	$eG22BRacxdlqsFIvqlrqr-Qvd_iC1A2LBLwmt_6BRhE	1631799008283
After	$n-E8aRFvMJ9Mt9X7lpSPtpeZt-7zAafvs7GStYYJPq4	1632149386597

Steps to reproduce

• We are unaware how to repeat the issue at this time. Before the first issue happened, a user accidentally used @room in a message to the room. As the user was neither a mod nor an admin, the @room function was not triggered. However, @SolderFumes then replied to the user, quoting their message (which included the @room), which then triggered the @room function. That event_id is $nE7PwQP1ljMqeRigAGI7p5cHh4bFNW0Jw68_WdpMmGM. That could be absolutely nothing, or perhaps a bug to report to the Element team.

• I cannot identify any relevant log messages from that time period at this time, though I can filter the results through Splunk and send a copy if needed.

Version information

• Version: Currently, thomcat.rocks is on 1.43.0. I believe it was on 1.42.0 during the first event. I am unsure if matrix.org was on one of the 1.43 RCs during the issue.

• Install method: docker-compose and watchtower to automatically update the docker.io/matrixdotorg/synapse:latest container.

• Platform: Ubuntu 20.04.3 LTS, Docker container

[erikjohnston]

I think this will be due to a state reset, which we're working on ironing out, c.f. #8629

[waclaw66]

That happend to me recently when description of federated room "Schildi Android" (!bfebJVBOZMnORmkVdO:matrix.org) has been changed.
Room title is empty now, therefore a placeholder is shown...

That is usualy fixed with the next sucessful update of that room.

[th0mcat]

The room just experienced another loss of state today. Will update if/when the issue resolves itself

[richvdh]

resolving as a duplicate of #8629