State of #dendrite:matrix.org is broken

[bones-was-here]

Background information

• Dendrite version or git SHA: 0.8.5
• Monolith or Polylith?: monolith
• SQLite3 or Postgres?: postgres
• Running in Docker?: no
• go version: 1.17.9
• Client used (if applicable): Element web

Description

• What is the problem: Dendrite's view of the room differs hugely from matrix.org's view, to the point that it's unusable. It considers the room to be invite-only, so it rejects the joins and messages of all users who joined after the room was (temporarily) invite only. It considers @neilalexander:dendrite.matrix.org to not be in the room.
• Who is affected:
• How is this bug manifesting:
• When did this first appear: it has never worked properly, but is worse than it used to be

Steps to reproduce

Other Dendrite users have reported a similar problem in chat, including that other users on their homeserver cannot join the room.
Possibly related to #2129

[neilalexander]

Please check your logs for any eventauth: entries for room ID !RcWPWcZrMeBxOGaalX:matrix.org. If there are things being rejected, those entries should reveal why.

[bones-was-here]

I only see hundreds of not in room and can't join invite only

level=warning msg="Error authing soft-failed event" context=missing error="eventauth: \"@nahsi:nahsi.dev\" is not allowed to change their membership from \"leave\" to \"join\" as join rule \"invite\" forbids it" event_id="$Fm8afJFf8HwshuT7UaSQ3ZZ5N5fOn8dvMwZcRIQh3ZY" kind=KindNew origin=nahsi.dev room_id="!RcWPWcZrMeBxOGaalX:matrix.org" type=m.room.member
level=warning msg="Stored rejected event" context=missing error="<nil>" event_id="$Fm8afJFf8HwshuT7UaSQ3ZZ5N5fOn8dvMwZcRIQh3ZY" kind=KindNew missing_prev=false origin=nahsi.dev room_id="!RcWPWcZrMeBxOGaalX:matrix.org" soft_fail=true type=m.room.member

level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@cnx:loang.net\" not in room" event_id="$iKculhDpJypnzpFVmPwmE7yCfytp7W7qsZ3IynaxkyU" kind=KindNew origin=loang.net room_id="!RcWPWcZrMeBxOGaalX:matrix.org" type=m.room.message
level=warning msg="Stored rejected event" context=missing error="<nil>" event_id="$iKculhDpJypnzpFVmPwmE7yCfytp7W7qsZ3IynaxkyU" kind=KindNew missing_prev=false origin=loang.net room_id="!RcWPWcZrMeBxOGaalX:matrix.org" soft_fail=true type=m.room.message

This dendrite has been in the room since shortly after you created it and the problems started quite early, but I don't have logs going back that far (just the systemd default of about a week).