strange issues because of #matrix:matrix.org

[4nd3r]

my hs started to consume lots of cpu and ram few weeks ago. can't pinpoint exact time, but around same time when federation meltdown happened. usual symptoms were very high message latency and lots of Rejecting messages in log (only messages I could connect with high loads without going too deep). restarting once per day helped, and after a while once every few hours. so basically it got worse.

after upgrading 0.19.1 and still experiencing the issue, I dug into logs:
grep Rejecting /var/log/matrix-synapse/* | sed -nr 's/.+in room (.+)\.$/\1/p' | sort | uniq

only result was !cURbafjkfsMDVwdRDQ:matrix.org (#matrix:matrix.org).

important part here is that I and nobody else in my hs wasn't in that room. I once was, but parted. after rejoining room ("event_id": "$14867256702sJbsx:XXXX.XX", it took many minutes and multiple tries) and very high load for a few minutes after rejoin (log raced like madman), everything calmed down and things have been stable since then ("event_id": "$148672624215JsZFh:XXXX.XX").

not sure what to think about it. somekind of splitbrain state where my hs knows that i'm not in the room anymore, but other hs thinks otherwise? :-(

[erdii]

i experience a similar issue with my hs i get bombarded with log warning like this:

Feb 19 17:46:07 matrix.werise.de python[10692]: 2017-02-19 17:46:07,806 - synapse.federation.federation_base - 142 - WARNING - PUT-45- Event content has been tampered, redacting $1471024686296609tuFpn:matrix.org: {'origin': 'matrix.org', 'signatures': {'matrix.org': {'ed25519:auto': 'hw++tASFhvPwWnwxmmBq8nfPCebLYHz3+8Mtx/1IMV98tDdccGCw5cz8bUz5rMA1jLrpdyIpD8P2mVgXzIoaDw'}}, 'origin_server_ts': 1471024686346, 'sender': '@freenode_LIndividu:matrix.org', 'event_id': '$1471024686296609tuFpn:matrix.org', 'prev_events': [['$1471024331295629oPWfu:matrix.org', {'sha256': '2Zm1Y2w09nfobMIv610hSRwMD/CsLGb9f4VpyrXDk+Y'}]], 'membership': 'join', 'state_key': '@freenode_LIndividu:matrix.org', 'unsigned': {'age': 16501677385}, 'content': {'membership': 'join'}, 'depth': 181879, 'prev_state': [], 'room_id': '!XqGwToACtMhvSUIypX:matrix.org', 'auth_events': [['$144133105873603UerXA:matrix.org', {'sha256': 'tLIxVK3FKyWjuJTIOuWyowxExliPkQ/NG/k6Xm408gU'}], ['$144133105873608eGkuM:matrix.org', {'sha256': '6zU10QpRxiiBQGW9FFW6/gTEJTCQ42C2P5s/I4PkPWU'}], ['$144133105873601dcEuf:matrix.org', {'sha256': '8jXHHXw11KXDuGWWJGeDgyHm30MMdnXO3lxlpi5gpLI'}]], 'hashes': {'sha256': 'Y4A1a3jqPDR6vuj0hyloaZQImLCBvEYA9YUn/fbDt1M'}, 'type': 'm.room.member'}

when the "DOS" starts my server becomes unresponsive and crashes after some minutes.

EDIT: this happened after i tried to join some rooms on matrix.org

EDIT2: it seems that my hs is initiating all these requests, because when i blocked connections from matrix.org to my federation port the warnings still came. i blocked connections in both directions for now.

EDIT3: now i get errors because my server still tries to connect to matrix.org
Feb 19 19:26:40 matrix.werise.de python[13791]: 2017-02-19 19:26:40,945 - synapse.push.httppusher - 298 - WARNING - - Failed to push https://matrix.org/_matrix/push/v1/notify

[4nd3r]

few days later, after reporting the issue, I left #matrix:matrix:org and everything stayed stable.

after upgrading to 0.19.3 same symptoms returned.

so I joined #matrix:matrix.org again and things are calm for now.

at least joining was pretty fast this time - less than a minute.

[richvdh]

Sounds like a rejoin thanks to #1953 , combined with a bunch of event spam thanks to #3422 .