Federation no longer upholds retry_interval for exponential backoff when talking to dead servers (SYN-504)

[matrixbot]

Submitted by @​matthew:matrix.org
We're trying to connect every ~10s to dead servers despite destinations.retry_interval being 3600000ms (1h). Also, each failed request logs the connection failure 12 times...

(Imported from https://matrix.org/jira/browse/SYN-504)

[matrixbot]

Jira watchers: @erikjohnston @ara4n

[matrixbot]

Links exported from Jira:

relates to #1463

[matrixbot]

It seems we (still) have a really serious regression on federation retries
arasphere is trying to hammer dead homeservers for every event it emits

sqlite> select * from destinations where destination='tyler.cat';
tyler.cat|1452137060029|15000000

implies to me that it should be trying tyler.cat every 4 hours? (15 million milliseconds)
but i'm seeing every event I emit into matrix hq from arasphere causing at least 5 retries over federation to that server:

2016-01-07 03:16:24,131 - synapse.http.matrixfederationclient - 183 - WARNING - GET-31539 - {GET-O-371} Sending request failed to tyler.cat: GET matrix://tyler.cat/_matrix/media/v1/download/tyler.cat/LPaWtHDwcLyglmqQlsDKBySP: ConnectionRefusedError - ConnectionRefusedError: Connection refused
2016-01-07 03:17:02,587 - synapse.http.matrixfederationclient - 183 - WARNING - - {PUT-O-3794} Sending request failed to tyler.cat: PUT matrix://tyler.cat/_matrix/federation/v1/send/1451931591285/: ConnectionRefusedError - ConnectionRefusedError: Connection refused
ConnectionRefusedError - ConnectionRefusedError: Connection refused
2016-01-07 03:24:20,340 - synapse.http.matrixfederationclient - 183 - WARNING - - {PUT-O-4600} Sending request failed to tyler.cat: PUT matrix://tyler.cat/_matrix/federation/v1/send/1451931592091/: ConnectionRefusedError - ConnectionRefusedError: Connection refused

-- @ara4n

[matrixbot]

Hmm, it seems to be working correctly on jki.re

-- @erikjohnston

[matrixbot]

After restarting arasphere.net to turn on manhole, the retry time for tyler.cat seems to have been reset and is now incrementing correctly.

No idea what's going on, will continue to monitor.

-- @erikjohnston

[richvdh]

I think we should assume, that modulo specific cases such as #1737, this has gone away.

Worth noting that the limiter is applied more-or-less per api call, rather than in the federation http layer, so it's entirely possible for some retries to be happening even though a server is considered "dead" in general.