Synapse appears to be sending the next transaction while a transaction has yet to be processed

[turt2live]

Description

[federation_reader_1] 2021-06-27 06:28:41,991 - synapse.federation.federation_server - 190 - WARNING - PUT-12720- Received another txn 1624530264445 from matrix.org while still processing 1624530235407
[federation_reader_1] 2021-06-27 06:29:00,937 - synapse.federation.transport.server - 412 - INFO - PUT-12986- Received txn 1624530264445 from matrix.org. (PDUs: 50, EDUs: 31)
[federation_reader_1] 2021-06-27 06:29:00,937 - synapse.federation.federation_server - 190 - WARNING - PUT-12986- Received another txn 1624530264445 from matrix.org while still processing 1624530235407
[federation_reader_1] 2021-06-27 06:30:09,229 - synapse.federation.transport.server - 412 - INFO - PUT-13590- Received txn 1624530264445 from matrix.org. (PDUs: 50, EDUs: 31)
[federation_reader_1] 2021-06-27 06:30:09,229 - synapse.federation.federation_server - 190 - WARNING - PUT-13590- Received another txn 1624530264445 from matrix.org while still processing 1624530235407

Steps to reproduce

• Be behind on federation
• Watch incoming transactions stack up

[richvdh]

this is pretty much expected. The sending server makes several attempts to send a transaction; if all attempts time out, the sendng server will move on to the next one. But the receiving server might well still be processing the first one.

They don't stack up per se; the later ones get dropped.

[turt2live]

I'm not sure I see the benefit of that. The second transaction appears to be retried forever, which is something that could happen to the first transaction. Presumably the sender has tried to move on because it assumes the transaction is just bad, but it's not immediately clear how the server is meant to escape that issue if the second transaction were to also be faulty in some way.

fwiw, the attempts at the second transaction are met with 429 errors, so I guess Synapse is watching for particular status codes to determine if it needs to retry or if it should give up on a transaction?

[richvdh]

The second transaction appears to be retried forever

I don't think so. I think it gets tried 10 times or so. The same number as the first transaction, anyway.

Presumably the sender has tried to move on because it assumes the transaction is just bad,

no, it moves on because it assumes the server is down, and will be so for some time. If a server goes down, we don't try and send all the events it has missed, but rather prioritise more recent ones.

[richvdh]

I think this will basically have gone away anyway as a result of #10272.