createRoom succeeded, but returned 500, due to replication lag

[matrixbot]

This issue has been migrated from #8625.

---

element-hq/element-web#15512 (comment) reports a createRoom request returning a 500.

The server-side logs say:

2020-10-21 13:29:51,969 - synapse.replication.tcp.client - 232 - INFO - POST-39220 - Waiting for repl stream 'events' to reach 1605923815
...
2020-10-21 13:30:21,987 - synapse.http.server - 85 - ERROR - POST-39220 - Failed handle request via 'RoomCreateRestServlet': <XForwardedForRequest at 0x7fb379ee6cf8 method='POST' uri='/_matrix/client/r0/createRoom' clientproto='HTTP/1.1' site=8080>
Capture point (most recent call last):
...
twisted.internet.defer.TimeoutError: Timed out after 30s
2020-10-21 13:30:22,015 - synapse.access.http.8080 - 311 - INFO - POST-39220 - 109.175.168.80 - 8080 - {@webdevguru_test13:matrix.org} Processed request: 39.142sec/-0.000sec (0.121sec, 0.012sec) (0.241sec/0.546sec/58) 55B 500 "POST /_matrix/client/r0/createRoom HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86
...
2020-10-21 13:31:30,491 - synapse.replication.tcp.handler - 543 - INFO - process-replication-data-4 - Caught up with stream 'events' to 1605923815

It seems that matrix.org was still catching up after an earlier outage at this time:

I'm not sure exactly what can be done about it, but it's certainly unsatisfactory that the room creation completes and then we return a 500 anyway.