federation device list updates cause high CPU on main process of matrix.org

[michaelkaye]

Something causes handling of m.device_list_updates over federation to take a lot of CPU time

See matrix.org homeserver.log from ~ 2020-10-21 12:45 to 2020-10-21 13:24 for logs

from a flamegraph, 95% of time was spent in synapse/storage/util/id_generators.py152

I couldn't narrow it down to one particular request or homeserver that was causing this high CPU; there wasn't a spike in requests from a single homeserver as far as I could see; and request volumes were fairly low.

The database appeared unloaded at the time, and this CPU usage had a knock-on efffect on other endpoints relying on the main process.