Spaces summary API sometimes has poor performance

[grinapo]

Description

Spaces room list (_matrix/client/unstable/org.matrix.msc2946/rooms/<room_id>) takes considerable amount of time to serve (20-60 seconds), which does not seem to be either justified or desirable. (A good example is #osm-space:matrix.org, which contains rooms and spaces as well and takes 20-30 seconds.)

First, room list is quite static most of the time, so caching it (and invalidate cache on change) seems to be desirable, especially if the long query time is justified.

However I do not see any real resource usage there (admittedly without analysing the code): both cpu and psql seems not to be busy.

Steps to reproduce

• see space room list
• observe the spinner

Version information

• Homeserver: grin.hu
• Version: 1.38.1

[clokep]

There's been reports of this, but I haven't yet had time to look at why it is slow. I don't plan to look at this until after #10495 since the API is changing anyway.

[clokep]

Now that #10495 has landed, we can take a look at this. I'll likely only handle this for the new /hierarchy endpoint, not the original /spaces endpoint, but overall that should make things better moving forward.

My plan here is two pronged:

• Support running the /hierarchy endpoint on multiple workers:
  • Persist the pagination sessions to the database.
  • (Maybe?) Update something about fetching data over federation (i.e. do we care about the current sharding?)
  • Update the worker documentation.
• Explore caching of requests made over federation.

[clokep]

Another thing to do here, if necessary, might be to store the space relationships in a dedicated table. I'd like to see all of this running for a bit before doing things such as that!

[clokep]

The last remaining bit of this (for now) is to test that running this on workers is fine (and document that). Once this is in the wild for a bit we might see unexpected performance issues though.

[clokep]

Most of the changes discussed in here are in Synapse 1.42.0, although 1.43.0 will include some more changes (worker support, properly stripping the m.space.child state events).

As of now, I think I'd like to see if those have good impact on the endpoint before trying to do more optimization. I also might need additional help getting profiles from smaller servers as I think the performance depends quite a bit on what data is being fetched over federation.