Reuse SSL contexts for .well-known lookups

[richvdh]

While I was investigating why Synapse 0.99 appeared to be slower than Synapse 0.34 (#4713), I tried disabling the .well-known lookup in the federation agent. Doing so appeared to make my synapse settle at 1.5G of memory usage rather than 2G.

The .well-known cache sits at around 1500 entries, so I can't believe it's all coming from there - it would imply about 333K per entry, which seems immense. I wonder if the connection pool is hanging on to data it shouldn't. It would probably be fruitful to investigate with a memory profiler.

[dkasak]

#4713 was closed in favour of this bug, but this bug only mentions memory. To me it also appears that our synapse 0.99 instance is much slower than it was on 0.34. Should there be a separate issue for the performance regression?

[neilisfragile]

If you have details of a performance regression that is not caused by memory usage, then please do create a new ticket.

[JJJollyjim]

I would guess that this shares the root cause with #5395 (assuming synapse 0.99 validates SSL certs during .well-known lookups)

[richvdh]

I would guess that this shares the root cause with #5395 (assuming synapse 0.99 validates SSL certs during .well-known lookups)

yes, absolutely. I think the only reason it was bad rather than catastrophic for .well-known is that we tend to make fewer connections for .well-known lookups, and then cache the results for a while.

We can probably apply the same fix to .well-known lookup that we did to #5395.

[Ralith]

As of 1.0, my notoriously beset homeserver, which historically settled around 1.4GB on a good day and occasionally exhausted my 4GB of ram entirely and brings down the whole server with swap thrashing, is using under 400MB.

[ara4n]

that might just mean that it doesn't have any extremities atm and so isn't having to do much state resolution

[Ralith]

I've actually been keeping a close eye on extremities lately; both before and after, no room has had more than 4 and only a handful have had more than 1. The improvement is without doubt due to a change between 0.99.5.2 and 1.0. I think #5417 has fixed a serious long-standing problem.

[ara4n]

anecdatally, my HS has gone from idling at 1.3GB to being at 650MB fwiw.

[richvdh]

I'm turning this issue into a tracker for applying the #5395 fix to .well-known lookups.

[aaronraimist]

This was fixed in #5794 right?

[richvdh]

yes!