media repo workers use an awful lot of memory

[richvdh]

A lot of it gets cleaned up when we eventually do a gen 2 GC, but 44G seems a bit insane for a process which is essentially a glorified web server:

It would be good to find out what is using that memory, with a view to either not using it in the first place, or making it easier to free so that it gets removed in a gen 0 or gen 1 GC.

[richvdh]

(this tends to cause the box to go into swap, which is obviously bad)

[benbz]

FYI the media repo's memory profile seem to change after a deploy on 18th Jan:

[erikjohnston]

Previous issue: #8653, though the situation seems to have changed somewhat

[richvdh]

I don't think it's minor.

[michaelkaye]

So we have a tactical cronjob in place on matrix.org for this ; are we going to be able to look into this soon or should we make the automatic restarting more robust?

[clokep]

This might be a regression from #9108 or one of the earlier PRs updating that code.

[clokep]

From @richvdh doing a bit of research:

it's mostly discovered by introspection of the heap; but treq.client._BufferedResponse is where the response gets buffered in memory

Looking a bit at this I think setting unbuffered=True on the call to treq.request "fixes" this. I'm not sure of the other ramifications of this though, so looking into that.

[clokep]

Hopefully the changes in #9421 will help out with this, but I guess it'll be a few days before we know.

[clokep]

After this change it seems that the huge spikes in memory allocation are gone:

It also seems that nothing is being allocated into gen2 anymore:

Due to the above I'm going to call this complete. But @richvdh or @benbz please let me know if I'm missing anything here!