Profile the matrix.org master process

[neilisfragile]

Then determine what else should be migrated away.

[clokep]

I threw py-spy at a the main process and was not able to find anything terrible obvious, but there were a few observations:

• 5 - 10% of samples were pruning old IPs.
• 3 - 5% of samples were updating stats deltas

Additionally I spent some time looking at a dummy worker that wasn't working on anything (it was essentially sitting idle, but using ~20% CPU). I was able to profile that without the --nonblocking flag which gave slightly more interesting results:

• The "main" JSON encoder/decoder being used is still simplejson (see switch back to stdlib json #7674).
• ~20 - 25% of samples were in the (Redis) replication code, ~15% here and ~8% here.

These numbers are all made worse if you look at only what has the gil.

[clokep]

• 5 - 10% of samples were pruning old IPs.

• 3 - 5% of samples were updating stats deltas

In addition to seeing something like the above it also seems that a decent amount of time is spent in:

• The executemany call in _rotate_notifs_before_txn.
• _fetch_event_list shows up sometimes as well.

When including all threads, most of them are waiting in new_transaction.

[clokep]

The current work in flight from this is to move some background tasks out of the main process.

[richvdh]

I don't think we're planning to do any more profiling in the near future.