Test client with highly populated server

[eloquence]

We'd like to understand better how the client performs when there are a lot of submissions stored on the server. While it's recommended to treat the server more like an inbox, some news organizations are likely to keep a high number of submissions on the server. We should test how the client performs with a highly populated server, e.g. (in decreasing plausibility):

• 100 submissions
• 1000 submissions
• 10000 submissions

We should also test variations with a high submission/reply footprint (e.g. 50 sources with 200 submissions and 100 replies).

Questions we're most interested in:

• how long does a full metadata sync take to perform?
• are there other performance bottlenecks, e.g., in UI updates and general client interaction?

It's possible to populate a server using the existing qa_loader script. The testing should be done with a staging or prod server, in Qubes. It would be good to insert timing measurements into the client code, in particular, for metadata syncs.

This should help us arrive at more realistic timeouts for operations like:

• get_sources
• get_submissions
• get_all_replies

[sssoleileraaa]

This is basically what I created this issue for: #648 but yours is clearer so let's keep yours

[eloquence]

In freedomofpress/securedrop-workstation#399 (comment) @kushaldas responded that the client becomes non-responsive at 1,000 sources with the use of the dev container.

With priority given to known release blockers, we've agreed to target doing some additional time-boxed profiling during the 2/20-3/4 sprint in the 100...1000 range, to identify potential performance bottlenecks that can still be addressed before the pilot.

[eloquence]

We've done significant testing since then and are working on fixes like #944 and #984, this is also covered in the QA test plan. Closing this issue for now in favor of landing WIP & then picking off other known ways to improve performance, e.g., through improvements of the server-side API (freedomofpress/securedrop#5104).