Create release test plan

[eloquence]

For workstation RCs and releases, we want to put together a standard test plan similar to the ones we follow for SecureDrop Core. For the client specifically, https://github.com/freedomofpress/securedrop-client/wiki/Test-plan can serve as a starting point, but of course a release test plan should be more comprehensive and cover provisioning as well.

[eloquence]

WIP here:
https://github.com/freedomofpress/securedrop-workstation/wiki/Workstation-Beta-Acceptance-Tests

[kushaldas]

Large file access:

• 180MB, downloaded properly
• 208MB, downloaded properly
• 380MB, error saying RunnableQueue raised an exception: KeyError: 'filename', but, the server shows response code 200, with 288867401 as data size.
• 380MB same file, downloaded properly this.

Also, the file size mentioned in the client is wrong for all the files, known issue?

[kushaldas]

For the number of sources, in a local system make dev container + the latest client can easily handle 300+ sources, for 1000 sources, it downloads everything, and then the UI gets non-responsive.

[eloquence]

Also, the file size mentioned in the client is wrong for all the files, known issue?

Please elaborate. Files are compressed server-side and decompressed client-side, so I'd expect some difference at different stages of the process. What are you seeing?

[kushaldas]

Please elaborate. Files are compressed server-side and decompressed client-side, so I'd expect some difference at different stages of the process. What are you seeing?

For example: the file which was showing 208MB in the client, is actually 231MB in the server.

[ntoll]

@kushaldas where was the key error thrown from (line of code if possible, or better still a stack trace)..?

[redshiftzero]

Regarding this KeyError during download, my hypothesis is that the cause is freedomofpress/securedrop-sdk#111. My investigation went as follows:

1. Since the error (RunnableQueue raised an exception: KeyError: 'filename') is occurring during the RunnableQueue's execution, this is an exception being raised when we call job._do_call_api.
2. Next, looking at the call API logic in the download submission job, there's nothing (that I can see at least) that would raise a KeyError (it would be an AttributeError if anything), the issue must be elsewhere in the stack.
3. Next, looking at the response that securedrop-proxy provides, it is indeed the case that if there is some sort of issue during download, there will be no filename in the body of the JSON response, instead there will be an error key. Getting warmer.
4. Looking at the SDK, there is a place where we incorrectly expect the filename key to be present, I think this is the cause: proxy-only KeyError when file fails to download securedrop-sdk#111

[ntoll]

@redshiftzero ack. I got as far as your step 2 and wondered about wider context.

Given this is a problem with the SDK and not the client should we mark as "won't fix" in the client or add some sort of guard in the code for bad data in the JSON payload? My preference would be for the former.

[eloquence]

We do have a draft test plan here:
https://github.com/freedomofpress/securedrop-workstation/wiki/Workstation-Beta-Acceptance-Tests

This issue has been informally used to track some QA findings, but I recommend we open a form release ticket for that purpose, and close this issue once the test plan is satisfactory.

@zenmonkeykstop What's left to do on the test plan itself, in your view?

[redshiftzero]

I'm gonna close this one as we're close to release and have been using the above test plan