Element-R: Manual room key export fails for big account

[BillCarsonFr]

Step to reproduce

1. Have a session with a lot of keys in store
2. Go to Settings > Security & Privacy and click on Export E2ERoom keys

=> Will fail with unknown error

Error exporting e2e keys: Error: DomException UnknownError (0): The operation failed for reasons unrelated to the database itself and not covered by any other error code.

[richvdh]

This is basically the same cause as #26488: we try and read all of the sessions from the db in one big transaction, which at best freezes the app for many seconds, and at worst OOMs or generally kills things. We'll need a flow in which keys can be extracted incrementally.

[richvdh]

See also element-hq/element-desktop#1394 which concerns import of large backups.

[richvdh]

@BillCarsonFr: why is this blocked?

[andybalaam]

@ara4n , @BillCarsonFr please could you rageshake after a failed export and link to this issue? I am getting an error in my setup but I have a sneaking suspicion it's not the same thing you are seeing.

[andybalaam]

First step done here: matrix-org/matrix-rust-sdk#3012

I am pausing this for now to concentrate on things that block Element Web R.

[BillCarsonFr]

See matrix-org/matrix-rust-sdk#2914

[BillCarsonFr]

I tried with the recently merged PR (new efficient crypto db format + use of exportAsJson), and it's now working with my account.
I have about ~200k keys, the resulting file is ~140MB and it took ~20s

So it's working 🎉
We still lack some feedback, during the process only the button is grayed out. But given that it's 20s it might be acceptable to just show some loading weel, maybe we don't need a progress bar.

IMO we can close this

[Hywan]

If an account has 1 million keys, it's gonna take 100s, be 1mn40s. I believe that if we can generate a progress bar, that could be good. Or at least: “You have a lost of keys, it can take a while, please wait and don't close this window”.

[andybalaam]

@ara4n any chance you could try exporting from Nightly? We think the existing improvements might be enough to make it work for you, and the deeper fixes we have been considering are proving difficult and time-consuming.

[andybalaam]

OK, we have confirmation from @ara4n that a large export succeeded, so we consider the fixes we've done so far to be adequate. Ideally, we'd make a fully-scalable solution but we are not going to work more on that now.