Change DecryptNotesResponse to use a sparse serialization format

[andiflabs]

Summary

In the vast majority of the cases, a DecryptNotesResponse will contain zero or close-to-zero decrypted notes. It makes sense to have the serialization of DecryptNotesResponse optimize for those cases.

This changes the serialization size of DecryptNotesResponse from O(null_notes + non_null_notes) to O(non_null_notes).

This introduces a small size overhead in the case where the are a lot of decrypted notes. We could in theory have DecryptNotesResponse choose between sparse and dense serialization based on how many decrypted notes there are, but that case is so rare, and the overhead so small, that it's not worth optimizing for it.

Testing Plan

Unit tests

Documentation

N/A

Breaking Change

N/A

[hughy]

do we need to return the null notes at all?

maybe i'm missing something but it looks like we always filter them out

[andiflabs]

Yes: upper-level code needs to know which nodes exactly were successfully decrypted. Here's an example: suppose you want to decrypt all notes in this block:

block:
  transaction A: [note 1, note 2, note 3, note 4]
  transaction B: [note 5, note 6]
  transaction C: [note 7, note 8]

You can submit a decrypt request with all the notes:

[note 1, note 2, note 3, note 4, note 5, note 6, note 7, note 8]

Once you get a response, you need to know which notes exactly were successfully decrypted, so that you can match them with the corresponding transactions. This is done by comparing the index of the non-null notes in the response, with the index of the notes in the request.

If this response returned only non-null values, it wouldn't be possible to decrypt an entire block at once (or even multiple blocks).

[andiflabs]

An alternative approach that I considered is to change the input from just note to (block, transaction, note), and repeat the block/transaction information in the response. This however would greatly increase the size of requests, and that's why I didn't go that route.