FileMeta verification guidance with client store representations

[asraa]

sigstore/root-signing#376 (comment)
@kommendorkapten brought up this issue during a change where we used to serve canonicalized repository metadata, and just now switched to normal encoding/json. This technically changes the bytes that are used to calculate the FileMeta checksum and lengths, for example, used in the timestamp.json file for pinning the snapshot.

If clients write metadata to their trusted store in canonicalized form, or in some other representation other than the bytes that were directly received from the repository, then the checksums of their stored metadata no longer match the ones in the trusted FileMeta.

This is particularly important if we have a shared cache that multiple TUF client implementations share: if one writes as canonicalized bytes, and the other in the direct bytes received, then one client will fail depending on the way that the repository wrote the FileMeta.

Should clients ALWAYS write metadata to their trusted store in the exact form that was received from the remote?
Should FileMeta ALWAYS be generated from the canonicalized bytes that were signed?

[trishankatdatadog]

Should clients ALWAYS write metadata to their trusted store in the exact form that was received from the remote?
Should FileMeta ALWAYS be generated from the canonicalized bytes that were signed?

Seems like a reasonable (albeit slightly expensive) workaround for now. Does anyone else have better ideas?

[lukpueh]

Should clients ALWAYS write metadata to their trusted store in the exact form that was received from the remote?
Should FileMeta ALWAYS be generated from the canonicalized bytes that were signed?

It's either one or the other, right? If the repo creates FileMeta from the exact metadata bytes it serves, and the client uses exactly those bytes to compute the hash, then no canonicalization is needed. I think this is how the spec pictures it.

Using a canonical form of the full(!) metadata (not only the signed bytes like) would be an alternative, if above is not possible.

Another alternative would be to generate FileMeta without hashes as described in Mercury.