Implement R2 multipart upload bindings #486
Conversation
|
Ah I know why that is 😅 That's not intentional and I'll get on fixing that
We're aware of this, and we're considering changing how the etag for multipart uploads is calculated to be compatible with how S3 does it, which is deterministic. |
| // TODO: R2's multipart ETags don't seem to be deterministic, should ours be? | ||
| // https://stackoverflow.com/a/19896823 | ||
| const hash = crypto.createHash("md5"); | ||
| for (const md5Hex of md5Hexes) hash.update(Buffer.from(md5Hex, "hex")); | ||
| return `${hash.digest("hex")}-${md5Hexes.length}`; |
There was a problem hiding this comment.
Assuming that this is how S3 calculates their etags, we might move to this way of calculating them ourselves in this manner too in the future.
mrbbot
force-pushed
the
bcoll/r2-multipart
branch
from
78b6356 to
4976419
Compare
This PR implements support for writing data to R2 buckets in multiple parts. See https://developers.cloudflare.com/r2/data-access/workers-api/workers-multipart-usage/. Miniflare's implementation is briefly described at the top of
packages/r2/src/multipart.ts.Unfortunately, Miniflare 2's storage abstraction is not very good at handling large data (doesn't support streaming reads/writes), or complex read/write operations (doesn't support transactions). This limits the reliability of this multipart implementation, but it should still be useful for testing. We should aim to improve this in Miniflare 3.
Whilst implementing this feature, I noticed a few quirks with R2:
httpMetadataandcustomMetadataare always{}in theR2Objectreturned fromR2MultipartUpload#complete()etags for completed multipart uploads with the same part contents are not the same. I'd expect these to be deterministic.