az storage copy does not preserve sparseness of page blobs copied between storage accounts

[rogeraustin]

I have used az storage copy to copy a VHD page blob between storage accounts (same region and subscription). The VHD has a nominal size of 127 GiB. The 'used capacity' metric for the source account shows that the blob uses about 6 GiB whereas the same metric for the destination account shows it using the full 127 GiB. The calculation described in https://docs.microsoft.com/en-us/azure/storage/scripts/storage-blobs-container-calculate-billing-size-powershell tells the same story.

It seems that the sparseness of the blobs is not preserved by the copy process. A similar issue was noted for az storage blob download (#5872) and has since been fixed (Azure/azure-sdk-for-python#7555).

A comment on a similar issue for Azcopy (Azure/azure-storage-azcopy#391 (comment)) notes that this issue (loss of sparseness) did not happen when using az storage blob start-batch to perform the copy. From the follow-up in Azure/azure-storage-net-data-movement#194 it appears that there are at least two different back-end copy methods, one sync and one async, and that the async method preserves sparseness while (presumably) the sync method does not.

I would like to use az storage copy because it is synchronous, but I would like it to preserve sparseness.

In the meantime, I will investigate using az storage blob copy start and a wait loop around az storage blob show.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: b8c4e415-fc8d-26d5-3213-da937fd6932c
• Version Independent ID: dcb2f2d6-fd80-495d-8582-8be859d68792
• Content: az storage
• Content Source: src/azure-cli/azure/cli/command_modules/storage/_help.py
• Service: storage
• GitHub Login: @rloutlaw
• Microsoft Alias: routlaw

[rogeraustin]

I have just noticed, under the closed issues on this page, that az storage copy uses azcopy under the hood, so this is the same issue as Azure/azure-storage-azcopy#391.

[haroldrandom]

@jsntcy Please take a look

[rogeraustin]

Reading Azure/azure-storage-azcopy#391 more closely, it seems that the sparseness issue has been largely solved in Azcopy 10.3.0. However, looking at the log files in my .azcopy directory, it seems that az storage copy is using Azcopy 10.1.0.

[Juliehzl]

owever, looking at the log files in my .azcopy directory,

Hi @rogeraustin, thanks a lot for you feedback and investigation!
Azcopy 10.3 is supported in azure cli now. Please try with the latest azure cli.
I will close the PR now and Iif you still have the problem, please feel free to reopen it or create a new one.