-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support file block copy on write on Windows #86681
Comments
Tagging subscribers to this area: @dotnet/area-system-io Issue DetailsOn ReFS, Windows supports a block copy operation, like copy on write but at a sub-file level. https://learn.microsoft.com/en-us/windows/win32/fileio/block-cloning This has existed for several years, but is suddenly more interesting now because of the announcement of dev drive. We should enlighten File.Copy to use this on Windows when possible, so that MSBuild, Nuget etc work extra fast on dev drive.
|
@hamarb123 if you comment on this issue, I'll be able to assign it to you. |
Comment :) |
cc @rainersigwald this should speed up MSBuild if it isn't already doing hardlinks. |
MSBuild has an option to do hardlinks (the |
sharing today's post for context -- https://devblogs.microsoft.com/engineering-at-microsoft/dev-drive-and-copy-on-write-for-developer-performance/ |
There's a Copy On Write MSBuild SDK that utilizes the ReFS CloneFile API for a custom implementation of the Copy MSBuild task. Might be useful. |
Yep, I think @hamarb123 is going to crib from that (https://github.com/microsoft/CopyOnWrite) |
I have been delayed in making a PR for this due to BSODs caused by my code/Windows, which were hard to track down. I've now found the root cause, and documented it at: microsoft/CopyOnWrite#24. Hopefully I can make a PR with a workaround soon. |
@hamarb123 how's it looking - I see they're fixing the BSOD (nice find) |
@danmoseley I've been working on some other projects this last week, I plan to have a go at getting a working PR ready tomorrow - assuming there's no other issues like this one, it should be do-able. Thanks |
@hamarb123 Sounds good - I actually didn't intend to hurry you.. was just curious |
@hamarb123 actually I'm now starting to have my eye on platform shutdown date for .NET 8. I believe this repo is about to stop feature work, but historically have allowed community contributions at all times into main, occasionally holding risky commits until after branching. Do you expect to have a PR soon? If not that's totally fine but I may ask around to see whether there's someone who can look nearer term. |
@danmoseley sorry, it slipped my mind. I have mostly functional code, will make a PR after confirming the code still currently works locally. |
What does this mean for, copy action in explorer (likely not), or the |
@erikmav do you know? ^ |
It sounds like a new syscall available? |
Not a new syscall, the Windows team liked the CoW approach and numbers from the .NET approach (https://aka.ms/EngMSDevDrive) and I've been working with them to turn it on by default in the I'm partway through perf testing the canary bits on internal repos with and without the .NET package turned on. Keep an eye on the Eng@MS blog for more info as soon as next week. |
Assuming the feature makes it to the February release without semantic changes, it would resolve this issue. Better to wait to resolve this until it's known that clone is on by default in the final release. |
FYI latest in blog series (encapsulates what's already here): https://devblogs.microsoft.com/engineering-at-microsoft/copy-on-write-in-win32-api-early-access/ |
I updated my machine to 24H2 recently, and the plain copy command seems to be CoW cloning on regular ReFS drive (not Dev Drive). Doing it in the syscall itself would probably have better performance. Do we still need this functionality? It will only benefit a limited set of configurations, and not required for "the future". |
No, I don't think so - I assumed the conclusion was that we will just benefit from OS support when available, and not worry about the other cases, since they're rare anyway. If it's still wanted, I could have a go at implementing it some time, but I had just assumed it wasn't needed anymore. |
On ReFS, Windows supports a block copy operation, like copy on write but at a sub-file level.
https://learn.microsoft.com/en-us/windows/win32/fileio/block-cloning
https://learn.microsoft.com/en-us/windows/win32/api/winioctl/ni-winioctl-fsctl_duplicate_extents_to_file
This has existed for several years, but is suddenly more interesting now because of the announcement of dev drive.
We should enlighten File.Copy to use this on Windows when possible, so that MSBuild, Nuget etc work extra fast on dev drive.
The text was updated successfully, but these errors were encountered: