-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
long file name support by default? #364
Comments
Someone will have to explain how to achieve to this in enough detail for me to do it, or someone else will need to submit a PR. |
I have a solution: |
@JFLarvoire Thanks for the pointer! Allow me to be clearer: I am not a Windows programmer. I know almost nothing about it or its ecosystem. Anything that complicates the build process is something I'm unlikely to want to add. I would rather not add additional C libraries (for Windows or Linux). If standard Windows tooling doesn't support long file paths, then I'm not sure what to do. Someone will need to explain the actual problem in depth. If it's fixable in Rust, then we should do that. Pointing to a giant C library doesn't really help me understand. Sorry. :-( Maybe you could point specifically to where long file paths are handled? |
A bit of history: This is a real problem: I have many systems at work with deep directory trees, with paths longer than 260 characters. Most publicly available WIN32 ports of Unix tools (like find or ag) break on these systems. (And as an aside, they usually also break on pathnames containing Unicode characters > \u00FF; And misbehave if there are symbolic links.) My MsvcLibX library redefines open(), fopen(), exec(), etc, and prepends that special "\\?\" string to all pathnames it receives, before passing it on the Windows' CreateFile(). It also defines Standard C library directory access routines missing in MS libc. Give me a few days: I'll try to build ripgrep with MsvcLibX, and I'll report about the difficulties and benefits. |
@JFLarvoire Thanks for the explanation! It sounds like the key problem will be that ripgrep uses Rust's standard library for file system stuff, so you can't "just" hook up fopen/exec/etc. I do believe that ripgrep should be able to handle symbolic links and Unicode characters in paths on Windows.
Interesting. I wonder if this is something Rust's standard library should be doing. @retep998 thoughts? |
MsvcLibX won't fix anything automatically for Rust because Rust doesn't use libc for IO so redefining libc functions is worthless. What would fix things is detecting non-verbatim paths that are over |
Well good luck fixing that in Rust's library. This will be really useful. I'm looking forward to getting a ripgrep version for Windows with the fix. |
FWIW, I probably won't be fixing this myself any time soon. If there were a simple way to patch ripgrep for some cases, then I'd be willing to accept a PR for that, even if it doesn't correctly handle every case. But I can't think of anything simple off the top of my head. |
Fortunately since Windows 10 version 1607 you can now enable long path support for all paths, not just verbatim paths, as long as you're calling certain W functions. All you need to do is either set a flag in your manifest to enable it for the application or edit your registry to enable it globally. I believe std calls all the right functions, but it may be worth verifying what various crates use. The allowed functions:
|
Indeed. My subroutine that does it is there, and it handles all the cases you mention plus a few others. (And conversely, if you know about cases I forgot, please tell me!)
This is a workaround for the technical users who know how to enable this option in the registry. For all others who are running Windows 10 with default settings, or Windows 7 or 8 anyway, the \\?\ method is the only one that will work. |
@chrmarti @roblourens Do either of you know how to enable long path name support on a per-application basis? Is there something I can do to the ripgrep release artifact that will fix it? |
@retep998 I do not believe the app manifest is capable of enabling opt-in long path support. I interpreted the docs the same way, but try as I might, it didn't work. It seems it can only be used to disable long path support per-app on a system that has opted in via the registry (or group policy). The approach Mesos and CMake took was to prepend It's probably only marginally useful as it's C++ not Rust (so like, this is what the Rust standard libraries should probably implement), but the implementation is here. |
@andschwa Interesting, thanks! How do you deal with relative paths? That is, what if ripgrep is already deep in a directory hierarchy and you run |
The dumb way: we use absolute paths "everywhere." I know, it's not a great answer, and it's not entirely true either. But when it breaks, we fix it to use an absolute path, since you can't use |
@andschwa Ah OK, great. And yeah no worries, honestly, I'm just trying to make sure I understand the problem and what people are doing to fix it. :-) That would definitely be a tough fix to make though. I was hoping for that application manifest trick to bail me out! |
I hoped for the same thing. It's possible I screwed up; but I had disabling ( |
@andschwa this may not be that helpful, but at least on Windows 10 it looks like you need both the (registry or GP) and the manifest setting. See "second gate" language in this blog post, which is contra the documentation. But, there is more real-world confirmation in a forum if you read far enough to this comment. @BurntSushi I haven't invested enough time to know if ripgrep builds for Windows even have the requisite manifest metadata, but it seems theoretically possible that it would be possible for Windows 10 users to enable long path support if ripgrep can include the manifest setting. |
@tedfordgif Unfortunately I'm not much help here. The Windows build is basically the bare minimum that I know how to do, and I'm pretty sure I just copied it from someone else. I know so little that I don't even know how to guide anyone to solve this problem. Whoever does it is going to need to get their hands dirty. :-) |
@BurntSushi, it looks like this probably belongs in rustc: librustc_trans/back/linker.rs. You need to pass /MANIFEST and /MANIFESTINPUT:manifest.xml to link.exe. The manifest.xml file should look something like this:
|
@tedfordgif Is there any simple way to hack that manifest in, even if it's just to test, without modifying rustc's internal linker? |
|
Glad to report that this is a very easy fix. Prerequisite: Set registry key Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem: LongPathsEnabled to 1. Then start a new powershell (it needs to be able to find the registry value you just set in order to create the test path).
There's probably a better way to set the rustflags in Cargo.toml, but I didn't discover the recipe just yet. Maybe this will help: https://users.rust-lang.org/t/solved-rust-project-how-build-like-gcc-mwindow/5168/8 |
@tedfordgif Oh wow, that's awesome!! It seems like I should be able to include a manifest in the release binaries at least. Needing to edit a registry key is still unfortunate, but perhaps good enough for a workaround for now for those that need it. |
@BurntSushi It isn't a workaround, it is the Right Way, at least for now, 'cuz Windows. |
@tedfordgif Anything that requires editing the registry is a work around IMO. There are ways to fix this without editing the registry, but require more pervasive changes to the code, e.g., by using the |
@tedfordgif You don't actually even need the manifest to enable long path support if you've enabled it in the registry. Enabling it in the registry enables it "globally," and the manifest is really only useful to opt-out (and disable it for an application running on a computer where it's been enabled). I'd be very happy to be disproved about this, but as I said earlier, it did not appear possible when I tried to enable long path support through the manifest on a per-app basis (that is, without enabling it in the registry or through GPO). But hey, the next version of the "unversioned" Windows 10 is coming out in a couple weeks, maybe things will improve! |
@andschwa, see my previous PS session that makes it pretty clear the manifest is a requirement. Also see the links to blog posts that clarify that you need both the registry and the manifest. Happy to make you very happy! Given that the official documentation seems to be saying you only need one of the two, I wouldn't be surprised if they actually make that the case in a Windows 10 update, but it is certainly not the case now. |
@BurntSushi I agree it is a workaround, didn't really mean to quibble. Another important consideration is that only some of the API supports the long path names, at least according to the documentation, which as we've seen is not always right. |
@tedfordgif I had to do further testing of this, and indeed, it is wishy-washy. Some APIs (like Fun thing to break: use Sorry, this isn't totally ripgrep related, but it sure was interesting. |
@andschwa Thanks for the follow-up and details. |
@BurntSushi I really hate this bug. Would you accept merge requests that implement the
|
Unlikely, because such a patch is non-trivial and has consequences for filtering. What I would like is a patch that fixes this via a manifest file. If that still requires a registry edit, then oh well. |
I really think filtering being potentially affected is a smaller problem than not being able to use ripgrep for long paths (and can be just a note on the help "instead of matching string start for file names, match on \?\ because windows is weird"). |
#2049 embeds the manifest file on windows. It still needs to be enabled in the registry. Maybe ripgrep could detect long paths on windows and print a warning if long paths are not enabled in the registry? |
I think this issue can be closed. I just tested ripgrep 13.0.0 with a path of length >750 with the following powershell code already presented in a comment above:
I did not modify ripgrep in any way, this is the normal Someone else should probably test it also on their machine, maybe I messed something up. |
Oh wow, I actually had no idea about that. Reading that PR, it does indeed look like this issue might be fixed. |
The problem appears to be fixed on my computer as well, at least when installing with However, the problem is NOT fixed when installing the latest prebuilt release (https://github.com/BurntSushi/ripgrep/releases/download/13.0.0/ripgrep-13.0.0-x86_64-pc-windows-msvc.zip). Presumably this is because that release was built with an older version of Rust. It might be a good idea to release a new prebuilt version that is built with a newer version of Rust. |
See the README and comments in the build.rs. Basically, this embeds an XML file that I guess is a way of setting configuration knobs on Windows. One of those knobs is enabling long path support. You still need to enable it in your registry (lol), but this will handle the other half of it. Fixes #364, Closes #2049
See the README and comments in the build.rs. Basically, this embeds an XML file that I guess is a way of setting configuration knobs on Windows. One of those knobs is enabling long path support. You still need to enable it in your registry (lol), but this will handle the other half of it. Fixes #364, Closes #2049
See the README and comments in the build.rs. Basically, this embeds an XML file that I guess is a way of setting configuration knobs on Windows. One of those knobs is enabling long path support. You still need to enable it in your registry (lol), but this will handle the other half of it. Fixes #364, Closes #2049
I'm still experiencing this issue with the latest Windows prebuilt release |
@nod5 This fix is not in a release yet. You should be able to compare the dates to arrive at that conclusion. I'll put a new release it when it's convenient for me to do so. See: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#release |
Seems long file names may not work in the Windows x64 build of ripgrep 0.4.0.
In cmd on windows 10 (Version 1607 OS Build 14393.693) do the following:
Execute rg:
rg "foobar"
Result:
./abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs: The system cannot find the path specified. (os error 3)
Try again:
Result:
\\?\c:\ar\abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs 1:"foobar"
It would be nice to have long file name support by default without having to add
\\?\%cd%
.The text was updated successfully, but these errors were encountered: