Support long paths v2 #122

kblees · 2014-02-08T18:51:26Z

Changes since v1 (#117):

based on @dscho's Revert #86 and #111 #114
add a test case for long repository paths (i.e. checkout things that really exceed MAX_PATH)
if len(cwd) + len(path) exceeds MAX_PATH, but the absolute path does not (e.g. due to eliminating '..'), use the absolute path.

The latter fixes @vangdfang's submodule test case: is_git_directorytries to access("../test/$longpath") from cwd "somedir/$longpath". All involved paths (cwd, the relative path and the final path on disk) are shorter than MAX_PATH, yet waccess/GetFileAttributesW fails with an error. So I guess the popular MSDN clause "the name is limited to MAX_PATH characters" must have some hidden meaning that I missed completely ;-)

Here's an installer with this patch included:
https://drive.google.com/file/d/0BxXUoUg2r8ftSVA3YmdLMjdJMkk/edit?usp=sharing

buildhive · 2014-02-08T19:49:09Z

MSysGit - the development behind Git for Windows » git #172 SUCCESS
This pull request looks good
(what's this?)

dscho · 2014-02-10T16:06:41Z

This looks neat! I will check it out later today and try to clean up #114.

However, I have a hunch that the test case added by @vangdfang might still have problems (in my tests, the problem was in non-Windows code that tried to clone into a path exceeding 256 characters...). Did you look at t7410?

Windows paths are typically limited to MAX_PATH = 260 characters, even though the underlying NTFS file system supports paths up to 32,767 chars. This limitation is also evident in Windows Explorer, cmd.exe and many other applications (including IDEs). Particularly annoying is that most Windows APIs return bogus error codes if a relative path only barely exceeds MAX_PATH in conjunction with the current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG. Many Windows wide char APIs support longer than MAX_PATH paths through the file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path. Notable exceptions include functions dealing with executables and the current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...). Introduce a handle_long_path function to check the length of a specified path properly (and fail with ENAMETOOLONG), and to optionally expand long paths using the '\\?\' file namespace prefix. Short paths will not be modified, so we don't need to worry about device names (NUL, CON, AUX). Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be limited to MAX_PATH (at least not on Win7), so we can use it to do the heavy lifting of the conversion (translate '/' to '\', eliminate '.' and '..', and make an absolute path). Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH limit. Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs that support long paths. While improved error checking is always active, long paths support must be explicitly enabled via 'core.longpaths' option. This is to prevent end users to shoot themselves in the foot by checking out files that Windows Explorer, cmd/bash or their favorite IDE cannot handle. Thanks-to: Martin W. Kirst <maki@bitkings.de> Thanks-to: Doug Kelly <dougk.ff7@gmail.com> Signed-off-by: Karsten Blees <blees@dcon.de>

kblees · 2014-02-11T20:16:02Z

@dscho Indeed. I've just force-pushed a fix for the submodule test that calculates $longpath based on the length of the test directory.

Playing around with the length, I found that we're hitting several core-git limits here. I agree with @kusma that these are separate issues:

>= 110: test 2 fails (in sha1_file_name):

fatal: insanely long object directory C:/git/msysgit/git/t/trash directory.t7410-submodule-long-path/home3/test/.git/modules/foo/modules/0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz01/objects

The "110" depends on the length of $(pwd).

I think its a sane requirement that the git object database should fit within PATH_MAX, even if we could handle longer paths in the working copy.

Workarounds: move the git-dir to some shorter path (GIT_DIR, --git-dir, gitfile), or use shorter submodule names (git submodule add --name shortname repo longpath).

>= 117: both tests fail (in is_git_directory / read_gitfile):

fatal: Too long path: 0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz012345678/../.git/modules/0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz012345678

Linking submodules via relative paths is by design, so that the super project can be moved without breaking the links. The error occurs because of string operations on a PATH_MAX-sized buffer, could be fixed by using strbuf instead.

This currently limits the submodule path to PATH_MAX / 2 minus a few bytes.

Workaround: use shorter submodule names

>= 160: both tests fail (test_cmp expected actual):

This time we hit the PATH_MAX-100 limit in git_path_submodule, which simply returns "/bad-path/". As nobody checks for this special value, all "git submodule" commands simply succeed without doing anything...

buildhive · 2014-02-11T20:49:10Z

MSysGit - the development behind Git for Windows » git #175 SUCCESS
This pull request looks good
(what's this?)

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

dscho · 2014-02-17T19:07:04Z

I actually merged it into 'master'... Thanks @kblees for your contribution and especially for the detailed analysis! Now everybody needing support for longer paths has a starting point to patch Git for Windows.

caralluma · 2014-03-05T19:05:07Z

Can someone provide me the Git preview version number which has this fix. I downloaded the latest Git-1.9.0-preview20140217.exe but the issue -Filename too long still exists with this. Is this fixed ?

error: unable to create file FunctionalTests/opalDomain/CheckOut/src/test/resources/FitNesseRoot/FunctionalTests/CheckOut/opalTests/ProcessOrderForCheckoutAsGuestUser/AdjustedQuantityScenarios/QualifiedToQuantityNormalItemUnavailableSplitItemAvailable/properties.xml (Filename too long)

kblees · 2014-03-05T19:22:25Z

@SunilPrabhu yes, its fixed in 1.9.0 (as you can see from the error message "Filename too long")

You have to explicitly enable long paths via 'git config core.longpaths true', but don't come back here and complain that it breaks Windows Explorer, cmd.exe, bash or whatever tools you're using.

caralluma · 2014-03-05T21:05:23Z

Thanks !

PapaMufflon · 2014-03-20T15:55:04Z

Surely I'm doing something wrong: I did git config core.longpaths true and git add . then git commit. Everything went well. But when I now do a git status, I get a list of files with "Filename too long", e.g.

node_modules/grunt-contrib-imagemin/node_modules/pngquant-bin/node_modules/bin-wrapper/node_modules/download/node_modules/request/node_modules/form-data/node_modules/combined-stream/node_modules/delayed-stream/test/integration/test-handle-source-errors.js: Filename too long

It is quite simple to reprodcue for me: just crate a yeoman web app with the angular-generator ("yo angular") and remove "node_modules" from the .gitignore file. Then repeating the aforementioned git commands.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

irandom · 2014-05-19T07:21:34Z

It seems that, for git-clean, this fix does not work:

D:\jenkins\workspace\ATM-hourly\SLAVE\build-win7-32>git --version
git version 1.9.2.msysgit.0

D:\jenkins\workspace\ATM-hourly\SLAVE\build-win7-32>git config core.longpaths true

D:\jenkins\workspace\ATM-hourly\SLAVE\build-win7-32>git config core.longpaths
true

D:\jenkins\workspace\ATM-hourly\SLAVE\build-win7-32>git clean -fdx
warning: failed to remove data/data/gui/html/common/jwe/node_modules/less/node_modules/request/node_modules/lodash.merge/node_modules/lodash._basecreatecallback/node_modules/lodash.bind/node_modules/lodash._createwrapper/node_modules
warning: failed to remove data/data/gui/html/common/jwe/node_modules/less/node_modules/request/node_modules/lodash.merge/node_modules/lodash._basemerge/node_modules/lodash.forown/node_modules/lodash.keys/node_modules/lodash._isnative
warning: failed to remove data/data/gui/html/common/jwe/node_modules/less/node_modules/request/node_modules/lodash.merge/node_modules/lodash._basemerge/node_modules/lodash.forown/node_modules/lodash.keys/node_modules/lodash._shimkeys

kblees · 2014-05-19T09:53:08Z

@irandom the following test works for me in t2025, so I suspect your problem is not long path related (eg. some process keeps the files open).

test_expect_success 'clean long untracked paths' '
    mkdir longpa~1/untrackeddir &&
    echo foobar > longpa~1/untrackeddir/untrackedfile &&
    git clean -fdx &&
    test_path_is_missing longpa~1/untrackeddir/untrackedfile &&
    test_path_is_missing longpa~1/untrackeddir
'

Windows paths are typically limited to MAX_PATH = 260 characters, even though the underlying NTFS file system supports paths up to 32,767 chars. This limitation is also evident in Windows Explorer, cmd.exe and many other applications (including IDEs). Particularly annoying is that most Windows APIs return bogus error codes if a relative path only barely exceeds MAX_PATH in conjunction with the current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG. Many Windows wide char APIs support longer than MAX_PATH paths through the file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path. Notable exceptions include functions dealing with executables and the current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...). Introduce a handle_long_path function to check the length of a specified path properly (and fail with ENAMETOOLONG), and to optionally expand long paths using the '\\?\' file namespace prefix. Short paths will not be modified, so we don't need to worry about device names (NUL, CON, AUX). Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be limited to MAX_PATH (at least not on Win7), so we can use it to do the heavy lifting of the conversion (translate '/' to '\', eliminate '.' and '..', and make an absolute path). Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH limit. Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs that support long paths. While improved error checking is always active, long paths support must be explicitly enabled via 'core.longpaths' option. This is to prevent end users to shoot themselves in the foot by checking out files that Windows Explorer, cmd/bash or their favorite IDE cannot handle. Test suite: Test the case is when the full pathname length of a dir is close to 260 (MAX_PATH). Bug report and an original reproducer by Andrey Rogozhnikov: msysgit#122 (comment) Note that the test cannot rely on the presence of short names, as they are not enabled by default except on the system drive. [jes: adjusted test number to avoid conflicts, reinstated && chain, adjusted test to work without short names] Thanks-to: Martin W. Kirst <maki@bitkings.de> Thanks-to: Doug Kelly <dougk.ff7@gmail.com> Signed-off-by: Karsten Blees <blees@dcon.de> Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com> Signed-off-by: Stepan Kasal <kasal@ucw.cz> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Windows paths are typically limited to MAX_PATH = 260 characters, even though the underlying NTFS file system supports paths up to 32,767 chars. This limitation is also evident in Windows Explorer, cmd.exe and many other applications (including IDEs). Particularly annoying is that most Windows APIs return bogus error codes if a relative path only barely exceeds MAX_PATH in conjunction with the current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG. Many Windows wide char APIs support longer than MAX_PATH paths through the file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path. Notable exceptions include functions dealing with executables and the current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...). Introduce a handle_long_path function to check the length of a specified path properly (and fail with ENAMETOOLONG), and to optionally expand long paths using the '\\?\' file namespace prefix. Short paths will not be modified, so we don't need to worry about device names (NUL, CON, AUX). Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be limited to MAX_PATH (at least not on Win7), so we can use it to do the heavy lifting of the conversion (translate '/' to '\', eliminate '.' and '..', and make an absolute path). Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH limit. Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs that support long paths. While improved error checking is always active, long paths support must be explicitly enabled via 'core.longpaths' option. This is to prevent end users to shoot themselves in the foot by checking out files that Windows Explorer, cmd/bash or their favorite IDE cannot handle. Test suite: Test the case is when the full pathname length of a dir is close to 260 (MAX_PATH). Bug report and an original reproducer by Andrey Rogozhnikov: msysgit#122 (comment) [jes: adjusted test number to avoid conflicts] Thanks-to: Martin W. Kirst <maki@bitkings.de> Thanks-to: Doug Kelly <dougk.ff7@gmail.com> Signed-off-by: Karsten Blees <blees@dcon.de> Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com> Signed-off-by: Stepan Kasal <kasal@ucw.cz> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>