Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Windows Longpath support does not fully work? Error: "fatal: '$GIT_DIR' too big" #3372

Closed
skdsp opened this issue Aug 18, 2021 · 7 comments
Closed

Windows Longpath support does not fully work? Error: "fatal: '$GIT_DIR' too big" #3372

skdsp opened this issue Aug 18, 2021 · 7 comments
Labels
up for grabs

Comments

@skdsp
Copy link

skdsp commented Aug 18, 2021 •
edited
Loading

  • [ x] I was not able to find an open or closed issue matching what I'm seeing

Setup

  • Which version of Git for Windows are you using? Is it 32-bit or 64-bit?
$ git --version --build-options

git version 2.31.1.windows.1
cpu: x86_64
built from commit: c5f0be26a7e3846e3b6268d1c6c4800d838c6bbb
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

 - Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.18363.1679]

  • What options did you set as part of the installation? Or did you choose the
    defaults?
# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

** insert your machine's response here **
Editor Option: Notepad++
Custom Editor Path: 
Default Branch Option:  
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: WinSSL
CRLF Option: CRLFCommitAsIs
Bash Terminal Option: ConHost
Git Pull Behavior Option: Merge
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled


 - Any other interesting things about your environment that might be related
   to the issue you're seeing?

Longpaths activated (C:\Program Files\Git\etc\gitconfig): 	
[core] 
  longpaths = true 

### Details

 - Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

git bash (MINGW64)

 - What commands did you run to trigger this issue? If you can provide a
   [Minimal, Complete, and Verifiable example](http://stackoverflow.com/help/mcve)
   this will help us understand the issue.

** insert your commands here **

  1. Cloning a Git Repo into a folder with an already long path
    git clone -c core.longpaths=true https://github.com/git/git.git --recurse-submodules //?/D:/eval/Git_Test/looooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooong
  2. Cloning a Git Repo with multiple levels of submodules (nested submodules). Here we are really running into issues with levels of e.g. 6 nested submodules, when trying to clone into already very short pathes. Shortening paths within the submodules is only solving temporally the issue. I'd like to have something that is really future proof.
    git clone -c core.longpaths=true --recurse-submodules //?/
  • What did you expect to occur after running these commands?

All Repos with submodules are fetched recursively.

  • What actually happened instead?

Error message: fatal: '$GIT_DIR' too big.
The error seems to point to a define (PATH_MAX) in setup.c code (https://github.com/git/git/blob/54e85e7af1ac9e9a92888060d6811ae767fea1bc/setup.c#L812)
Thus I am afraid the PATH_MAX is set to 260 for the installation already?

  • If the problem was occurring with a specific repository, can you provide the
    URL to that repository to help us with testing?

e.g. https://github.com/git/git.git
@dscho
Copy link
Member

dscho commented Aug 19, 2021

Cloning a Git Repo into a folder with an already long path

That sounds like a catch-22: now, when Git wants to read the config, it has to open a long path. But it does not yet know whether it is supposed to support long paths. So it doesn't (which is the default). And therefore it can never read the config that tells it that it should turn on the long path support.

@skdsp
Copy link
Author

skdsp commented Aug 19, 2021

Hmm, I think this is a general issue with paths > 260. Or am I doing something wrong?

In the first example, the path is 230 chars long. Regardless of the passed parameters to the clone command, it is not possible to clone the repo:

  1. git clone -c core.longpaths=true https://github.com/git/git.git --recurse-submodules D:/eval/Git_Test/looooooooooooooo
    oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooong
    Cloning into 'D:/eval/Git_Test/looooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooong'...
    fatal: '$GIT_DIR' too big
  2. git clone https://github.com/git/git.git --recurse-submodules D:/eval/Git_Test/loooooooooooooooooooooooooooooooooooooo
    oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
    ooooooooooooooooooooooooooooooooooooooooooooooooooong
    Cloning into 'D:/eval/Git_Test/looooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooong'...
    fatal: '$GIT_DIR' too big

Hint: The longpath setting is also set in my system git core settings.

I think the define PATH_MAX is set for windows by limit.h to 260 and thus the longpath feature is prohibited
(https://github.com/git/git/blob/54e85e7af1ac9e9a92888060d6811ae767fea1bc/git-compat-util.h#L320-L325).

As mentioned, the first example is something to show the general issue, the second example (nested submodules) is the real issue at our side.

@skdsp
Copy link
Author

skdsp commented Aug 19, 2021 •
edited
Loading

Okay, maybe now I got your point. I have removed some of the chars in the path of the first example and the clone is fine.

I thought by setting the system git core settings, the default behavior for longpaths should have changed? How can I change the behavior otherwise?

Anyhow, I still don't get how this is connected to the mentioned error message and the define? Is the function "setup_explicit_git_dir" not called when longpath is activated?

@skdsp
Copy link
Author

skdsp commented Aug 19, 2021

Maybe my issue is a duplication of: #2075
Yet it seems there was no solution?

@dscho
Copy link
Member

dscho commented Aug 24, 2021

The solution is probably a bit more involved than a casual contributor is prepared for.

Currently, the way things are handled look like this: 5eb8d3a

The strategy chosen is to have a platform-specific mingw_core_config() function that interprets the value of core.longPaths, and then to make sure that each and every command reads the config.

Before the config is read, the core.longPaths setting is not respected, and that includes the time when we actually try to read the config (or for that matter, to detect whether we are in a Git worktree or repository).

The way things should be done is via a function similar to git_config_get_bool().

I say similar to because we cannot use it. You see, if you call git_config_get_bool() when trying to determine whether to use long path support or not, you might be in the call that tries to determine whether we're inside a Git worktree or not, and if called at that stage, it will read the config without heeding any worktree config (and cache the results!).

So what we would need is something akin to git_config_get_bool(), but a variant that emulates the read_very_early_config() function in that it does read the system and global config, and does not cache anything.

However, calling such a function would be wasteful because Git already calls read_very_early_config() as part of tr2_sysenv_load(), which is called from trace2_initialize_fl(), which in turn is called from main().

So: the most appropriate solution to this would be to:

  • Refactor the part of trace2_initialize() that reads the config,
  • Probably by making tr2_sysenv_cb() a public function,
  • No longer calling it from trace2_initialize(), but from a static callback function in common-main.c,
  • Calling read_very_early_config() explicitly in main(), with that static callback function that calls into tr2_sysenv_cb().
  • Then extend that static callback function to allow for platform-specific callback functions,
  • Implement such a thing for Windows, and parse core.longPaths in that function.

One particular complication with this entire plan is that read_very_early_config() ignores the command-line config passed via git -c <key>=<value> ....

As you probably suspect by now, this requires a lot more care and effort than you might originally have thought.

Having said that, if you are interested, I can help you implement it.

@skdsp
Copy link
Author

skdsp commented Aug 30, 2021

Thank you for your reply.
I am sorry to say that I won't have the time for implementing this.

kevin-worm added a commit to kevin-worm/git that referenced this issue May 30, 2022
@kevin-worm
Fix error: "fatal: '$GIT_DIR' too big" when checking out in a long pa…
1168301 
…th on

Windows with core.longpaths = true.

This is a fix for issue: git-for-windows#3372

Use case:
Allow git to checkout a repository or submodule in a directory with a long
path when core.longpaths = true.

Example:
> ./git.exe config --global core.longpaths true
> ./git.exe clone https://github.com/git/git.git --recurse-submodules \
/c/eval/git_test/loooooooooooooooooooooooooooooooooooooooooooooooooooooooo\
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo\
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo\
oooooooong

Context:
$ sh --version
	GNU bash, version 4.4.23(1)-release (x86_64-pc-msys)
$ ./git.exe --version --build-options
	git version 2.36.1.windows.1
	cpu: x86_64
	built from commit: e2ff68a
	sizeof-long: 4
	sizeof-size_t: 8
	shell-path: /bin/sh
	feature: fsmonitor--daemon

Error:
fatal: '$GIT_DIR' too big.

Problem analysis:
setup_explicit_git_dir in setup.c uses PATH_MAX to check if the git dir is
to long. On Windows PATH_MAX is set by limit.h to 260 and
setup_explicit_git_dir ignores core.longpaths.

Solution:
The implementation is based on the solution proposed by Johannes
Schindelin, see:
git-for-windows#3372 (comment)

* Refactor the part of trace2_initialize() that reads the config.
* Make tr2_sysenv_cb() a public function.
* No longer calling it from trace2_initialize(), but from a static callback
  function in common-main.c.
* Calling read_very_early_config() explicitly in main(), with that static
  callback function that calls into tr2_sysenv_cb().
* Extend the static callback function for Windows, and parse core.longPaths
  in that function.
* Extend startup_info struct in cache.h with 'int max_long_path' so we can
  use it in setup_explicit_git_dir instead of PATH_MAX.

Signed-off-by: Kevin Worm <kevin.worm_1@signify.com>
kevin-worm added a commit to kevin-worm/git that referenced this issue May 30, 2022
@kevin-worm
Fix error: "fatal: '$GIT_DIR' too big" when checking out in a long pa…
40ff106 
…th on

Windows with core.longpaths = true.

This is a fix for issue: git-for-windows#3372

Use case:
Allow git to checkout a repository or submodule in a directory with a long
path when core.longpaths = true.

Example:
> ./git.exe config --global core.longpaths true
> ./git.exe clone https://github.com/git/git.git --recurse-submodules \
/c/eval/git_test/loooooooooooooooooooooooooooooooooooooooooooooooooooooooo\
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo\
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo\
oooooooong

Context:
$ sh --version
	GNU bash, version 4.4.23(1)-release (x86_64-pc-msys)
$ ./git.exe --version --build-options
	git version 2.36.1.windows.1
	cpu: x86_64
	built from commit: e2ff68a
	sizeof-long: 4
	sizeof-size_t: 8
	shell-path: /bin/sh
	feature: fsmonitor--daemon

Error:
fatal: '$GIT_DIR' too big.

Problem analysis:
setup_explicit_git_dir in setup.c uses PATH_MAX to check if the git dir is
to long. On Windows PATH_MAX is set by limit.h to 260 and
setup_explicit_git_dir ignores core.longpaths.

Solution:
The implementation is based on the solution proposed by Johannes
Schindelin, see:
git-for-windows#3372 (comment)

* Refactor the part of trace2_initialize() that reads the config.
* Make tr2_sysenv_cb() a public function.
* No longer calling it from trace2_initialize(), but from a static callback
  function in common-main.c.
* Calling read_very_early_config() explicitly in main(), with that static
  callback function that calls into tr2_sysenv_cb().
* Extend the static callback function for Windows, and parse core.longPaths
  in that function.
* Extend startup_info struct in cache.h with 'int max_long_path' so we can
  use it in setup_explicit_git_dir instead of PATH_MAX.

Signed-off-by: Kevin Worm <kevin.worm_1@signify.com>
@kevin-worm kevin-worm mentioned this issue May 30, 2022
Open
dscho pushed a commit to kevin-worm/git that referenced this issue May 30, 2022
@kevin-worm @dscho
mingw: respect core.longPaths when initializing a new gitdir
7a3a4c7 
Use case:
Allow git to checkout a repository or submodule in a directory with a long
path when core.longpaths = true.

Example:
> ./git.exe config --global core.longpaths true
> ./git.exe clone https://github.com/git/git.git --recurse-submodules \
/c/eval/git_test/loooooooooooooooooooooooooooooooooooooooooooooooooooooooo\
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo\
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo\
oooooooong

Context:
$ sh --version
	GNU bash, version 4.4.23(1)-release (x86_64-pc-msys)
$ ./git.exe --version --build-options
	git version 2.36.1.windows.1
	cpu: x86_64
	built from commit: e2ff68a
	sizeof-long: 4
	sizeof-size_t: 8
	shell-path: /bin/sh
	feature: fsmonitor--daemon

Error:
fatal: '$GIT_DIR' too big.

Problem analysis:
setup_explicit_git_dir in setup.c uses PATH_MAX to check if the git dir is
to long. On Windows PATH_MAX is set by limit.h to 260 and
setup_explicit_git_dir ignores core.longpaths.

Solution:
The implementation is based on the solution proposed by Johannes
Schindelin, see:
git-for-windows#3372 (comment)

* Refactor the part of trace2_initialize() that reads the config.
* Make tr2_sysenv_cb() a public function.
* No longer calling it from trace2_initialize(), but from a static callback
  function in common-main.c.
* Calling read_very_early_config() explicitly in main(), with that static
  callback function that calls into tr2_sysenv_cb().
* Extend the static callback function for Windows, and parse core.longPaths
  in that function.
* Extend startup_info struct in cache.h with 'int max_long_path' so we can
  use it in setup_explicit_git_dir instead of PATH_MAX.

This fixes git-for-windows#3372

Signed-off-by: Kevin Worm <kevin.worm_1@signify.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
@dscho
Copy link
Member

dscho commented Aug 29, 2023

Closing as stale.

@dscho dscho closed this as not planned Won't fix, can't repro, duplicate, stale Aug 29, 2023
@skdsp skdsp mentioned this issue Oct 8, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
up for grabs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dscho @skdsp