Various fixes

[Byron]

Tasks

• fix Nondeterministic macOS is_symlink assertion failure in overwriting_files_and_lone_directories_works #1373
• fix feature sha1/asm is enabled on windows #917
• update dependencies

Original Checklist from private PR

This PR lays down the infrastructure to further validate tree-names before they are used, and make sure validation happens in the right places.

That way, writes to forbidden locations inside or outside of the repository should be prevented.

Tasks

• update progress with the configuration flags to be implemented
• add validation to gix-validate
  • slashes in name
  • relative path components
  • .git and variants
  • HFS specific
  • NTFS specific
    • trailing whitespace gets truncated by Windows automatically
  • backslashes in name?
• a way to disallow .gitmodules as symlinks
• let worktree-delegate validate the trailing component with gix-validate
  • add more detailed mode information
• obtain validation-options from configuration and pass it through
• prevent index creation from invalid names - improve awareness
• add CON protection
• check .git\hooks\precommit
• fix Nondeterministic macOS is_symlink assertion failure in overwriting_files_and_lone_directories_works #1373
• fix feature sha1/asm is enabled on windows #917
• e2e-tests
• forbid invalid Windows names in refs (probably via gix-validate) for now (see this issue for details)

References

• https://lore.kernel.org/git/xmqqr21cqcn9.fsf@gitster-ct.c.googlers.com/T/#u
• https://github.com/git/git/blob/19fe900cfce8096b7645ec9611a0b981f6bbd154/read-cache.c#L971

[Byron]

@EliahKagan If you take a look at the CI failure on Windows you will see that the error there is due to a lock file that couldn't be obtained. I think this is akin to what Git does. The problem here is that it shouldn't even get to that point - it should fail when trying to read the reference, which I think it might actually do here but without issue. Unfortunately, I couldn't reproduce it in the VM, the test succeeds, but I will keep trying with added debug information.

[Byron]

Turns out I was blind! CI is actually reproducing the correct issue, and the problem was that the error has handled too late.

[Byron]

Alright, I think CI will pass now, and once done, a new gitoxide release will be created.

[EliahKagan]

When I run the tests locally on Windows, at 5f1c865, this still fails:

        FAIL [   0.846s] gix-archive::archive from_tree::basic_usage_zip

The same thing seems to be happening on CI for it, with these details.

[Byron]

Oh, that's a new failure probably due to Windows not really having symlinks. Maybe there is a way to support that better, but for now it's not to be expected to be a symlink on Windows.
It's interesting as in an older version of the zip crate, symlinks didn't work at all, so that's an improvement.

[EliahKagan]

LGTM.

I have installed on Windows from e955770 with cargo install --path . and repeated the manual tests of the refs-related features that had not been fully implemented yet when I had tested them yesterday. Those are working now.

Once this passes CI, I think it can be merged. It seems to me that the pull request description should eventually be updated with the other relevant checklist items, but that this could be done later (which might even be preferable).

[RivenSkaye]

Symlinks on Windows do exist, but currently they're locked behind one of two options:

1. Admin only to create and edit
2. Developer Options allow you to toggle making them free for all

The latter is also what you need to have a hassle-free symlinks=native-strict experience in MSYS2/cygwin envs. More info on this 2016 windows developer blog article. For most intents and purposes, they're not reliably present.

Edit:
Current win32 docs and the corresponding page for the function in the headers seem to indicate Developer Mode is now only required for UWP and makes no mention of perms anymore

[EliahKagan]

These comments about the semantics of joining Windows paths are mainly to identify areas of interest for forthcoming work.

I've posted them as review comments because that seemed like an easier way to identify context than posting a regular comment here or a separate discussion question.

Some potentially identify areas where the tests may be improved, but they do not represent worries about the correctness of the code under test that was introduced in this PR.

I have two main goals:

• In some cases, there are clarifications that can be made in the custom messages, which I plan to do soon in a small PR.

  Edit: I've opened #1528 and added links to the comments below to corresponding comments in that PR about the related changes.

• When working on this, I recall that we had discussed the possibility that there was at least one area where the behavior of the standard library should be improved, and you suggested that I might wish to contribute an improvement. I recall being interested in doing so, and also, I think, that you may have been willing to help review or otherwise provide guidance with that. But unfortunately I don't remember what that was!

  It may be possible to figure it out from comments we exported from the temporary private fork in which most of the work that made it into this PR occurred. But those exports, even taken together, were not quite complete, and I have not yet managed to figure out which behavior of the standard library we thought was wrong. I'm hoping you may recall that. These comments include details about behaviors that I am confident at least on re-examination are correct but that may seem wrong, as well as about things that I am not sure are correct.

I also hope these comments may be useful to refer back to later when working on other parts of the code where these concepts are relevant. So if you identify any technical mistakes in the comments themselves, please let me know.

[EliahKagan]

c:relative doesn't work like c:\relative, except when the current directory on the C: drive is the root. Sorry I missed this before. I'll propose a fix to the wording shortly. Note that the asserted behavior here is correct, both in the sense of being what the standard library does and in the sense that it is not a bug for the standard library to do this. It is only the custom message argument that should be changed.

See also #1528 (comment).

[EliahKagan]

This makes sense, because taking a path $P$ and concatenating a path $Q$ should ideally produce a path to the location specified by $Q$ when it is evaluated with $P$ as the current location, to the extent that this can be computed independently of filesystem contents and process environment.

If we are located at d:\ and we use C: as a path, the path C: refers to the current directory on the C: drive. But the only way to express that without resolving paths situationally is as C:.

The unintuitive aspect of this is that one might assume that taking an absolute path $P$ and joining a relative path $Q$ should produce an absolute path. Even if one is accustomed to Windows, it feels like resolving a relative path under an absolute path should be an absolute path. But this is not always what happens, due to the semantics of Windows drive letters.

See also #1528 (comment).

The weirder sub-case of this is when both the absolute path on the left and the relative path on the right have the same drive letter. Different libraries handle this differently.

---

Paths like C: or C:bar that have a drive letter but no root, and thus are relative to the drive-specific current directory on the drive they specify, are one of two major cases of non-UNC Windows paths that may feel absolute but are not. The other major case is a path that has a root but no drive letter, i.e. it starts with / or \ but not \\, and thus is relative to the current drive, but not relative to the current directory on that drive. (UNC paths are always absolute.)

Both these cases can lead to bugs in code written to assume more Unix-like semantics of the relative/absolute path distinction. The first case, where $Q$ has a drive letter but no root, can be surprising because the concatenation of $P$ and $Q$ can be relative, even when $P$ is absolute. In contrast, the second case, where $Q$ has a root but no drive letter, can be surprising because the concatenation of $P$ and $Q$ may be outside of $P$, even when $P$ is absolute and $Q$ is a relative path containing no .. component.

The other factor that makes this hard, at least for me, is that the terminology in this part of the official documentation is extremely strange, for example by characterizing paths like \file.txt as both relative and absolute, and by seeming to imply that a path with any .. component anywhere is relative regardless of how it starts. I think the more commonly used terminology is what this other article uses, even though that is (a Windows-specific) part of the .NET documentation, rather than being part of the Windows documentation. That is the terminology I have used here.

[EliahKagan]

The device-namespace-unc part of the custom message suggests that this may have been meant to start with p("\\\\.\\") rather than p("\\\\."). As currently written, I am not sure what the ideal behavior would be here, because I am not sure the trailing \ of the prefix \\?\ is strictly speaking functioning as a path separator. The Rust standard library does sometimes treat \\.\ and \\. differently as the left side of a path join, though both for the relative path C: and the absolute path C:\ on the right, the result is the same.

However, for the same reason as in the above comment, that C: is relative should not be a reason to expect anything other than this behavior. It's less clear what it means to be located at \\. or \\.\. But I don't think it should be an exception to the usual effect of C: on the right side.

Furthermore, although \\.\C: and \\.\C:\ are both valid paths, we cannot correctly form either one of them by concatenating \\. or \\.\ with C:, and the reason is because a path C: is a relative path:

• \\.\C: is in the device namespace and ends in C: with no path separator, so that C: is a device name. Therefore \\.\C: refers to the C: drive as a device; the C: in \\.\C: is not a reference to the current directory, root directory, or any file or directory on the C: drive, but instead to the C: drive itself. (\\.\C: is an absolute path, but not to a file or directory.)
• \\.\C:\, due to the trailing \, is the root directory on the C: drive, and thus generally equivalent to \\?\C:\ (though I don't know that all APIs that accept one will accept the other or that all APIs that accept both will necessarily treat them identically). But this is an absolute path to the root of the C: drive, which C: by itself does not designate because it is a relative path whose meaning depends on the current directory on the C: drive.

See also #1528 (comment).

[EliahKagan]

These are not raw strings, so this is indicating that / joined with \\localhost gives \localhost with one leading \, and I don't understand why this is what happens. It also does not appear to be what this meant to assert, since it characterizes the effect of the join as = win-absolute-unc, but a path that begins with only one \ is not a UNC path. So I don't know what's going on here.

See also #1528 (comment).

[Byron]

That's a mistake, as it indeed doesn't consider that "\…" is not a raw string.
I don't know what an alternative comment would be, but it definitely isn't = win-absolute-unc.

[Byron]

• I recall being interested in doing so, and also, I think, that you may have been willing to help review or otherwise provide guidance with that. But unfortunately I don't remember what that was!

I also have faint memory of that, but not the specific thing we thought could be fixed. Hopefully that will come back once these cases are improved and/or fixed up.

[RivenSkaye]

Considering recent changes in std in the same domain, could you have been considering implementing/adding something along the lines of std::path::absolute perhaps? Came across that when Av1an had a clash with stuff in other langs and proposed that as a fix for the issues with extended path length syntax caused by not all Windows APIs supporting the syntax

[EliahKagan]

Considering recent changes in std in the same domain, could you have been considering implementing/adding something along the lines of std::path::absolute perhaps?

Although that makes sense and would have been a good idea, I don't think so. If I remember correctly, I think some behavior of the standard library shown or commented on in those tests had seemed like it was wrong, at the time.

[EliahKagan]

#1374 (comment) @RivenSkaye

Edit: Current win32 docs and the corresponding page for the function in the headers seem to indicate Developer Mode is now only required for UWP and makes no mention of perms anymore

It looks to me like that is a bug in the Microsoft documentation.

Over time, but more recently than that article was edited to mention UWP (see below on the history), I've attempted to use a limited user account (without extra privileges added), as well as with a UAC-unelevated administrator account, to create symbolic links on a few Windows 10 and Windows 11 systems, including multiple versions (builds) of both those operating systems, as well as on one Windows Server 2022 system. All of this was with non-UWP executables. Besides being compiled as traditional rather than UWP or WinRT executables, the executables were fairly diverse, and included:

• cmd.exe (via its mklink builtin).
• MSYS2 ln.exe with MSYS=winsymlinks:nativestrict to cause symlink-creation operations via msys-2.0.dll to create real Windows symlinks (#1444).
• git.exe (Git for Windows) with the core.symlinks configuration option set to true (git.exe, unlike ln.exe, is a native Windows program).

In these tests, I have never been able to create symlinks without enabling Developer Mode, performing UAC elevation, or using an administrator account for which UAC was not enabled. These experiments were after the relevant edit to the above-linked CreateSymbolicLinkA article. That edit was made in MicrosoftDocs/sdk-api#1284 with the commit message:

Proposed change is based upon testing under MSIX on Windows 10 21H2. I do not know whether developer mode is still required for UWP, so that should be checked prior to accepting this change.

I am unsure if "under MSIX" is a reference to operations that take place during an MSIX installation process, or if this is referring to the behavior of all software installed via MSIX, or if this referring to software built as an MSIX-related project type in Visual Studio, or if "MSIX" here is meant to refer to all non-UWP (or all non-WinRT) software.

Although it seems unlikely, it could be that the above observations were due to the functions not being called with the necessary flags, or something peculiar about the builds (for example, some seem to have been built against msvcrt.dll, as happens in MINGW64 builds, even though the Microsoft recommendation for programs that are not part of the operating system is to build against a specific versioned MSVCRT or against the UCRT), or a difference in the behavior of CreateSymbolicLinkW and CreateSymbolicLinkA (more on that below). Furthermore, I did not keep detailed notes about exactly what builds of Windows itself the above observations were from, nor do I know with certainty that I didn't skip or miss something.

For those reasons, I have now also written and run a small test program in C that tests out both the CreateSymbolicLinkW and CreateSymbolicLinkA functions, each for both file and directory symlinks, making sure to pass SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE in all calls. I did this on a Windows 11 system for which WindowsBuildLabEx is 22621.1.amd64fre.ni_release.220506-1250 and compiled it with Visual Studio 2022 17.12.3 and version 10.0.22621.0 of the Windows SDK. I can probably manage to run it on other versions and builds of Windows if a difference is suspected. The code, and further details, can be seen in the repo. The results were the same: with Developer Mode turned off, I needed to elevate in order for the symlink creation attempts to succeed.

Although that shows that the documentation is at least partially wrong and incomplete, I regard the test using that program to be the most limited source of information I have about this, just because I have not built it against multiple SDK versions or run it on multiple Windows operating systems or versions of the same Windows operating system.

The other thing I have noticed, which I think is the most important of all, is that the change in the CreateSymbolicLinkA article mentioning UWP was never applied to the CreateSymbolicLinkW article, which in the "meaning" column for SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE still says only:

Specify this flag to allow creation of symbolic links when the process is not elevated. Developer Mode must first be enabled on the machine before this option will function.

But when the Windows API has pairs of "A" and "W" functions, the "A" function is an ANSI wrapper for the "W" function. An "A" function converts ANSI to Unicode for any strings referenced by input parameters, calls the corresponding "W" function, and then converts Unicode to ANSI for any strings referenced by output parameters or return values. (This is how they work on NT systems. In Windows 9x/Me with MSLU, it was the other way around, with "W" functions being Unicode wrappers for the "A" functions.)

So the API documentation does still say that Developer Mode is needed, in the documentation for the "W" version of the function. This is furthermore the primary version. Thus a revision only to the documentation of the "A" version seems unintentional in and of itself.

(Something I have noticed is that, when I search using the learn.microsoft.com interface--and sometimes even with general purpose search engines--I get the "A" versions of functions first. This seems to be an unfortunate result of the Microsoft website disambiguating by picking the name that sorts earlier alphabetically--and then search engines ranking partly based on links that go to the "A" versions that were found that way. When one of the unsuffixed macros, such as CreateSymbolicLink, is used, these days it expands to the "W" version--here, CreateSymbolicLinkW--in the most common configurations, i.e., "Unicode" builds. But the version of the documentation most likely to be consulted seems often to be the "A" version.)

[RivenSkaye]

That is some A+ effort there. Good to know the docs and reality are out of sync on that front. I hope they'll be widely supported in the future. Perhaps a compile-time check or feature flag could be implemented at some point, but as-is it's not a safe default. Thanks for the poke!