Expose reparse point tags

[iSazonov]

On Windows if we enumerating file system we should take into account reparse points and reparse point tags too - that is symbolic links, OneDrive and AppX.
For example, attempting to read OneDrive reparse point causes an unwanted file download from the Internet. AppX reparse point would be considered like symbolic link. (It seems the last does not taken into account in Directory.Delete())

FileSystemEntry exposes FileAttributes there we can check reparse point flag. But we don't have reparse point tags.
FileSystem also exposes only FileAttributes without reparse point tags.

The request comes from PowerShell Core repo. I am trying to utilize new FileSystemEnumerable and have to use extra P/Invoke to get reparse point tags.
It will be great to have reparse point tags in FileSystemEntry and FileSystem types or another way to work with the scenarios in a Core standard way.

/cc @JeremyKuhne

[JeremyKuhne]

I agree that it would be nice to have this, the trick is that we haven't fully rationalized how we want to expose platform specific data. As this impacts OneDrive scenarios as mentioned we should try and answer this in 5.0 if we can.

[iSazonov]

@JeremyKuhne PowerShell 7.0 was released yesterday and I hope in next PowerShell milestone we will address OneDrive experience - this helps to understand how .Net Core could be enhanced.

[hamarb123]

@carlossanlop I've implemented this myself in my own project for symlinks and directory junctions on Windows. I'd be happy to help with any of the following code relating to reparse points (not sure which other issues this is related to).
I have code for:

• Symlink target getting on Mac using /usr/bin/readlink (probably not the preferred method)
• Symlink target writing on Mac using /bin/ln (probably not the preferred method)
• Determining which type of reparse point a file is (except hard links) ie. None, Symlink file, Symlink Directory, Directory Junction
• Creating symbolic links on Windows using Kernel32 methods
• Getting symlink target on Windows using Kernel32 methods
• Creating directory junctions on Windows using Kernel32 methods
• Checking if a directory is a junction on Windows using Kernel32 methods
• Getting a directory junction's target on Windows using Kernel32 methods

None of my code checks hard links properly as I've not looked into them.
Edit: Looks like readlink and ln (using symlink) can be done with libc calls.

[carlossanlop]

The conversation was growing in #24271, so I requested to move it here.

There's interest in having a boolean IsLink property in FileSystemInfo to let the user now if the current file is a link. I think the information would be insufficient, because there are multiple types of links.

@jozkee and I would prefer to have an enum that lists all the reparse tags. We initially described it in the second proposal, but we decided not to propose it in the API review meeting because there wasn't enough consensus.

+   public enum ReparseTag
+   {
+       None = -1,                // Default, indicates the file is not a Reparse Point (zero is already reserved)
+       MountPoint = 0xA0000003,  // Maps to IO_REPARSE_TAG_MOUNT_POINT, describes a Junction
+       Symlink = 0xA000000C,     // Maps to IO_REPARSE_TAG_SYMLINK, the only value used by Unix
+       AppExecLink = 0x8000001B, // Maps to IO_REPARSE_TAG_APPEXECLINK
+   }

    public abstract partial class FileSystemInfo
    {
+       public ReparseTag ReparseTag { get; }
    }

We now have more information that confirms this proposal makes more sense:

• Some filesystems (those used in Unix) only have one type of link: symbolic link.
• NTFS has multiple types of links. One of them is symbolic link, but there are also junctions, AppExecLink, and others, which are known as Reparse Tags (spec).
• Filesystems are independent of the OS, meaning that an NTFS drive can be properly supported in Linux if the proper driver is installed (for example, ntfs-3g supports reparse tags).
• We need to try to keep in mind future expansions, and an enum would be a great way to ensure we clearly describe the type of link the user is handling, and can be expanded.
• We don't need to initially add all the reparse tag values to the enum, but the enum values can represent the unique hex values of the reparse tags. This way, if the user encounters an enum currently not listed (like IO_REPARSE_TAG_IIS_CACHE), and queries the ReparseTag property, they would get the value 0xA0000010, which is outside the enum range, but it's a valid value.
• The name ReparseTag is not really Windows-specific, it's NTFS-specific, but would be consistent with the FileAttributes.ReparsePoint value we already have, which represents a symbolic link in Unix.

@hamarb123 thank you for your offer to help. Sorry I didn't reply earlier. Let's first discuss what the proper course of action is, and then we can decide how we will implement it.

[hamarb123]

That's fine. I'd also like to have the enum, since that's what my codebase would require.

[mklement0]

There's interest in having a boolean IsLink property in FileSystemInfo to let the user now if the current file is a link. I think the information would be insufficient, because there are multiple types of links.

It is indeed insufficient for knowing what type of link, but it is important enough in its own right to know that something is file-system link, i.e. that it points to another path (which may itself be a link, though that's not typical).

It is important if you want to inspect or modify the (ultimate) target item's properties, primarily, but not limited to, the ultimate name and directory location.
(A link's own properties (surfaced via FileSystemInfo representations), such as permissions, timestamps are usually irrelevant.)

In most cases, you do not care about the specific type of link:

• You only care about what item the link points to.
• and you may need to understand that .Delete() deletes only the link, not its target, whereas .Create() truncates the target (if it exists).

The ReparseTag property being proposed here does not provide the desired is-this-a-link abstraction, because not every reparse point is a link.

Even with the addition of a NameSurrogateBit enum value, as proposed by @iSazonov in #52666 (comment), you'll have to infer whether something is a link rather than having a direct API expression of it:

• info.ReparseTag & ReparsePointTag.NameSurrogateBit - awkward, not least for its NTFS-centeredness.
• info.LinkTarget is not null - better, but needlessly determines the target path.

Reparse points are one file system's (NTFS') implementation detail - it just so happens that the abstract concept of file-system links is implemented via them in NTFS, but they serve other, non-link purposes, and are generally thought of as opaque data to be interpreted only by the applications that define them and associated application-specific file-system filters - see the docs.

In other words: A ReparseTag property may be called for to support NTFS, but we also need an abstract, platform-neutral IsLink property.

---

Further thoughts on reparse points:

It strikes me as unlikely that additional general-purpose file-system links - as opposed to application-specific reparse points - will be introduced in the future: symbolic links on NTFS, a later addition, appear to the more flexible successors to the legacy link types called junctions / mount points. In other words: going forward, symbolic links cover all general-purpose file-system link needs, just as they do on Unix.

The generally opaque nature of reparse data also means that the .NET APIs need to have specialized knowledge of each reparse-point type in order to interpret that data - and I don't think we want to play continuous catch-up with respect to adding support for new, application-specific reparse tags - even if they're tagged as name surrogates, i.e. links (which OneDrive reparse points are not, by the way).

Instead, if general support for reparse points is really needed, the best option would be to expose the raw reparse-point data, so that any caller can interpret it itself.

As for link detection:

The name-surrogate bit being set unequivocally identifies a reparse points as a link, but in order to support LinkTarget and ResolveLinkTarget(), .NET needs to understand the given reparse-point type's specific data format (which is what PowerShell already does for symbolic links, junctions / mount point, and AppX reparse points).

AppX reparse points are an interesting hybrid case in that they do point to another (file) path, but there's more to their definition, namely a package family name and publisher ID - and in some cases (e.g. MicrosoftEdge.exe) this information is needed to actually launch the desired target application. However, from a file-system-link API perspective, reporting the target path alone is arguably enough (again, it is what PowerShell already does).

One challenge is that the abstract ability to detect an NTFS link - via the name-surrogate bit - will not necessary mean that inspecting / resolving the target will be supported (because it requires open-ended, type-specific data-format knowledge).

[tmds]

It is indeed insufficient for knowing what type of link, but it is important enough in its own right to know that something is file-system link

+1

In most cases, you do not care about the specific type of link:

+1

The name-surrogate bit being set unequivocally identifies a reparse points as a link, but in order to support LinkTarget and ResolveLinkTarget(), .NET needs to understand the given reparse-point type's specific data format

Yes, when you determine LinkTarget, in the process you've determined the type too.
So if LinkTarget is a property per #24271, then it makes sense to for the api of the type to be a property too.

[iSazonov]

Interesting, if underlying Windows API (which .Net uses) works with RPs as expected (in operations like create, delete, move, enumerate) then it is possible that we do not need an explicit API for RPs at all.

[hamarb123]

After reading @mklement0 's comment, I think I'd agree that there should be some form of an IsLink property as well (also note that the NameSurrogateBit & None would be not zero in the case of #24271 (comment)).

Instead, if general support for reparse points is really needed, the best option would be to expose the raw reparse-point data, so that any caller can interpret it itself.

I would very much like this as I personally thought that all reparse points just had the data as the link's target, and so I'd like the ability to just copy the link data from one file to another (unless I recognise the link type), as not only would this be faster, but would work in almost all edge cases as well.

[iSazonov]

the best option would be to expose the raw reparse-point data

What do you mean under raw data? Every RP is identified by unique mask. It is open list. However, we can simply expose this field as a property. But to interpret the RP (ex., get a target) we need more data - every RP has a specific struct with the data. It is not possible to publish this information without describing these structures in code beforehand, they may be undocumented.

[carlossanlop]

Seems most opinions are now leaning towards having a bool IsLink property that tells the user if the file is a symlink, a junction or an AppExecLink. This API is opaque in purpose: it limits the user from knowing the specific kind of link they're dealing with, but because it doesn't seem to be relevant to anyone.

We want to make sure this API is useful enough for anyone asking for it here (particularly our PowerShell contributors). Is anyone against this request, or can I proceed to prepare the API proposal? (I'll open a separate issue).

One more question - @mklement0 you mentioned the following:

The ReparseTag property being proposed here does not provide the desired is-this-a-link abstraction, because not every reparse point is a link.

What about OneDrive files? (the spec defines a bunch of IO_REPARSE_TAG_CLOUD_* types) In this issue's description, @iSazonov explicitly calls out these reparse points because:

attempting to read OneDrive reparse point causes an unwanted file download from the Internet

[hamarb123]

@carlossanlop I would also definitely like to be able to tell which type of link it is as well. For my code I'd very much like to be able to know what it is. The reason that I think we should also have the bool IsLink property is because it is confusing to be able to tell if it's actually a link to another file on the computer or something else (eg. normal file, onedrive etc.) without it.

Without the bool IsLink property, you'd need to have something like info.ReparseTag != ReparseTags.None && (info.ReparseTag & ReparsePointTag.NameSurrogateBit != 0)

[hamarb123]

the best option would be to expose the raw reparse-point data

What do you mean under raw data? Every RP is identified by unique mask. It is open list. However, we can simply expose this field as a property. But to interpret the RP (ex., get a target) we need more data - every RP has a specific struct with the data. It is not possible to publish this information without describing these structures in code beforehand, they may be undocumented.

@iSazonov I was under the impression that it could simply return a byte[] of the data.

[carlossanlop]

Let's keep this issue open to continue discussing how to expose the reparse tags specifically.

I opened a separate API proposal for IsLink: #53577

[hamarb123]

+   public enum ReparseTag
+   {
+       None = -1,                // Default, indicates the file is not a Reparse Point (zero is already reserved)
+       MountPoint = 0xA0000003,  // Maps to IO_REPARSE_TAG_MOUNT_POINT, describes a Junction
+       Symlink = 0xA000000C,     // Maps to IO_REPARSE_TAG_SYMLINK, the only value used by Unix
+       AppExecLink = 0x8000001B, // Maps to IO_REPARSE_TAG_APPEXECLINK
+   }

    public abstract partial class FileSystemInfo
    {
+       public ReparseTag ReparseTag { get; }
    }

@carlossanlop, would FileSystemInfo.ReparseTag have a value for a OneDrive file even though it is not in the enum, or would the runtime just set it to None, and if so, why should it have to follow the same structure as NTFS (link)?

[carlossanlop]

@hamarb123 thanks for continuing the discussion about reparse tags.

To answer your question: the property would return the value of the reparse tag (defined in the link you shared), even if that value is not yet included in the enum.

@mklement0 raised some interesting points, and made me wonder: how useful would it be to a user to learn if a file has a particular reparse tag, like IO_REPARSE_TAG_CLOUD_*, for example? What would the user do with that information?

[hamarb123]

@carlossanlop I think that it would just allow anyone who knows what's meant to be there to be able to see if it's that or not. Also, would this proposal also allow a property to allow raw access to the reparse point data? (presumably as a byte[] or something similar).

[iSazonov]

how useful would it be to a user to learn if a file has a particular reparse tag, like IO_REPARSE_TAG_CLOUD_*, for example? What would the user do with that information?

1. All we need is correct and expected file operations.
   For example, for symbolic link we expect Delete() method removes only the link entity itself, for OneDrive we expect removal of a content of the OneDrive.

2. What is file operations we say about? Create, Read, Update, Delete and Enumerate.
   2.1 Create
   We should use specific struct and Windows API call for every RPT (reparse point tag) kind.
   Thus, PowerShell can create (other than files and directories) symbolic links, mount points (hard links too) in different code paths.
   Since we have to use specific structs and RPT list is open we can not create generic method, we can only create a method for predefined like CreateSymbolicLink, CreateMountPoint, CreateHardLink. Of cause we could introduce single method using Enum parameter like Create(ReparsePointTag reparsePointTag) and throw on unsupported values.
   2.2 Read/Update (meta data like FileInfo, not file content)
   Really here is two operations - read/update data (1) about entity itself and (2) about target.
   Currently .Net implements an hybrid behavior in FileInfo/DirectoryInfo. It works well for most of scenarios. However, if we want more, we need an API for operations on the entities themselves. It is clear that we can only operate with general properties and attributes. I don't know if there is generic Windows API that can do this for any RPTs. Probably not and we will have to create a specific API for each kind of RTPs again.
   2.3 Delete
   Again two operations is here - regular Delete and recursive Delete.
   2.3.1 Regular Delete always removes entity itself, not target. It works well for most of scenarios. Otherwise, if we want remove the target we should resolve the target and then remove it. So we can say about ResolveTarget() method or DeleteTarget() method. In specific SymbolicLink class the methods would look well but in FileSystemInfo class they is under big question as it is unknown whether it is a link or not. So we seem to need a preliminary call to the IsLink() method. But as you can see this makes the user code more complicated and opens up opportunities for misuse.
   2.3.2 Recursive Delete. The behavior depends on the kind of RPT. So .Net excludes recursion for name surrogates
   

   runtime/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Windows.cs

   Lines 179 to 200 in 44f050a

   	public static void RemoveDirectory(string fullPath, bool recursive)
   	{
   	if (!recursive)
   	{
   	RemoveDirectoryInternal(fullPath, topLevel: true);
   	return;
   	}
   	Interop.Kernel32.WIN32_FIND_DATA findData = default;
   	GetFindData(fullPath, ref findData);
   	if (IsNameSurrogateReparsePoint(ref findData))
   	{
   	// Don't recurse
   	RemoveDirectoryInternal(fullPath, topLevel: true);
   	return;
   	}
   	// We want extended syntax so we can delete "extended" subdirectories and files
   	// (most notably ones with trailing whitespace or periods)
   	fullPath = PathInternal.EnsureExtendedPrefix(fullPath);
   	RemoveDirectoryRecursive(fullPath, ref findData, topLevel: true);
   	}

   
   PowerShell does the same with IsReparsePointLikeSymlink() (but take into account APPEXECLINK too).
   Also PowerShell can not directly use the .Net method and does manual recurse removal because PowerShell does some magic things (like user-friendly error messages since it is shell). As result probably PowerShell needs .Net method like IsReparsePointLikeSymlink() to implement manual recurse removal. Although perhaps we could use .Net file enumeration API if it will be enhanced with RPT support (see Allow opting out of following directory links (reparse points / symlinks) during recursive file-system enumerations #52666 (comment))
   Again it can be a problem to get the method generic since RPT list is open. Although we could probably make it configurable through an additional property.
   2.4 Enumerate
   .Net does not consider RPTs and always recurse. Proposal to enhance the API Allow opting out of following directory links (reparse points / symlinks) during recursive file-system enumerations #52666 (comment) - if this will be implemented we could use this in recurse Delete too.
   PowerShell can benefit from this greatly. Draft PR.
   2.5 Get/Resolve target
   PowerShell uses this in two places.
   2.5.1 In Get-ChildItem (aka dir)

dir C:\temp\Test

    Directory: C:\temp\test

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----           4/14/2021 10:17 AM                dir
l----            6/2/2021  4:32 PM                dirLink -> C:\temp\test\dir
d----           4/21/2021  9:17 AM                qq

It is PowerShell NameString() method which uses GetTarget() method.
As you can see it uses a pre-check IsReparsePointLikeSymlink() (but we can get rid of it here).
2.5.2 In IsWindowsApplication() method
We use GetTarget() method because of SHGetFileInfo() limitations.

Interesting GetTarget() can not be generic since there is used a specific struct for every RTPT kind. See https://github.com/PowerShell/PowerShell/blob/1ba2960f5c5df9de4c079c97644e9af7bf34385b/src/System.Management.Automation/namespaces/FileSystemProvider.cs#L8462-L8485
It is exactly like with Create method and we could use a similar approach in the implementation as described above.

---

My intermediate conclusion is we need new API in FileSystemEntry class (see #52666 (comment)) and in FileSystemInfo class (no API proposal yet).

[mklement0]

Let me try to give a conceptual overview, which hopefully also answers your questions, @iSazonov and @carlossanlop:

• Reparse points are a general, open-ended NTFS-specific mechanism for associating user-defined(!) data with file-system entries; a reparse point is composed of:

  • a "public" reparse tag, which is bit-field of known format that contains metadata such as whether the reparse point is Microsoft-defined and whether it is a name surrogate, indicating a file-system link, and an open-ended, user-defined type identifier (claiming such an identifier requires coordination with Microsoft, it seems).
  • associated "private" data in an unspecified format, to be used transparently by whatever application creates a specific type of reparse point, via application-specific file filters (@iSazonov, this is the raw data previously mentioned).

• In other words:

  • That file-system links are based on reparse points is a mere NTFS-specific implementation detail.
  • Not every reparse point is also link - on the contrary - and the two shouldn't be conflated.
    • A prime example are OneDrive reparse points, which aren't links, and are therefore not relevant to a .NET API focused on links; such reparse points do not point to another file-system path; instead, they contain information for loading content on demand, at that path, from a remote location (cloud) whose address is an implementation detail.

• Providing a complete reparse-point API is a separate concern:

  • Doing so would require exposing the reparse-point operations (which includes the ability to retrieve the reparse-point data).

    • Of course, to a caller who knows the format of a given type of reparse point, that data could be useful, @carlossanlop.
    • Knowing in the abstract that accessing a path - presumably just its content - could trigger an on-demand cloud download may also be helpful, but for that checking the reparse tag for IO_REPARSE_TAG_CLOUD may be sufficient - though two questions remain:
      • How can a caller check whether the remote content is presently already downloaded or not?
      • With respect to .NET file-system enumeration APIs: Should the do-not-recurse-into-directory-link-targets opt-in discussed in Allow opting out of following directory links (reparse points / symlinks) during recursive file-system enumerations #52666 also apply to not-yet-downloaded cloud directories?

  • Beyond this, there's little .NET can do, because the format and content of the data are not standardized and defined by whatever (third) party that introduces a given reparse-point type.

  • As an aside: In the abstract you will be able to identify non-link reparse points as follows, once an .IsLink property is exposed (see API Proposal: tell if a filesystem entry is any type of link #53577): info.Attributes.HasFlag(FileAttributes.ReparsePoint) && ! info.IsLink

• As for a link API:

  • For mere content operations, even file-system links can be treated as opaque: reading a file link's content and enumerating a directory link's entries implicitly and transparently operates on the link's target.

  • However, knowing that a given path refers to a link and being able to query what other path the link - directly or ultimately - points to can situationally be of interest, such as when PowerShell lists directory entries both in terms of their own name and their ultimate target path; specifically, it is important:

    • to know whether you're deleting an actual file / directory or merely a "pointer" (link) to it.
    • to be able to query the target's metadata (timestamps), given that a link's own metadata - other than the target path - is usually irrelevant.

  • The NTFS challenge is that in order to explicitly identify a link's target (path), the reparse point's type-specific, non-standardized data format must be known:

    • It's not hard to do that for symbolic links, mount point / junctions and AppX reparse points; PowerShell has already done that, and it makes sense to provide at the level of a .NET API.

    • Additionally, the following among the officially listed reparse tags also have the name-surrogate bit set and are therefore allegedly links:

      • IO_REPARSE_TAG_IIS_CACHE, IO_REPARSE_TAG_GLOBAL_REPARSE, IO_REPARSE_TAG_LX_SYMLINK, IO_REPARSE_TAG_WCI_TOMBSTONE, IO_REPARSE_TAG_PROJFS_TOMBSTONE, IO_REPARSE_TAG_WCI_LINK, IO_REPARSE_TAG_WCI_LINK_1
      • Their specific purpose is unknown to me (beyond the abstract description on the linked page), so I cannot say whether supporting them via a .NET link API makes sense too.

    • However, any third party can come along and define new types of links, using arbitrary data formats - all you would be able to tell is that something is a link, via the reparse tag, but there would be no predictable way to extract the target information from the data (which may or may not be expressed directly as the target's path).

    • To put it differently:

      • There is no way that .NET can generically provide link-target query functionality for all hypothetical future NTFS reparse points that self-advertise as file-system links (have the name-surrogate bit set in the reparse tag).
      • Then again, pragmatically speaking, the introduction of non-link reparse points, similar to the load-on-demand-from-the-cloud ones for OneDrive (though note that the description of IO_REPARSE_TAG_CLOUD suggests that other could vendors can hook into this too), strikes me as much more likely than true link reparse points (pointing to another, user-visible path in the file-system).

[hamarb123]

There is no way that .NET can generically provide link-target query functionality for all hypothetical future NTFS reparse points that self-advertise as file-system links (have the name-surrogate bit set in the reparse tag).

I agree this is not possible, but I was under the impression that the reason we're doing it this way was to make it forwards compatible if say in 20 years we don't like symbolic links anymore and want to make a new better one, or something else like that.

Also, I think that it is important to provide a "safe" API for links (ie. .NET knows what the link type is, how to determine the target etc., or it throws an error), but I think that it is also important to be able to provide an "unsafe" API as well to allow for reading/writing raw link data on any ReparseType so that the developers don't have to P/Invoke it on Windows (should also expose on Unix for symbolic links so there is definitely no resolution happening / can set it to anything if it is desired).

Maybe we could have something like this:

+   public enum ReparseTag
+   {
+       None = -1,                // Default, indicates the file is not a Reparse Point (zero is already reserved)
+       MountPoint = 0xA0000003,  // Maps to IO_REPARSE_TAG_MOUNT_POINT, describes a Junction
+       Symlink = 0xA000000C,     // Maps to IO_REPARSE_TAG_SYMLINK, the only value used by Unix
+       AppExecLink = 0x8000001B, // Maps to IO_REPARSE_TAG_APPEXECLINK
+   }

    public abstract partial class FileSystemInfo
    {
+       public ReparseTag ReparseTag { get; }
+       public string ReparseTarget { get; set; } //only works on supported .NET reparse points
+       public byte[] RawReparseData { get; set; } //throws when setting with ReparseTag.None
    }

and similar apis for creating etc. (having variants for both string and raw where string fails if .NET doesn't know explicitly what to do)
Also related to #24271

[mklement0]

Sounds good in principle, @hamarb123, but I think it is important to keep the link and the reparse-point APIs separate.

Note that the API approved in #24271 (comment) already includes a - notably read-only - public string? LinkTarget { get; } property to report the target path as defined in the link (assuming .NET understands the link type on NTFS); changing the target path requires re-creation of the link, which I think is reasonable.

I don't think a (read/write) ReparseTarget property is needed - that's what public byte[] RawReparseData { get; set; } would cover, for all types of reparse points, not just those that happen to be links.

[tmds]

It would probably make sense to add ReparseTag to System.IO.Enumeration.FileSystemEntry too:

ref struct FileSystemEntry
{
    public ReparseTag ReparseTag { get; }
}