Add platform native long support to runtime

[OtherCrashOverride]

Currently, C# long maps to System.Int64 in the runtime. However, the native long may be 32bit or 64bit depending on platform. We need a new type that transitions from 32bit to 64bit as appropriate for the platform in much the same way as IntPtr does. This issue is present when you compile a C/C++ project on 64bit platforms and need to perform interop.

Windows is a LLP platform.
Windows 32bit: native long = 32bits
Windows 64bit: native long = 32bits

Linux is a LP platform.
Linux 32bit: native long = 32bits;
Linux 64bit: native long = 64bits;

For clarity, IntPtr is ALWAYS 32bit on 32bit platforms and ALWAYS 64bit on 64bit platforms. The native long may be 32bit or 64bit depending on the 64bit platform in use. This is related to #4216. The goal is to make the p/invoke signature match without rewriting the import for each target platform (windows/linux).

I have searched the issues and did not find this as a duplicate. However, if it is, please feel free to mark it as such and close it.

[dlech]

Duplicate of #4233?

This is a good point that has not been brought up before in that discussion. However, I would like to also point out that on Windows libraries can be compiled with gcc (LP) in addition to msvc (LLP), so native long on Windows doesn't depend on the fact that the OS is Windows, but rather on which compiler was used for the native library.

The problem of "natural floats" discussed in #4233 is pretty much the same problem. The need for it is only indirectly tied to the processor size. In the use case in that issue, the need for nfloat primarily comes the fact that CGFloat in CoreGraphics on OSX/iOS is float in ILP32 and double in LP64.

So, I think an important criteria for a solution here is that any "native" types (other than the already exiting native int/IntPtr) should not depend on processor architecture. Additionally, it should not depend on the operating system. Instead, it has to be specified at compile time or at runtime based on the specific binary version of the library you are targeting.

The CLR runtime doesn't have a way to know what compiler was used to compile a "native" library and it doesn't seem like it should. One solution I have thought of is to use the "bait and switch" trick that is used with PCLs. It would work something like this... You define all of your [DllImport] stuff in a common library using nlong (or whatever you would like to call it). Then you define the actual nlong struct in a separate assembly. There will be 2 versions of this assembly one for LP64 and one for ILP32/LLP64. In the LLP64/ILP32 versions, nlong is 32-bit and in the LP64 version, nlong is 64-bit. (If there were additional data types with different sizes in ILP32 and LLP64, then you would need a 3rd version of the assembly). But, this solution is not so good currently because you can't inherit from value types. You have to implement all of the operators and whatnot manually.

Another possible solution is to use #if along with using. One downside is that you have to do this in every single file that calls the extern methods. Another downside is that you can't have a "universal" binary that runs anywhere. It would look like this:

#if LP64
using nlong = System.Int64;
#elif LLP64 || ILP32
using nlong = System.Int32;
#else
#error Must specify native library compiler type.
#endif

[OtherCrashOverride]

The difference between this issue and #4233 is that LP and LLP are system wide. They are the standard for the platform as a whole. The CGFloat issue differs in that it is specific to a single API. It singularly affects CoreGraphics and is not a native type: it is an API defined type. It only affects code targeting the CoreGraphics API. native float is always 32bits regardless of the definition of CGFloat.

native long however may be 32 or 64bit and affects all system operation including kernel system calls since it is part of the platform ABI. Supporting a compiler or toolchain that does not produce the correct ABI for the target platform is outside the scope of this issue. There is no need to switch the size at runtime. native long follows the same rules for size changes as IntPtr does. For example, running a 32bit compiled runtime on a 64bit platform makes both native long and IntPtr 32bit.

Another downside is that you can't have a "universal" binary that runs anywhere.

The purpose of native long is to prevent having more than one versions of the same assembly as is currently required today. It allows for "universal" binaries the exact same way IntPtr does.

[dlech]

One of the points that I was trying to make is that LP64 and LLP64 are not necessarily system wide. For example, Cygwin uses LP64 on Windows. So, if you say nlong is 32-bit on Windows, it doesn't work as expected if you pinvoke a cygwin compiled library.

[OtherCrashOverride]

One of the points that I was trying to make is that LP64 and LLP64 are not necessarily system wide.

For the intent and purpose of this issue, they are always system wide. "foreign environments" will inherit the behaviour of the native environment: if a developer is using a LP64 environment on a native LLP64 system, they will be responsible for accommodating it just as is required for C/C++ without the presence of CoreCLR.

CoreCLR is, as the name suggests, minimal. It has a greater need for interop than the previous "full frameworks". Now that LP64 systems (mac/linux) are officially supported targets, a mechanism is needed support interop on them. There are many native libraries that compile natively (sans Cygwin) for windows, linux and mac. If the public APIs of such libraries exposes long, p/invoke will work for mac/linux or window, but not both, when the runtime environment is 64bits.

The heart of this issue is that there is currently no way to express a native long in CoreCLR. This was not an issue before because on windows native long == int32 always. Supporting this types means the CoreCLR runtime can be compiled to target LP64 or LLP64. There is no restriction implied that disallows LP64 compilation on a LLP64 target. A developer can use a LP64 compiled version of the runtime on a LLP64 target (windows) if they are targeting Cygwin. However, there is no need to "switch-at-runtime". As an example, a 32bit CoreCLR does not switch at runtime to 64bits. Instead there are 32bit and 64bit builds. The same would apply to LP64 and LLP64 being separate builds.

[mjsabby]

I'm not sure there is anything here for CoreCLR to do, the support for a pointer-size type already exists in MSIL as "native int", and in C# as IntPtr. This deferred resolution of size at runtime or architecture-specific compile time is all that is needed for you to implement native interop where the size of a parameter/return type is dependent on the size of the pointer.

long/longlong and abstract data model issues exist outside the realm of the runtime. Taking the example of C++, what happens when two different compilers compiling two different translation units where "long" is used need to interact. Is it 32-bit, or 64-bit?

Portable C++/native code should define their data type size concretely as X bytes, or if it is dependent on architecture pick whatever data type represents that, for MSVC++ that is SIZE_T.

The real issue here (it seems) is that the same "source code" can be compiled differently by different compilers on the same (or different) platforms and long/longlong may only be one specific instance of this difference.

So you have a couple of options, you can either make the native code more portable by being explicit about your data type sizes or rely on pointer-size widths if that suits your purpose (and is of course semantically correct, etc.), and both of these are supported by the runtime.

[OtherCrashOverride]

So you have a couple of options, you can either make the native code more portable by being explicit about your data type sizes or rely on pointer-size widths if that suits your purpose (and is of course semantically correct, etc.), and both of these are supported by the runtime.

Rewriting every existing library on mac and linux to use explicit sizes is not a realistic solution. The converse argument is that Microsoft should just rewrite Windows to be a LP64 platform like everyone else. Both are unrealistic premises. Additionally, I was explicit about "why IntPtr is not a solution" in the opening post. I will reiterate it again here:

For clarity, IntPtr is ALWAYS 32bit on 32bit platforms and ALWAYS 64bit on 64bit platforms. The native long may be 32bit or 64bit depending on the 64bit platform in use.

Knowing the size of IntPtr on a 64bit platform (which is ALWAYS 64bit), does not provide any knowledge about the size of native long.

I'm not sure there is anything here for CoreCLR to do

Copy/paste the code that provides IntPtr and UIntPtr. Give them a new name. Change the size rules to be a compile constant based on platform (Windows/everyone else) in addition to the rule that is based on architecture (32/64bit).

This is not about supporting every possible combination of compiler. There is absolutely no need to consider anything other than supported platforms and supported compilers. Cygwin and different translation units with different compilers are simply red herrings. The issue is that there is no way to express that despite IntPtr.Size = 8 on all 64bit platforms, NativeLong.Size=4 on windows and NativeLong.Size=8 on Mac and Linux.

Assuming a 64bit environement, the following illustrates the problem to solve:

C code:
long SomeFunction(long argument);

C# code to call it on Windows:
[DllImport(LibraryName)] public static extern int SomeFunction(int argument);

C# code to call it on non Windows:
[DllImport(LibraryName)] public static extern long SomeFunction(long argument);
(Note that this breaks when run on 32bit)

The following works on Mac/Linux, but not Windows:

[DllImport(LibraryName)] public static extern IntPtr SomeFunction(IntPtr argument);

The desired solution (using the platform provided compiler and toolchains) is:

C# code to call it on ALL platforms and ALL architectures:
[DllImport(LibraryName)] public static extern NativeLong SomeFunction(NativeLong argument);

[Edit]
It is implied so I will also explicitly state there is a need for both NativeLong and NativeULong.

[dlech]

I buy the argument that we can ignore the fact Cygwin is LP64 on Windows and just say that Windows is LLP64. It actually makes sense that if you want to run coreclr in Cygwin that you would use a binary version of coreclr that targets Cygwin that is more like the current Linux version of coreclr rather than using a Windows version of the coreclr.

I also second the argument that the "fixing" the code of every library we pinvoke is quite unrealistic. The most common use case for pinvoke by far is system libraries and other well-know widely-used libraries that are not going to change because it would break everybody everywhere.

I'm on board with what @OtherCrashOverride is suggesting here other than I might prefer to call it CLong and CULong since it is tied to the C data type (using the C prefix to be like CallingConvention.Cdecl).

[KrzysztofCwalina]

IntPtr.Size is not always 8 "on 64-bit platforms" (if by platform you mean processor architecture). If you compile to x86 specifically (as opposed to AnyCpu), Windows process will be forced to 32-bit, and so IntPtr.Size will be 4.

Can this be used to address the "native long" issue? i.e. you would build two dlls with forced 32-bit and forced 64-bit, and then select the appropriate one depending on LP/LLP?

[OtherCrashOverride]

IntPtr.Size is not always 8 "on 64-bit platforms"

Yes, running a 32bit process on a 64bit platform is the same as a 32bit platform and the 32bit rules apply.

you would build two dlls with forced 32-bit and forced 64-bit, and then select the appropriate one depending on LP/LLP?

There are various ways to "work around" it including #if and platform specific assemblies. Without first class support in the runtime, the typical "work around is" to simply not support a platform (typically Windows) since all the interop APIs using "long" where this matters are non-Widows libraries. The impact to developers is the same as if IntPtr did not exist in the choices they must make.

[fanoI]

I do not think that C# should add all "no sense" types of C / C++ long that is 32 or 64 bit "at random", long long (how a thing could be more "long" than "long"), if not we will need soon "native char" (UTF-8 or UTF-16 based on the platform?).

Why this mess cannot solved using marshalling attributes? Without adding types that make no sense in the clean C# type system?

[OtherCrashOverride]

I do not think that C# should add all "no sense" types of C / C++ long that is 32 or 64 bit "at random"

There is nothing "random" about it. It is a real world issue faced by real developers.

Why this mess cannot solved using marshalling attributes?

An attribute can not solve the issue anymore than it could solve the issue for IntPtr. Its need is exactly the same as IntPtr. Windows is the only platform where an IntPtr can not be used to express "native long".

[rolfbjarne]

Maybe something more generic can be implemented, which would work for other type variations as well:

class NativeLongFieldTypeAttribute : ConditionalFieldTypeAttribute {
    public override Type GetFieldType () {
        if (windows) {
            return typeof (int);
        } else {
            return IntPtr.Size == 8 ? typeof (long) : typeof (int);
        }
    }
}

struct nlong {
    [NativeLongFieldType]
    ValueType value;
}

And at runtime the JIT calls ConditionalFieldTypeAttribute.GetFieldType () and replaces the type of the value field with the returned type.

[fanoI]

Could not be done simply in this way:

extern void func([MarshalAs(UnmanagedType.NativeLong)] long a);

on Windwows32 / Windows64 'a' will be the same thing of System.Int32 on Linux32 it will be System.Int32 while on Linux64 it will be System.Int64!
While I find nint / nuint of limitate usage as a proper type in C# itself (but I hope to see used in low level code seeing people use nint as the "i" variable of a for will make me puke!) I find nlong useful only on Native Interop, a nlong type will create confusion with the "real" C# long.

There is the precedent of SysInt/ SysUInt so SysLong / SysULong will be the more correct name:
https://msdn.microsoft.com/en-us/library/system.runtime.interopservices.unmanagedtype(v=vs.110).aspx

[OtherCrashOverride]

Could not be done simply in this way:

The attribute would need to apply to structures in addition to p/invoke signatures. The proposal would need to be tested in real-world scenarios, but its better than nothing.

[xanatos]

For structs, adding a MarshalAs that can change the size of the marshaled type could change a struct from blittable to non-blittable, making marshaling much slower on 64 bit linux (and similar LP systems).

[jkotas]

Duplicate of #27530

[DeafMan1983]

Nice idea - You know I have ported from Java Native Access to C# but not complete implements for C#. Thank you for trick with Type GetFieldType() with int and long. But what is about Pointer and PointerType for C# under Dotnet. Thanks

Update
I add new project library "Dotnet.Native.Access"
If you make sure for alternative to IntPtr.

Good luck!