Last p/invoke error is not 0 after calling a p/invoke with SetLastError=true that doesn't set the system error

[elinor-fung]

See #51505 (comment).
Test case is under src/tests/Interop/PInvoke/SetLastError

For p/invokes with SetLastError=true, the system error should be set to 0 before calling the target function and the system error saved as the last p/invoke error after the call, such that if the target function doesn't update the system error, the stored p/invoke error will be 0. This does not seem to be the case on mono.

• Call a P/Invoke that has SetLastError=true but that doesn't set the system error
• Call Marshal.GetLastPInvokeError or Marshal.GetLastWin32Error

Example:

[DllImport("NativeLib", SetLastError = true)]
public static extern void SetError(int error, byte shouldSetError);
...
SetError(100, shouldSetError: 1);
// Marshal.GetLastPInvokeError/GetLastWin32Error return 100, as expected

SetError(50, shouldSetError: 0);
// Marshal.GetLastPInvokeError/GetLastWin32Error should return 0, but does not

extern "C" DLL_EXPORT void STDMETHODCALLTYPE SetError(int err, bool shouldSetError)
{
    if (!shouldSetError)
        return;

#ifdef WINDOWS
    ::SetLastError(err);
#else
    errno = err;
#endif
}

[lambdageek]

#51505 (comment)

Should work the same way as CoreCLR - errno should be set to 0 before the call, and then after the call mono_marshal_set_last_error should get called to push the current errno value to the last pinvoke error thread-local variable.

It's possible that it isn't working right (maybe while JITing the wrapper method for the pinvoke, the JIT sets errno to 1 for some reason).

[SamMonoRT]

@naricc - maybe tackle this next week.