Linux and Mac have CryptographicException with generic HResult for bad password opening a PFX

[heng-liu]

The Invalid ASN and bad password have both generic HResult in netcore5.0, so we're not able to differentiate them as we did in netcore3.0.
Can you pls help us change HResult for bad password? Thanks!

[danmoseley]

so we're not able to differentiate them as we did in netcore3.0.

Is this a change from 3.0 to 3.1 then?

[bartonjs]

Is this a change from 3.0 to 3.1 then?

No, 3.x -> 5 (the custom PFX loader change)

[danmoseley]

Thanks, misread.

[vcsjones]

Windows appears to use ERROR_INVALID_PASSWORD converted to an HRESULT (0x80070056). Is this basically just adding the HRESULT to this line?

runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.Unix/UnixPkcs12Reader.cs

Line 228 in 73eec1b

throw new CryptographicException(SR.Cryptography_Pfx_BadPassword);

[bartonjs]

@vcsjones Yep. You wanna take it? :)

[vcsjones]

@bartonjs sure.

[vcsjones]

Looking at this a bit more, the Win32 PAL is inconsistent between the PAL using an HRESULT and a Win32 status code when setting the HResult on the CryptographicException.

In some cases, Marshal.GetHRForLastWin32Error().ToCryptographicException() is used:

runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.Windows/CertificatePal.Import.cs

Line 153 in de12f85

throw Marshal.GetHRForLastWin32Error().ToCryptographicException();

So HResult will be 0x80070056.

In another case, Marshal.GetLastWin32Error().ToCryptographicException() is used:

runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.Windows/StorePal.Import.cs

Line 77 in de12f85

throw Marshal.GetLastWin32Error().ToCryptographicException();

So HResult will be 0x56 and won't really be an HR.

I suppose a question for @bartonjs then is, which is preferable for the Unix PAL to do? Do we want to do anything about the Windows PAL?

[bartonjs]

Looks like .NET Framework threw with HResult=0x80070056 in both the collection and the single-cert case. (Largely because in .NET Framework we or-in 80070000 if the high bit isn't already set: https://referencesource.microsoft.com/#mscorlib/system/security/cryptography/crypto.cs,108; perhaps that's missing in the ToCryptographicException extension method)