RSA.VerifyData terminates with "Out of memory." on Linux when passing an all-zero modulus

[Thealexbarney]

Description

If an all-zero modulus is passed to RSA.VerifyData on Linux, the program will unexpectedly terminate instead of throwing an exception.

Reproduction Steps

using System.Security.Cryptography;

var param = new RSAParameters{Modulus = new byte[256], Exponent = new byte[3]};
using (var rsa = RSA.Create(param))
{
    rsa.VerifyData(new byte[512], new byte[256], HashAlgorithmName.SHA256, RSASignaturePadding.Pss);
}

Expected behavior

VerifyData throws a CryptographicException like it does on Windows.

Actual behavior

The program terminates with Out of memory. Command terminated by signal 6

Regression?

Issue occurs on .NET 6 and 7. Unsure about earlier versions.

Known Workarounds

No response

Configuration

I've tested running .NET 6.0 and 7.0 on both x64 Ubuntu 22.04 and Debian 11.

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

If an all-zero modulus is passed to RSA.VerifyData on Linux, the program will unexpectedly terminate instead of throwing an exception.

Reproduction Steps

using System.Security.Cryptography;

var param = new RSAParameters{Modulus = new byte[256], Exponent = new byte[3]};
using (var rsa = RSA.Create(param))
{
    rsa.VerifyData(new byte[512], new byte[256], HashAlgorithmName.SHA256, RSASignaturePadding.Pss);
}

Expected behavior

VerifyData throws a CryptographicException like it does on Windows.

Actual behavior

The program terminates with Out of memory. Command terminated by signal 6

Regression?

Issue occurs on .NET 6 and 7. Unsure about earlier versions.

Known Workarounds

No response

Configuration

I've tested running .NET 6.0 and 7.0 on both x64 Ubuntu 22.04 and Debian 11.

Other information

No response

Author:	Thealexbarney
Assignees:	-
Labels:	area-System.Security
Milestone:	-

[vcsjones]

Note, I can only reproduce this with OpenSSL 1.1. Using OpenSSL 3, I get an "invalid modulus" CryptographicException error which seems reasonable.

I'll take a look at what's going on with OpenSSL 1.1.

[vcsjones]

A few questions about the report:

I've tested running .NET 6.0 and 7.0 on both x64 Ubuntu 22.04

Both .NET 6 and .NET 7 should use OpenSSL 3, which should be present on Ubuntu 22.04. @Thealexbarney can you confirm if you are using OpenSSL 3 or not? If you aren't certain you can add Console.WriteLine(SafeEvpPKeyHandle.OpenSslVersion.ToString("X")); somewhere in your program. If that output starts with 3, then you should be using OpenSSL 3.

I can't reproduce this under OpenSSL 3.

And this:

VerifyData throws a CryptographicException like it does on Windows.

I don't get an error from VerifyData. I get an error from ImportParameters (called by RSA.Create). Are you seeing the import succeed and VerifyData throw, or is it RSA.Create that is really throwing?

For OpenSSL 1.1, this isn't a "real" OutOfMemoryException. This is us translating the error OpenSSL gave us to the appropriate exception.

This is our managed stack:

fff7b9eeac [HelperMethodFrame: 0000ffffffffe638] 
0000FFFFFFFFE780 0000FFFFB83BC090 Interop+Crypto.RsaVerifyHash(System.Security.Cryptography.SafeEvpPKeyHandle, System.Security.Cryptography.RSASignaturePaddingMode, IntPtr, System.ReadOnlySpan`1<Byte>, System.ReadOnlySpan`1<Byte>) [/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.Rsa.cs @ 190]
0000FFFFFFFFE7D0 0000FFFFB843E708 System.Security.Cryptography.RSAOpenSsl.VerifyHash(System.ReadOnlySpan`1<Byte>, System.ReadOnlySpan`1<Byte>, System.Security.Cryptography.HashAlgorithmName, System.Security.Cryptography.RSASignaturePadding) [/_/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs @ 835]
0000FFFFFFFFE820 0000FFFFB843E630 System.Security.Cryptography.RSAOpenSsl.VerifyHash(Byte[], Byte[], System.Security.Cryptography.HashAlgorithmName, System.Security.Cryptography.RSASignaturePadding) [/_/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs @ 824]
0000FFFFFFFFE880 0000FFFFB84227D4 System.Security.Cryptography.RSA.VerifyData(Byte[], Int32, Int32, Byte[], System.Security.Cryptography.HashAlgorithmName, System.Security.Cryptography.RSASignaturePadding) [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs @ 624]
0000FFFFFFFFE8D0 0000FFFFB8280EF4 Program.<Main>$(System.String[]) [/app/Program.cs @ 6]

The top frame is pointing here:

runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.Rsa.cs

Line 190 in d099f07

throw CreateOpenSslCryptographicException();

So, this is not a "real" OutOfMemoryException. What CreateOpenSslCryptographicException does is throw the exception that OpenSSL told us to. In this particular case, OpenSSL gave us an ERR_R_MALLOC_FAILURE, which we translate in to an OutOfMemoryException.

runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.ERR.cs

Line 75 in d099f07

if (isAllocFailure)

There are a couple of options here:

1. Do nothing. We throw what OpenSSL reports, and in this case OpenSSL said ERR_R_MALLOC_FAILURE. This appears fixed in OpenSSL 3.
2. We check Modulus during import in managed code. This would bring consistency with OpenSSL 3, Windows, and macOS, which fail to import the key in the first place. We would need to verify all of the appropriate places that need to check the modulus (parameters, RSAPublicKey, SPKI, etc). OpenSSL 1.1.1 seems like the only outlier in behavior here.

@bartonjs thoughts?

[vcsjones]

For OpenSSL 1.1, this isn't a "real" OutOfMemoryException.

As a work around, in this case you can catch this OutOfMemoryException.

using System;
using System.Security.Cryptography;

var param = new RSAParameters{Modulus = new byte[256], Exponent = new byte[3]};
using (var rsa = RSA.Create(param))
{
    try
    {
        rsa.VerifyData(new byte[512], new byte[256], HashAlgorithmName.SHA256, RSASignaturePadding.Pss);
    }
    catch (Exception e) when (e is CryptographicException or OutOfMemoryException)
    {
        Console.WriteLine("bad key");
    }
}

Console.WriteLine("All done!");

Gives me the output:

bad key
All done!

[Thealexbarney]

Both .NET 6 and .NET 7 should use OpenSSL 3, which should be present on Ubuntu 22.04.

My mistake. It was Ubuntu 20.04 and Debian 11 that I tested the issue on.

I don't get an error from VerifyData. I get an error from ImportParameters (called by RSA.Create). Are you seeing the import succeed and VerifyData throw, or is it RSA.Create that is really throwing?

It's the VerifyData call that's throwing.

using System;
using System.Security.Cryptography;

var param = new RSAParameters{Modulus = new byte[256], Exponent = new byte[3]};
using (var rsa = RSA.Create(param))
{
    try
    {
        rsa.VerifyData(new byte[512], new byte[256], HashAlgorithmName.SHA256, RSASignaturePadding.Pss);
    }
    catch (Exception e) when (e is CryptographicException or OutOfMemoryException)
    {
        Console.WriteLine("bad key");
    }
}

Console.WriteLine("All done!");

Gives me the output:

Running VerifyData
bad key
All done!

[vcsjones]

It's the VerifyData call that's throwing.

Just to confirm - my question was about Windows where you stated that "VerifyData throws a CryptographicException like it does on Windows."

So you are seeing Windows throw during VerifyData? If so, what version of Windows is that? Here is my understanding of things:

1. OpenSSL 3: Fails in RSA.Create with CryptographicException.
2. OpenSSL 1.x: Fails in RSA.VerifyData with OutOfMemoryException.
3. Windows 11: Fails in RSA.Create with CryptographicException.
4. macOS 13: Fails in RSA.Create with CryptographicException.

[Thealexbarney]

Just to confirm - my question was about Windows where you stated that "VerifyData throws a CryptographicException like it does on Windows."

Ah, okay. You're right. It's RSA.Create that's throwing the exception on Windows

[bartonjs]

Hopefully the problem is just something like that we end up turning all zeros into the NULL BN* and we fix that to be BN_zero() (at least, I assume there's a BN_zero()) and now the key sanity test fails and the exception moves to Import.

I'm a bit wary of adding custom validation to key parameters, especially if it's not universal. That said, if the only hole we know of is a non-null-but-zero Modulus then I guess consistency is good. (Presumably that'd be done with our fancy new vectorized searches if (modulus.IndexOfAnyExcept(0) < 0))

[vcsjones]

This got reverted because it broke the x86 Linux build, so re-opening so we can get it in again.