add support for PEM_write_bio_PrivateKey_traditional

[samuel40791765]

Issues:

Resolves CryptoAlg-2546

Description of changes:

The main difference between PEM_write_bio_PrivateKey_traditional and the modern PEM_write_bio_PrivateKey is that PEM_write_bio_PrivateKey writes a PKCS#8 encoding, while the traditional one writes with DER encoding.
OpenSSL's PEM_write_bio_PrivateKey_traditional calls into i2d_PrivateKey which calls into another additional old_priv_encode EVP_PKEY_ASN1_METHOD hook. We also need to make the call into i2d_PrivateKey, but luckily for us BoringSSL removed the hook and let the call point to the corresponding i2d_RSA/DSA/ECPrivateKey methods instead. We can built on this directly to reimplement PEM_write_bio_PrivateKey_traditional.

• aws-lc/crypto/x509/i2d_pr.c

  Lines 65 to 79 in 51d9a8d

  	int i2d_PrivateKey(const EVP_PKEY *a, uint8_t **pp) {
  	switch (EVP_PKEY_id(a)) {
  	case EVP_PKEY_RSA:
  	return i2d_RSAPrivateKey(EVP_PKEY_get0_RSA(a), pp);
  	case EVP_PKEY_EC:
  	return i2d_ECPrivateKey(EVP_PKEY_get0_EC_KEY(a), pp);
  	case EVP_PKEY_DSA:
  	return i2d_DSAPrivateKey(EVP_PKEY_get0_DSA(a), pp);
  	default:
  	// Although this file is in crypto/x509 for layering reasons, it emits
  	// an error code from ASN1 for OpenSSL compatibility.
  	OPENSSL_PUT_ERROR(ASN1, ASN1_R_UNSUPPORTED_PUBLIC_KEY_TYPE);
  	return -1;
  	}
  	}

• OpenSSL
  • Usage of ASN1 hook: https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/crypto/asn1/i2d_pr.c#L17-L33
  • Check for ASN1 hook: https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/crypto/pem/pem_pkey.c#L105-L120

PEM_read_bio_PrivateKey already has logic to parse the "traditional form" of these PEM files. I've also added tests to verify.

Call-outs:

N/A

Testing:

Tests for each type writable/readable. A single test for DH, which shouldn't be writable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.34%. Comparing base (c664abe) to head (ba048f1).
Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1845      +/-   ##
==========================================
- Coverage   78.56%   78.34%   -0.22%     
==========================================
  Files         583      584       +1     
  Lines       98807    99236     +429     
  Branches    14161    14188      +27     
==========================================
+ Hits        77623    77743     +120     
- Misses      20558    20864     +306     
- Partials      626      629       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[WillChilds-Klein]

minor nit

Suggested change

	BIO_snprintf(pem_str, 80, "%s PRIVATE KEY", x->ameth->pem_str);
	BIO_snprintf(pem_str, sizeof(pem_str), "%s PRIVATE KEY", x->ameth->pem_str);

and how was 80 selected?

[samuel40791765]

No particular reason, this was taken from OpenSSL. They probably just chose a size large enough to fit the string with "pem_str" and PRIVATE KEY.