Design for support of HMAC precomputed keys #1574

fabrice102 · 2024-05-02T19:30:00Z

Issues:

N/A

Description of changes:

This commit implements an optimization of HMAC described in NIST-FIPS-198-1 (Section 6) and in RFC2104 (Section 4).

Instead of using the original key, it is possible to use a precomputed key consisting of two intermediate hashing results. Using this precomputed key removes at least two calls to the hash compression function when computing a HMAC value.

The first commit from May 06, 2024 is an initial design. As such, only SHA-256 is supported and all other hashes are disabled from HMAC. This makes some unit tests fail. Once this design is validated, we will add back the other hash functions in additional commits.

Changes:

May 6, 2024: this PR was split in two. The ability to export state of SHA-256 was moved to Initial design for export of Merkle-Damgard hash state #1577

Call-outs:

This commit adds a few fields to the methods field of the HMAC context. This slightly increases the size of the HMAC_context (by 24 bytes on a 64-bit architecture).

Testing:

Testing was added in two places:

the C++ test harness based on Google Test (file crypto/hmac_extra/hmac_test.cc)
the ACVP harness (file util/fipstools/acvp/modulewrapper/modulewrapper.cc)

To allow partial testing, we updated hmac_tests.txt to only contain the HMAC-SHA256 tests. The original file is hmac_tests_full.txt

Testing can be done in the CMake build folder using:

crypto/crypto_test --gtest_filter='*InitAndGetStateSHA256*:HMACTest.TestVectors' for the new/changed unit tests
ninja acvp_tests for the ACVP tests (failure is expected for anything using HMAC-xx where xx is not SHA-256)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

This commit provides a building block to implement HMAC with precomputed keys from aws#1574. It adds internal functions to export the state of SHA-256 and use such exported state to initialize a SHA-256 context. Once the design is reviewed and approved, we will similar functions to the other Merkle-Damgard hash functions (MD4, MD5, SHA-1, SHA-2).

This commit implements an optimization of HMAC described in NIST-FIPS-198-1 (Section 6) and in RFC2104 (Section 4). Instead of using the original key, it is possible to use a precomputed key consisting of two intermediate hashing results. Using this precomputed key removes at least two calls to the hash compression function when computing a HMAC value. As an initial design, only SHA-256 is supported and all other hashes are disabled from HMAC. This makes some unit tests fail. Once this design is validated, we will add back the other hash functions. To allow partial testing, we updated hmac_tests.txt to only contain the HMAC-SHA256 tests. The original file is hmac_tests_full.txt Testing can be done in the CMake build folder using: * `crypto/crypto_test --gtest_filter='HMACTest.TestVectors'` for the changed unit tests. * `ninja acvp_tests` for the ACVP tests (failure is expected for anything using HMAC-xx where xx is not SHA-256).

nebeid

Some comments on the header files. Will continue reviewing the implementation.

include/openssl/digest.h

include/openssl/hmac.h

nebeid

I'm continuing to review hmac.c; I was focused on the state transition and will continue reviewing the new functions.

include/openssl/digest.h

include/openssl/hmac.h

crypto/fipsmodule/hmac/internal.h

crypto/fipsmodule/hmac/hmac.c

nebeid

After you make changes, Fabrice, I'll spend more time on the new implementations in sha256.c and the tests.

crypto/fipsmodule/hmac/hmac.c

include/openssl/digest.h

crypto/fipsmodule/hmac/hmac.c

crypto/fipsmodule/sha/internal.h

crypto/fipsmodule/sha/sha256.c

crypto/fipsmodule/sha/internal.h

nebeid

I will review the tests shortly.

include/openssl/hmac.h

nebeid · 2024-06-12T15:41:01Z

crypto/fipsmodule/hmac/hmac.c

In many external API calls here, we're directly accessing ctx->state or other members of ctx without checking it's not null. I think we should handle this as was done in #1398. This can be its own PR.

crypto/fipsmodule/hmac/internal.h

crypto/fipsmodule/hmac/hmac.c

Co-authored-by: Nevine Ebeid <66388554+nebeid@users.noreply.github.com>

include/openssl/hmac.h

crypto/fipsmodule/hmac/hmac.c

1. Improve error management: add `OPENSSL_PUT_ERROR` when appropriate. 2. Change behaviour of `out_len` from `HMAC_get_precomputed_key` as per review.

include/openssl/hmac.h

util/fipstools/acvp/modulewrapper/modulewrapper.cc

crypto/fipsmodule/hmac/internal.h

crypto/hmac_extra/hmac_test.cc

Co-authored-by: Nevine Ebeid <66388554+nebeid@users.noreply.github.com>

Previous commits in this PR/branch were restricted to SHA-256 to verify the design. Restore the full HMAC tests, which were restricted to HMAC-SHA-256 in a previous commit from this PR/branch.

crypto/fipsmodule/sha/internal.h

include/openssl/hmac.h

`crypto/err/hmac.errordata` generated using `go run ./util/make_errors.go hmac` CMakeLists file updated to add the errordata file.

crypto/fipsmodule/sha/internal.h

crypto/fipsmodule/hmac/hmac.c

From review from PR aws#1574

* Update HMAC proofs for PR aws/aws-lc#1574 HMAC precomputed keys PR aws/aws-lc#1574 adds fields to `hmac_methods_st`, which made the original proof failed. This commit updates the proof. This is a breaking change: the updated proof only works after PR aws/aws-lc#1574 is merged. * Fix .gitmodules src to point to github.com/fabrice102/aws-lc --------- Co-authored-by: Fabrice Benhamouda <yfabrice@amazon.com>

This commit implements an optimization of HMAC described in NIST-FIPS-198-1 (Section 6) and in RFC2104 (Section 4). Instead of using the original key, it is possible to use a precomputed key consisting of two intermediate hashing results. Using this precomputed key removes at least two calls to the hash compression function when computing a HMAC value. Call-outs: This commit adds a few fields to the methods field of the HMAC context. This slightly increases the size of the HMAC_context (by 24 bytes on a 64-bit architecture).

@ecdeye

## What's Changed * Added options to x509 tool by @ecdeye in #1696 * Add support to detect Neoverse V2 cores by @andrewhop in #1706 * Move OCSP functions for Ruby out of internal.h by @samuel40791765 in #1704 * Add aes-256-xts to EVP_get_cipherbyname by @torben-hansen in #1707 * Match using CMAKE_SYSTEM_PROCESSOR_LOWER by @justsmth in #1709 * Update MySQL to 9.0.0 by @skmcgrail in #1685 * [EC] Unify scalar multiplication for P-256/384/521 by @dkostic in #1693 * Adds const qualifier to ciphertext parameter in EVP_PKEY_decapsulate by @maddeleine in #1713 * Upstream merge 2024 06 24 by @nebeid in #1661 * NIST SP 800-108r1-upd1: KDF Counter Implementation by @skmcgrail in #1644 * Upstream merge 2024 07 09 by @nebeid in #1694 * Design for support of HMAC precomputed keys by @fabrice102 in #1574 * Fix for select point from table in ec_nistp scalar_mul by @dkostic in #1719 * X509toolcomparison by @ecdeye in #1714 * AWS-LC s2n-bignum update 2024-07-22 by @dkostic in #1718 * Add OpenVPN to CI by @smittals2 in #1705 * Lower required Go version, add CI test for specific version by @andrewhop in #1717 * ec2-test-framework enhancements and graviton 4 testing by @samuel40791765 in #1715 * sha + chacha: Move AArch64/X86-64 dispatching to C. by @justsmth in #1625 * Show number of pruned ec2 instances in dashboard by @samuel40791765 in #1728 * rsa and md5 tools by @ecdeye in #1722 * FIPS 203 IPD update: ML-KEM-IPD-768 and ML-KEM-IPD-1024 by @jakemas in #1724 * bump mysql CI to 9.0.1 by @samuel40791765 in #1727 * Support utility OCSP request functions by @samuel40791765 in #1708 * add support for OCSP_SINGLERESP functions by @samuel40791765 in #1703

This commit adds support for HMAC Precomputed Keys in ACCP. See aws/aws-lc#1574 This commit uses branch `hmac-precomputed-key-size-define` from https://github.com/fabrice102/aws-lc and thus cannot be merged as is. It can only be merged once the above branch is merged to AWS-LC.

This commit adds support for HMAC Precomputed Keys in ACCP. See aws/aws-lc#1574

fabrice102 mentioned this pull request May 6, 2024

Initial design for export of Merkle-Damgard hash state #1577

Closed

Fabrice Benhamouda added 2 commits May 6, 2024 13:09

fabrice102 force-pushed the hmac-precompute branch from 4acba09 to 7058ed3 Compare May 6, 2024 17:11

skmcgrail requested review from skmcgrail and nebeid May 7, 2024 16:01

skmcgrail added the reviewers-assigned label May 7, 2024

nebeid reviewed May 10, 2024

View reviewed changes

nebeid reviewed May 16, 2024

View reviewed changes

nebeid reviewed May 31, 2024

View reviewed changes

crypto/fipsmodule/sha/internal.h Show resolved Hide resolved

Fabrice Benhamouda added 4 commits June 10, 2024 17:27

Addressing comments from review of PR aws#1574

3a3a9ad

Clarifying definition of EVP_MAX_MD_CHAINING_LENGTH following review

f4b67b1

Merge branch 'main' into hmac-precompute

7148248

Remove redundant function declaration in HMAC/SHA256 trampoline

8b5dbea

nebeid reviewed Jun 12, 2024

View reviewed changes

Fabrice Benhamouda and others added 2 commits June 13, 2024 13:15

Function comments improvements from review of PR aws#1574

6734e02

Update function comment in crypto/fipsmodule/hmac/internal.h

7c68e2f

Co-authored-by: Nevine Ebeid <66388554+nebeid@users.noreply.github.com>

nebeid reviewed Jun 13, 2024

View reviewed changes

Fabrice Benhamouda added 2 commits June 14, 2024 16:14

Function comments improvements from review of PR aws#1574

1f6b510

Improve error management and check out_len - from review of PR aws#1574

a11cb33

1. Improve error management: add `OPENSSL_PUT_ERROR` when appropriate. 2. Change behaviour of `out_len` from `HMAC_get_precomputed_key` as per review.

nebeid reviewed Jun 14, 2024

View reviewed changes

fabrice102 and others added 5 commits June 14, 2024 18:38

Apply suggestions from code review

41fd25d

Co-authored-by: Nevine Ebeid <66388554+nebeid@users.noreply.github.com>

Apply suggestions from code review

6f14823

Function comments improvements from review of PR aws#1574

73debcf

Co-authored-by: Nevine Ebeid <66388554+nebeid@users.noreply.github.com>

Extend PR aws#1574 to the other hash functions

580159c

Previous commits in this PR/branch were restricted to SHA-256 to verify the design. Restore the full HMAC tests, which were restricted to HMAC-SHA-256 in a previous commit from this PR/branch.

Fix warnings when assert disabled in release mode

35b4f90

fabrice102 marked this pull request as ready for review July 11, 2024 20:17

fabrice102 requested a review from a team as a code owner July 11, 2024 20:17

torben-hansen reviewed Jul 11, 2024

View reviewed changes

crypto/fipsmodule/sha/internal.h Show resolved Hide resolved

skmcgrail reviewed Jul 17, 2024

View reviewed changes

include/openssl/hmac.h Show resolved Hide resolved

Add hmac.errordata

4ec5b37

`crypto/err/hmac.errordata` generated using `go run ./util/make_errors.go hmac` CMakeLists file updated to add the errordata file.

fabrice102 dismissed nebeid’s stale review via 4ec5b37 July 17, 2024 19:52

skmcgrail and others added 2 commits July 17, 2024 21:25

python3 ./util/generate_build_files.py

34e5089

Merge branch 'main' into hmac-precompute

f096cfb

nebeid reviewed Jul 18, 2024

View reviewed changes

crypto/fipsmodule/sha/internal.h Outdated Show resolved Hide resolved

crypto/fipsmodule/hmac/hmac.c Show resolved Hide resolved

Fix Windows ARM64 compilation + comment improvements

cd102d6

From review from PR aws#1574

skmcgrail approved these changes Jul 19, 2024

View reviewed changes

nebeid added 2 commits July 19, 2024 13:45

Merge branch 'main' into hmac-precompute

1fd75d9

Merge branch 'main' into hmac-precompute

d77ee73

nebeid approved these changes Jul 19, 2024

View reviewed changes

nebeid merged commit 98ccf4a into aws:main Jul 19, 2024
101 of 103 checks passed

pennyannn mentioned this pull request Jul 19, 2024

Flip back src submodule to be main of aws-lc awslabs/aws-lc-verification#158

Merged

skmcgrail mentioned this pull request Aug 1, 2024

Prepare Release for v1.33.0 #1734

Merged

This was referenced Aug 1, 2024

Update aws-lc-sys to AWS-LC v1.33.0 aws/aws-lc-rs#479

Closed

Update aws-lc-sys/aws-lc to AWS-LC v1.33.0 aws/aws-lc-rs#480

Merged

fabrice102 mentioned this pull request Aug 7, 2024

Hmac with precomputed key corretto/amazon-corretto-crypto-provider#396

Open

fabrice102 pushed a commit to fabrice102/amazon-corretto-crypto-provider that referenced this pull request Aug 28, 2024

Support for HMAC Precomputed Key (only for non-FIPS builds)

af3f66e

This commit adds support for HMAC Precomputed Keys in ACCP. See aws/aws-lc#1574

fabrice102 pushed a commit to fabrice102/amazon-corretto-crypto-provider that referenced this pull request Aug 28, 2024

Support for HMAC Precomputed Key (only for non-FIPS builds)

881ffef

This commit adds support for HMAC Precomputed Keys in ACCP. See aws/aws-lc#1574

fabrice102 pushed a commit to fabrice102/amazon-corretto-crypto-provider that referenced this pull request Aug 28, 2024

Support for HMAC Precomputed Key (only for non-FIPS builds)

50bcab8

This commit adds support for HMAC Precomputed Keys in ACCP. See aws/aws-lc#1574

fabrice102 pushed a commit to fabrice102/amazon-corretto-crypto-provider that referenced this pull request Aug 28, 2024

Support for HMAC Precomputed Key (only for non-FIPS builds)

39e6205

This commit adds support for HMAC Precomputed Keys in ACCP. See aws/aws-lc#1574

fabrice102 pushed a commit to fabrice102/amazon-corretto-crypto-provider that referenced this pull request Aug 28, 2024

Support for HMAC Precomputed Key (only for non-FIPS builds)

8097f53

This commit adds support for HMAC Precomputed Keys in ACCP. See aws/aws-lc#1574

lilpoozie2005 approved these changes Sep 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Design for support of HMAC precomputed keys #1574

Design for support of HMAC precomputed keys #1574

fabrice102 commented May 2, 2024 •

edited

Loading

nebeid left a comment

nebeid left a comment

nebeid left a comment

nebeid left a comment

nebeid Jun 12, 2024

Design for support of HMAC precomputed keys #1574

Design for support of HMAC precomputed keys #1574

Conversation

fabrice102 commented May 2, 2024 • edited Loading

Issues:

Description of changes:

Call-outs:

Testing:

nebeid left a comment

Choose a reason for hiding this comment

nebeid left a comment

Choose a reason for hiding this comment

nebeid left a comment

Choose a reason for hiding this comment

nebeid left a comment

Choose a reason for hiding this comment

nebeid Jun 12, 2024

Choose a reason for hiding this comment

fabrice102 commented May 2, 2024 •

edited

Loading