XMSS RFC 8391 Update #1858

mgierlings · 2019-03-15T10:26:01Z

Updates XMSS (draft status) to conform to RFC 8391

randombit · 2019-03-15T12:40:54Z

src/lib/pubkey/xmss/xmss_parameters.h

-         XMSS_SHAKE256_W16_H16 = 0x0b00000b,
-         XMSS_SHAKE256_W16_H20 = 0x0c00000c
+         XMSS_SHA2_10_256 = 0x00000001,
+         XMSS_SHA2_16_256 = 0x00000002,


So the format changed between draft-06 and RFC? :/

Is there any reasonable way we can continue supporting older keys/sigs? It seems this change would invalidate all existing uses.

At the very least we'll also want to use a different OID for keys so it is not possible to mix them.

Yes, unfortunately the OID format did change. There exists another, new draft status document of interest regarding XMSS Algorithm Identifiers for HSS and XMSS for Use in the Internet X.509 Public Key Infrastructure. Also somewhere down the road XMSS will likely be assigned an "official" OID. So this may not be the last incompatible change to XMSS we see. There is also a disclaimer in RFC 8391:

This document is not an Internet Standards Track specification; it is
published for informational purposes.

This document is a product of the Internet Research Task Force
(IRTF). The IRTF publishes the results of Internet-related research
and development activities. These results might not be suitable for
deployment. This RFC represents the consensus of the Crypto Forum
Research Group of the Internet Research Task Force (IRTF). Documents
approved for publication by the IRSG are not candidates for any level
of Internet Standard; see Section 2 of RFC 7841.

Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc8391.

A possibility to support older formats would be to integrate a key conversion utility into the botan cli.

See #1858

randombit · 2019-03-15T12:58:16Z

I sent out a warning letting anyone who is actively using current draft-06 support to let us know https://lists.randombit.net/pipermail/botan-devel/2019-March/002278.html and added a warning to the 2.10 release notes. If we don't hear from anyone by say May 1st, I say go ahead and merge this.

In the mean time we should still work to minimize disturbance:

Increase XMSS datestamp in info.txt
Change to a new OID arc (update also doc/oids.txt)
Maybe even change BOTAN_HAS_XMSS to BOTAN_HAS_XMSS_RFC8391?

randombit · 2019-04-07T13:34:25Z

Haven't heard from anyone about this so likely this is good to merge, but lets wait until end of the month just to be sure.

randombit · 2019-04-19T00:53:53Z

@mgierlings Can you rebase to address the merge conflict?

mgierlings · 2019-04-19T08:34:52Z

@randombit Will do. I also have another update to this PR in the pipeline, which I'll push soon.

randombit · 2019-04-22T14:07:16Z

It looks like you merged master into your branch rather than rebasing, this makes history very hard to read. Please rebase instead.

Changes XMSS and XMSS WOTS algorithm names and OIDs to correspond to RFC 8391.

- Replaces XMSS test vectors with new vectors that were generated using Bouncy Castle's XMSS implementation. - Adjusts the XMSS test bench to recognize the new XMSS algorithm naming scheme.

Internally XMSS uses a 64 Bit type for the leaf index. This patch removes the four leading zero bytes from the XMSS leaf index and serializes it as a four byte value as described in RFC 8391. Test cases are adjusted accordingly. The 64 Bit type is kept internally which potentially allows for code reuse when implementing XMSS^MT on top of the current XMSS code.

mgierlings · 2019-05-09T14:35:03Z

@randombit The broken commit should be fixed. I'm not sure though what causes the softhsm error in some of the Travis CI builds.

randombit · 2019-05-09T14:40:39Z

@mgierlings Thanks. I think the problem is bench_xmss in speed.cpp needs to be updated for the new param names.

randombit · 2019-05-10T14:42:13Z

I don't think any change in performance was expected either way, but ftr I did check master vs this branch, they had only insignificant differences in sign and verify time.

codecov-io · 2019-05-10T15:29:12Z

Codecov Report

Merging #1858 into master will decrease coverage by <.01%.
The diff coverage is 69.23%.

@@            Coverage Diff             @@
##           master    #1858      +/-   ##
==========================================
- Coverage   91.38%   91.37%   -0.01%     
==========================================
  Files         531      532       +1     
  Lines       56462    56556      +94     
  Branches     5877     5882       +5     
==========================================
+ Hits        51596    51677      +81     
- Misses       4866     4879      +13

Impacted Files	Coverage Δ
src/lib/pubkey/xmss/xmss_publickey.cpp	`94.11% <ø> (ø)`	⬆️
src/lib/pubkey/xmss/xmss_signature_operation.cpp	`100% <ø> (ø)`	⬆️
src/lib/pubkey/xmss/xmss_wots_publickey.cpp	`100% <ø> (ø)`	⬆️
src/cli/speed.cpp	`90.75% <ø> (ø)`	⬆️
src/lib/pubkey/xmss/xmss_signature.cpp	`100% <100%> (ø)`	⬆️
src/lib/pubkey/pk_algs.cpp	`94.81% <100%> (ø)`	⬆️
src/lib/pubkey/xmss/xmss_privatekey.cpp	`95.74% <100%> (ø)`	⬆️
src/lib/pubkey/xmss/xmss_wots_parameters.cpp	`86.11% <50%> (ø)`	⬆️
src/tests/test_xmss.cpp	`93.02% <50%> (ø)`	⬆️
src/lib/pubkey/xmss/xmss_parameters.cpp	`90% <69.44%> (ø)`	⬆️
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c9caac...e390adc. Read the comment docs.

randombit · 2019-05-10T17:21:48Z

@mgierlings I fixed the test issue in 10630cc. Is this PR ready to merge?

mgierlings · 2019-05-10T21:54:17Z

@randombit Looks good to me! One remark though on the XMSS parameter name changes introduced with 10630cc: The RFC actually uses "-" between XMSS and the hash function name, see RFC 8391 Section 5.3 XMSS Parameters. By replacing the dash with an underscore we technically use non standard compliant algorithm names.

securitykernel · 2019-05-12T06:15:05Z

@mgierlings BouncyCastle uses the same naming scheme, maybe this is a thing? Anyway, whatever we decide here, we should probably make it consistent in both libraries.

mgierlings · 2019-05-12T07:50:35Z

@securitykernel Agreed, we should be consistent. If others start implementing XMSS, they will likely use names from the standard document. The cleanest solution would probably be to adhere to the RFC's naming scheme (in both BouncyCastle and Botan). If we come up with our own now we might need to revisit this and change naming again in the future.

randombit · 2019-05-12T12:26:05Z

I renamed the params because that's what the patched handbook uses for the param names. I find combining hyphen and underscore really awkward. But if that is the standard naming scheme, probably best to follow it.

mgierlings · 2019-05-12T12:44:32Z

@randombit The algorithm names in the handbook are supposed to correspond to the xmss_algorithm_t enum items, since this is what is passed as argument in the code snippet below. However I did omit the bulky prefix Botan::XMSS_Parameters::xmss_algorithm_t, as a result the algorithm list in the handbook is somewhat ambiguous. To be more clear, we could change the handbook to list the available algorithms as follows:

Botan::XMSS_Parameters::xmss_algorithm_t::XMSS_SHA2_10_256 ("XMSS-SHA2_10_256")
Botan::XMSS_Parameters::xmss_algorithm_t::XMSS_SHA2_16_256 ("XMSS-SHA2_16_256")
Botan::XMSS_Parameters::xmss_algorithm_t::XMSS_SHA2_20_256 ("XMSS-SHA2_20_256")
...

or

"XMSS-SHA2_10_256" (Botan::XMSS_Parameters::xmss_algorithm_t::XMSS_SHA2_10_256)
"XMSS-SHA2_16_256" (Botan::XMSS_Parameters::xmss_algorithm_t::XMSS_SHA2_16_256)
"XMSS-SHA2_20_256" (Botan::XMSS_Parameters::xmss_algorithm_t::XMSS_SHA2_20_256)
...

randombit · 2019-05-24T08:10:34Z

I switched param names back. So I think this is now good to merge? It's been almost two months since the notification went out warning about the change and it looks like nobody sees an issue with it.

mgierlings · 2019-05-24T21:05:17Z

@randombit Yes, I'd say we're good to merge.

randombit reviewed Mar 15, 2019

View reviewed changes

randombit added a commit that referenced this pull request Mar 15, 2019

Add a warning about XMSS changes

bb19424

See #1858

securitykernel mentioned this pull request Mar 25, 2019

Update XMSS OIDs bcgit/bc-java#482

Closed

randombit mentioned this pull request Apr 7, 2019

2.11.0 Release #1882

Closed

18 tasks

randombit added the changes required label May 1, 2019

mgierlings force-pushed the xmss_rfc8391 branch from 91983bc to 0c6a107 Compare May 5, 2019 08:40

mgierlings added 6 commits May 6, 2019 10:00

Updates XMSS parameters corresponding to RFC 8391

5993c9e

Changes XMSS and XMSS WOTS algorithm names and OIDs to correspond to RFC 8391.

Updates references to XMSS the standard document

4818ae0

Updates XMSS manual entry

80a831d

Updates copyright notices

51f4e5c

Adds new test cases

b1976eb

- Replaces XMSS test vectors with new vectors that were generated using Bouncy Castle's XMSS implementation. - Adjusts the XMSS test bench to recognize the new XMSS algorithm naming scheme.

mgierlings force-pushed the xmss_rfc8391 branch from 0c6a107 to 6bbccff Compare May 6, 2019 08:05

Fix param names

10630cc

randombit removed the changes required label May 10, 2019

Fix param names to match RFC

e390adc

randombit merged commit e390adc into randombit:master May 24, 2019

randombit added a commit that referenced this pull request May 24, 2019

Merge GH #1858 Update XMSS to conform to RFC 8391

7de8754

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

XMSS RFC 8391 Update #1858

XMSS RFC 8391 Update #1858

mgierlings commented Mar 15, 2019

randombit Mar 15, 2019 •

edited

Loading

mgierlings Mar 15, 2019

randombit commented Mar 15, 2019

randombit commented Apr 7, 2019

randombit commented Apr 19, 2019

mgierlings commented Apr 19, 2019

randombit commented Apr 22, 2019

mgierlings commented May 9, 2019

randombit commented May 9, 2019

randombit commented May 10, 2019

codecov-io commented May 10, 2019 •

edited

Loading

randombit commented May 10, 2019

mgierlings commented May 10, 2019

securitykernel commented May 12, 2019

mgierlings commented May 12, 2019

randombit commented May 12, 2019

mgierlings commented May 12, 2019 •

edited

Loading

randombit commented May 24, 2019

mgierlings commented May 24, 2019

XMSS RFC 8391 Update #1858

XMSS RFC 8391 Update #1858

Conversation

mgierlings commented Mar 15, 2019

randombit Mar 15, 2019 • edited Loading

Choose a reason for hiding this comment

mgierlings Mar 15, 2019

Choose a reason for hiding this comment

randombit commented Mar 15, 2019

randombit commented Apr 7, 2019

randombit commented Apr 19, 2019

mgierlings commented Apr 19, 2019

randombit commented Apr 22, 2019

mgierlings commented May 9, 2019

randombit commented May 9, 2019

randombit commented May 10, 2019

codecov-io commented May 10, 2019 • edited Loading

Codecov Report

randombit commented May 10, 2019

mgierlings commented May 10, 2019

securitykernel commented May 12, 2019

mgierlings commented May 12, 2019

randombit commented May 12, 2019

mgierlings commented May 12, 2019 • edited Loading

randombit commented May 24, 2019

mgierlings commented May 24, 2019

randombit Mar 15, 2019 •

edited

Loading

codecov-io commented May 10, 2019 •

edited

Loading

mgierlings commented May 12, 2019 •

edited

Loading