KMAC AFT tests seem incorrect

[dghgit]

environment
Demo

testSessionId
118292

vsId
340784,340785

Algorithm registration
[{"acvVersion":"1.0"},{"isSample":true,"algorithms":[{"algorithm":"KMAC-128","revision":"1.0","hexCustomization":true,"xof":[true,false],"msgLen":[{"min":0,"max":1024,"increment":8}],"keyLen":[{"min":128,"max":1024,"increment":8}],"macLen":[{"min":32,"max":512,"increment":8}]},{"algorithm":"KMAC-256","revision":"1.0","hexCustomization":false,"xof":[true,false],"msgLen":[{"min":0,"max":65536,"increment":8}],"keyLen":[{"min":128,"max":4096,"increment":8}],"macLen":[{"min":32,"max":4096,"increment":8}]}]}]

Endpoint in which the error is experienced
https://demo.acvts.nist.gov:443/acvp/v1 GET

Expected behavior
I think I expect the AFT test responses I'm generating to pass. The MVT tests now pass (fully) for both KMAC-128 and KMAC-256, however the AFT tests universally fail. I've tried swapping parameters and ignoring some, but I do not seem to be able to reproduce any of the AFT sample responses. The same code generating them is generating the MVT test results, it almost seems like the setup of the KMAC function in the server has an issue.

Additional context
A simple test vector for KMAC-128,
xof=true,
msg="",
customization="",
key="",
macLen=256,
mac="3f9259e80b35e0719c26025f7e38a4a38172bf1142a6a9c1930e50df03904312"

for KMAC-256 the same inputs produce:

mac="2c9683c318165466c0d3f9467ce77f0cea513f643ae3bd5b0969165aafae3f71"

[Kritner]

Thanks @dghgit, we agree on the above tests. Looks like there's something going on with the test case "set up" for AFT, rather than the crypto portion itself.

[Kritner]

I know we covered some customization string issue that was impacting KMAC from issue usnistgov/ACVP#904 - looking through the issue though I'm not actually seeing if you signed off on the KMAC fix. Previously I guess everything was failing under KMAC, but I guess even after the customization fix the AFT tests still weren't passing?

Looking at what I believe is the issue, I can see why the MVTs would be passing but not the AFT, I'll put in a change.

[Kritner]

A change has been pushed out https://github.com/usnistgov/ACVP-Server/releases/tag/v1.1.0.12-1 that would at a minimum fix the problem I found with the AFT test generation.

Not sure if that would have been the entirety of the problem, but let me know how it goes, thanks!

[dghgit]

Getting 500 errors when trying to connect. This might have to wait till tomorrow.

[Kritner]

Thanks @dghgit, tracking that issue in usnistgov/ACVP#1035, sorry about that

[dghgit]

Okay, so it is looking better. Still seeing some errors though.

I've discovered 2 things:

1. If I have hexCustomization set to true I see more errors, they don't seem to be consistent.

2. If I have hexCustomization false and limit the length of the messages to 4096 everything works. If I up the message length to 65536 I see one or two failures in the AFT tests for both types of KMAC.

Here's a sample for KMAC-256,

msg = "205EF7EBD63E28D5F0C10A68E97CF6B92167D7F682A0FE7D04E9A4DCB707757A289075BBF7E1D692B09D4F22A05D1F6FC86CE1DCAF4120961AEECDADD0D8723FE4ABC508EBBE2602438738A336186BD19675B42A475926EE9861791DE41A19E841853B0C2EB6E47D972835D630B9F7940CB1C8EAC1B3C51FB7F4BFD03CBD2A795D685D2EAC78835F5F75DD0C79DDE25731B5DB0F9477509EB4E784B11544F37435AEB31211294AE7C05D61115179073BA6D4142F325F28117B6E57F65F44BCC0872F868018892C46E50E6FBD88CB0C0BDEE3F6D1047AA4BB2FCC9C3D6F2FF52F1AE9DD5E85E61F9CA619AAD71B58DD17ACC7255574D6AA658E0B673825D25F8FE58A1665E3B8D304396AA8AFE032C79704C210CFBEDBBA304FC11759EA58064EC88A0AE2EFE2D9BC97E4302E07FA26A56452912C212819EF6D0548BC756AE86A98AFA907D82CE996EE31F8D8CB2736977E8CC3D3C39FFE7291E20ED978C8EE8FBCFC563D92A432282654CC1F9AABD27CDFC08464FAA566EEED3BAC0A550FEABA5193839DA3155802F0BE78298EEBAD3DEC64E8628240223F99F953EAA723E79E7D5A0B48E9BFE0C82307A41DE4FFE84026532D3CAE67155DDB6C5C83DE7C0A95DC5D29762CD3911AA097CA989DFEACF67D3167B2B8C3FCA6D1371C7AD9805FEEA0411B90F565718D06E500AB0EE694F7E735043D6AABD7F8EE1EDC9E6197FFA042D97F31F6932B826FD15D8FBBEBC5AED57D12D921A2C71F73122421E8E15E6475BB96D4956A75F90F26DF32CA4A0A1240C57FC0FB755CD8AD149C6B8C3BA84E0328DAC6FD0673553D0E4E7E7A7534107EB05C9030628C4F8194DA08BFB3075EA45AA7FE5B88C39D55AF2288C5DE6C881D5C1F94994DA9AB27354EC1A3C9046096135CE7FE516C9D45F6DED90DB385D0557ED1C4139E710142A550C72508933489303C4AAFC206EE2032320B4FEC5177FD74CA37FD8D90D35E0CAA04408CCF78012DC55E2B824520B5CF51A371D00DECAA9054F58095614E8E67CDF4E2BCBE72D6A4DB5CD8FB08516A4BCC5FDC1E3A2C008DC1FEFEA61CDC8A9E5A33D4AF6B812E2EC2ACFC0E7A6B8E539C9D71ECEDE99BA1A44D3EECA09950D86B526797A406E83E4BC908D6CDA5DBFFEB9F3F6DB5F5F1C1CA79B653C07AF8CA82CAB3556A7AF2DB0B27E1DFAA661B5BA1FE669863D3A21EA8D21197E2F3652DA9B33CE8007CFD5137A34157149994DC10994725A7A17DBFB3B3D168CC297F4C7EF6783F422BF4E9230E39E19355D2793ADDC06140E32321F27F14F56520607085D4ADE713CEB9D1FA8709CAC3703EC7A837882C9A04212BDD8C0DFBFCEF702CD8FC51D11D7BB1C3095A66610F00AE007B11E4D64D386CD7A6CA62C492C32C61E919CB7170B9D98D0DD66507DBC7F63F509FBE28505946A36DC28BF609BE996E6241290640FF9F393A343093DCDA9932C16B832B318EA3121F0B700FF313C23D6A41E275FAB8ABD4CBF0B7F36C65AEDED17F09CBFC04B044B723BEEBD36B2AF1289CF98BB96BB1CB377F2FE1D25A35DEF6447B1E04EADA63CD9BE0AC83E253FBD2894DA144A5224D9D0EB6D6677DDF97CF42CE2767FC7B763E3E31DBB84FD9ED56975FB2F10521C56BE21A7F6118A2FE7799FB5CD0AA93313C3FA7666B8D30B88D17EF0A7B5730CA9F9FD1FA2EB7B5E5A49C69E907660546A2DA0922D0C0653CDD493616FE469AE18203C3D6F0350B45617726D0CA0E01C42EED501538B3CE6276C2303AFC175DCB6C34F7092B3433F2F041D59E23DBFE0E83332D5B1107157A0CEE0A2428FC8737164AABC91C85F75C49621D4E2DDB5181E65836834E731518FE148E23907D550D8F0279755CE567B9B77BCF2620138E9B119BA17D09163990BF08AE843EAED818893D62BB76631C88D6409A8703D421AD98CDD16896DE4BF1BEF0D8796D3DF95103D3FB41C12668E304385B5508F78198821CC13800DBEC847E23E2DF4721C03295A27F846CA3C95A05D2448263C914A0D0209A2D708FF4835868E48A816EB648BABF1F1D9852B81C4A32C8F45F44ED63AD634EFD6A7BF1B4C018D725C0138CCDC7A3C7502DC2041F1D04A96144B49B2F8EA49E0D3C7DEC820F4CDF9D71D621F3930CA9FA737B084B718B7A56FD0001A0B648FCE10720D9709E3ACB360E76AB985BA662455451778A86A52D16ED8BBDDB82ED141C6F7C9AF651FE4CEB068FF19FA268E2BADC848ECA656B5EA1162B6EEB26D1C5FF164D013D613EC9E21CDDF6E54744425C76E3AD48ED2A6A70BAFC341BE168C9FC7438E3E4C5589E89CC7BF5CFDC199E4A464F53EFCE3937FA31C696FFC7965E11ADD3FEE58E89ACC9D37B8124259C3750FBFCB13F7976CF721D726D712271044E9F1A6C14699947F88FCD669EAB556D9F37B871CCE4FE2B19E7687FD6813B9739192D7A923B6E096CC9963DA590B6DA9D48EFACB9AB1E7B1AACA7827306108FCC72F65486F2B58EE4721CEA6018D7C16ADB86E6110709306EB36C47CA537B89B4F7557C5121B5232F00B4C3DB8C2F86F471D5EF90C48722EF47753B15E58F9835F777909ECB3240CA16AB982AF877E4EF822D59EBB1812B88F44884618039DA99DEBC53701E0921CBC1A011F00F1737CBD7BB4FBF2977F24AE8AEA7EDFE016B279BA8413043BBABD1EC8263A3CB13369C358262EB61754220EC962CC9DA0D4D2544DD0346B36D4D39A7B0356B98DFE1147882FA4C27AF049EEEF59D91F5528221EDFB82AD28AF1F30CE8A9A651430158A27211C5C2F3B20C78A2BF6D39D1CC44215499CAD69093595DF2A8847778ECAE4C83BADB994976B8AA9ECB65D44FF5A56D212F4DF15DEC5D5DAB0F49AC6CAB440893AD2F87296540A29B28C2FDE1045D4C54E122BCF541F9B331F4FE8A1943C351B956E8AD41FA4E43DC3343579F9F52A99962E0200CFD6F8C5A618177A91CD5DE80528D33ADF6BF35112FBA54CE5414B36CA7A6FF1243B4469A86A3F13F5F73869811BBB452CF4130620273B143FA02FE62B0C71CA36DE03C16D002FF37214784E0FE4ED152877433B44A6CF28FCB269D75DB1286E45474D6E31E3BC703058ADF04F38C6A8788D98547BB711844D34A42A705F4B1F04C887D58D66A3C0CBBFF3562B1D5CA1DCD937A9D12D36CC1DDBD5D4A3882F23AED80DACD9A381AF366981B2842F5F220848B39F82F9F7127561780E16D9294D7177858B3F65E99078BAA3C83FAC2D96E9EBADCDD6A09A33634D12C333FA6AD932C15EB1C5490DB616893F1D2D5B5F8724AC0980B35C0ECF294AD44D1BFF19B8D8D0249F335242F36C602AE46FC47E389A2EC34CB342AA56609104073D7917406DAFE2AC93F9B5E8F9BCC25DB36E612D5212C733408768BA7DB899E93A3BB02EBEDD53D1F1D14A616104D41BFC270FF9"
key = "7B3E43EDA8D51EC181D37DDE5B17ECCDD8BE84C268DC6C950070085F48E98EFA97F400260888ED43A0BBAE5E674BD169631D048A22A6EE7B654B769618BE7D2F32E42A2D804FEADD49A6685758D1E17706211DC518C0DFC1228175F36636EA0813062F49C2D88B81E31D4A016711FDDB3FDCCD02C79036612EEC050CD0CDFB94A8CF68"
customization = "km}zS=U8={k/T m"
macLen = 1024
xof = true

I get:
"mac":"33e503b4201462a26a9d9246d906c712d043aca2971d827bf65183ec7ff48f094887f8c4a20466fb3fe3943434dedf74d90237311e34c899748b5f1380c3b3204bb54809e8fae2489b259bd7e594137319ef9009bc5f3b78cff954ba1ba1c5eba8ea3a7e07548b2b51e886b06ae4a68df79cf0fafcc750745967c8809f6f5c4d"
the sample file with the test indicated:
"mac":"8506FC9B1CF4B1DE588507B4AB8C5E9C4B1E09D4940B0F84C5F54368566E12978D4070908B4A7BE2F07A7C781F3E3D697E88AB6AC03673A7A3AF854C816ED2DC6F113D35FEA0C4A100300FA22F5017AE37E685E2FD3A187362A37AF6227D96F3B7A42D8F1B82C628927C10F2524CC4898BB2707832B4B3694566C26D5414E5B5"

Note, longer messages in the same test run passed. I think it's connected to the message length though (although there's obviously something going on with hexCustomisation which might be related), it seems particular long messages trip something over. I don't think it's anything I'm doing here as the failures are only in AFT, so we're in sync as far as MVT goes.

If it's any help the vector set URLS for the above run were:
"/acvp/v1/testSessions/118553/vectorSets/342665"
"/acvp/v1/testSessions/118553/vectorSets/342666"

[Kritner]

Looking through the vector sets it's tough to see the rhyme or reason to our disagreements, especially since it's only one or two per vector set, and with such large lengths.

To see if we can find out at what point in the process we disagree, I'm going to use the test case you posted in #14 (comment) and post some intermediate values with and without a customization (so using your original test case, and one modified without a customization to see if it could be around that).

---

newX

"01880204187B3E43EDA8D51EC181D37DDE5B17ECCDD8BE84C268DC6C950070085F48E98EFA97F400260888ED43A0BBAE5E674BD169631D048A22A6EE7B654B769618BE7D2F32E42A2D804FEADD49A6685758D1E17706211DC518C0DFC1228175F36636EA0813062F49C2D88B81E31D4A016711FDDB3FDCCD02C79036612EEC050CD0CDFB94A8CF68205EF7EBD63E28D5F0C10A68E97CF6B92167D7F682A0FE7D04E9A4DCB707757A289075BBF7E1D692B09D4F22A05D1F6FC86CE1DCAF4120961AEECDADD0D8723FE4ABC508EBBE2602438738A336186BD19675B42A475926EE9861791DE41A19E841853B0C2EB6E47D972835D630B9F7940CB1C8EAC1B3C51FB7F4BFD03CBD2A795D685D2EAC78835F5F75DD0C79DDE25731B5DB0F9477509EB4E784B11544F37435AEB31211294AE7C05D61115179073BA6D4142F325F28117B6E57F65F44BCC0872F868018892C46E50E6FBD88CB0C0BDEE3F6D1047AA4BB2FCC9C3D6F2FF52F1AE9DD5E85E61F9CA619AAD71B58DD17ACC7255574D6AA658E0B673825D25F8FE58A1665E3B8D304396AA8AFE032C79704C210CFBEDBBA304FC11759EA58064EC88A0AE2EFE2D9BC97E4302E07FA26A56452912C212819EF6D0548BC756AE86A98AFA907D82CE996EE31F8D8CB2736977E8CC3D3C39FFE7291E20ED978C8EE8FBCFC563D92A432282654CC1F9AABD27CDFC08464FAA566EEED3BAC0A550FEABA5193839DA3155802F0BE78298EEBAD3DEC64E8628240223F99F953EAA723E79E7D5A0B48E9BFE0C82307A41DE4FFE84026532D3CAE67155DDB6C5C83DE7C0A95DC5D29762CD3911AA097CA989DFEACF67D3167B2B8C3FCA6D1371C7AD9805FEEA0411B90F565718D06E500AB0EE694F7E735043D6AABD7F8EE1EDC9E6197FFA042D97F31F6932B826FD15D8FBBEBC5AED57D12D921A2C71F73122421E8E15E6475BB96D4956A75F90F26DF32CA4A0A1240C57FC0FB755CD8AD149C6B8C3BA84E0328DAC6FD0673553D0E4E7E7A7534107EB05C9030628C4F8194DA08BFB3075EA45AA7FE5B88C39D55AF2288C5DE6C881D5C1F94994DA9AB27354EC1A3C9046096135CE7FE516C9D45F6DED90DB385D0557ED1C4139E710142A550C72508933489303C4AAFC206EE2032320B4FEC5177FD74CA37FD8D90D35E0CAA04408CCF78012DC55E2B824520B5CF51A371D00DECAA9054F58095614E8E67CDF4E2BCBE72D6A4DB5CD8FB08516A4BCC5FDC1E3A2C008DC1FEFEA61CDC8A9E5A33D4AF6B812E2EC2ACFC0E7A6B8E539C9D71ECEDE99BA1A44D3EECA09950D86B526797A406E83E4BC908D6CDA5DBFFEB9F3F6DB5F5F1C1CA79B653C07AF8CA82CAB3556A7AF2DB0B27E1DFAA661B5BA1FE669863D3A21EA8D21197E2F3652DA9B33CE8007CFD5137A34157149994DC10994725A7A17DBFB3B3D168CC297F4C7EF6783F422BF4E9230E39E19355D2793ADDC06140E32321F27F14F56520607085D4ADE713CEB9D1FA8709CAC3703EC7A837882C9A04212BDD8C0DFBFCEF702CD8FC51D11D7BB1C3095A66610F00AE007B11E4D64D386CD7A6CA62C492C32C61E919CB7170B9D98D0DD66507DBC7F63F509FBE28505946A36DC28BF609BE996E6241290640FF9F393A343093DCDA9932C16B832B318EA3121F0B700FF313C23D6A41E275FAB8ABD4CBF0B7F36C65AEDED17F09CBFC04B044B723BEEBD36B2AF1289CF98BB96BB1CB377F2FE1D25A35DEF6447B1E04EADA63CD9BE0AC83E253FBD2894DA144A5224D9D0EB6D6677DDF97CF42CE2767FC7B763E3E31DBB84FD9ED56975FB2F10521C56BE21A7F6118A2FE7799FB5CD0AA93313C3FA7666B8D30B88D17EF0A7B5730CA9F9FD1FA2EB7B5E5A49C69E907660546A2DA0922D0C0653CDD493616FE469AE18203C3D6F0350B45617726D0CA0E01C42EED501538B3CE6276C2303AFC175DCB6C34F7092B3433F2F041D59E23DBFE0E83332D5B1107157A0CEE0A2428FC8737164AABC91C85F75C49621D4E2DDB5181E65836834E731518FE148E23907D550D8F0279755CE567B9B77BCF2620138E9B119BA17D09163990BF08AE843EAED818893D62BB76631C88D6409A8703D421AD98CDD16896DE4BF1BEF0D8796D3DF95103D3FB41C12668E304385B5508F78198821CC13800DBEC847E23E2DF4721C03295A27F846CA3C95A05D2448263C914A0D0209A2D708FF4835868E48A816EB648BABF1F1D9852B81C4A32C8F45F44ED63AD634EFD6A7BF1B4C018D725C0138CCDC7A3C7502DC2041F1D04A96144B49B2F8EA49E0D3C7DEC820F4CDF9D71D621F3930CA9FA737B084B718B7A56FD0001A0B648FCE10720D9709E3ACB360E76AB985BA662455451778A86A52D16ED8BBDDB82ED141C6F7C9AF651FE4CEB068FF19FA268E2BADC848ECA656B5EA1162B6EEB26D1C5FF164D013D613EC9E21CDDF6E54744425C76E3AD48ED2A6A70BAFC341BE168C9FC7438E3E4C5589E89CC7BF5CFDC199E4A464F53EFCE3937FA31C696FFC7965E11ADD3FEE58E89ACC9D37B8124259C3750FBFCB13F7976CF721D726D712271044E9F1A6C14699947F88FCD669EAB556D9F37B871CCE4FE2B19E7687FD6813B9739192D7A923B6E096CC9963DA590B6DA9D48EFACB9AB1E7B1AACA7827306108FCC72F65486F2B58EE4721CEA6018D7C16ADB86E6110709306EB36C47CA537B89B4F7557C5121B5232F00B4C3DB8C2F86F471D5EF90C48722EF47753B15E58F9835F777909ECB3240CA16AB982AF877E4EF822D59EBB1812B88F44884618039DA99DEBC53701E0921CBC1A011F00F1737CBD7BB4FBF2977F24AE8AEA7EDFE016B279BA8413043BBABD1EC8263A3CB13369C358262EB61754220EC962CC9DA0D4D2544DD0346B36D4D39A7B0356B98DFE1147882FA4C27AF049EEEF59D91F5528221EDFB82AD28AF1F30CE8A9A651430158A27211C5C2F3B20C78A2BF6D39D1CC44215499CAD69093595DF2A8847778ECAE4C83BADB994976B8AA9ECB65D44FF5A56D212F4DF15DEC5D5DAB0F49AC6CAB440893AD2F87296540A29B28C2FDE1045D4C54E122BCF541F9B331F4FE8A1943C351B956E8AD41FA4E43DC3343579F9F52A99962E0200CFD6F8C5A618177A91CD5DE80528D33ADF6BF35112FBA54CE5414B36CA7A6FF1243B4469A86A3F13F5F73869811BBB452CF4130620273B143FA02FE62B0C71CA36DE03C16D002FF37214784E0FE4ED152877433B44A6CF28FCB269D75DB1286E45474D6E31E3BC703058ADF04F38C6A8788D98547BB711844D34A42A705F4B1F04C887D58D66A3C0CBBFF3562B1D5CA1DCD937A9D12D36CC1DDBD5D4A3882F23AED80DACD9A381AF366981B2842F5F220848B39F82F9F7127561780E16D9294D7177858B3F65E99078BAA3C83FAC2D96E9EBADCDD6A09A33634D12C333FA6AD932C15EB1C5490DB616893F1D2D5B5F8724AC0980B35C0ECF294AD44D1BFF19B8D8D0249F335242F36C602AE46FC47E389A2EC34CB342AA56609104073D7917406DAFE2AC93F9B5E8F9BCC25DB36E612D5212C733408768BA7DB899E93A3BB02EBEDD53D1F1D14A616104D41BFC270FF90001"

newX with || separating "pieces"

"01||
88||
02||
0418||
7B3E43EDA8D51EC181D37DDE5B17ECCDD8BE84C268DC6C950070085F48E98EFA97F400260888ED43A0BBAE5E674BD169631D048A22A6EE7B654B769618BE7D2F32E42A2D804FEADD49A6685758D1E17706211DC518C0DFC1228175F36636EA0813062F49C2D88B81E31D4A016711FDDB3FDCCD02C79036612EEC050CD0CDFB94A8CF68||
205EF7EBD63E28D5F0C10A68E97CF6B92167D7F682A0FE7D04E9A4DCB707757A289075BBF7E1D692B09D4F22A05D1F6FC86CE1DCAF4120961AEECDADD0D8723FE4ABC508EBBE2602438738A336186BD19675B42A475926EE9861791DE41A19E841853B0C2EB6E47D972835D630B9F7940CB1C8EAC1B3C51FB7F4BFD03CBD2A795D685D2EAC78835F5F75DD0C79DDE25731B5DB0F9477509EB4E784B11544F37435AEB31211294AE7C05D61115179073BA6D4142F325F28117B6E57F65F44BCC0872F868018892C46E50E6FBD88CB0C0BDEE3F6D1047AA4BB2FCC9C3D6F2FF52F1AE9DD5E85E61F9CA619AAD71B58DD17ACC7255574D6AA658E0B673825D25F8FE58A1665E3B8D304396AA8AFE032C79704C210CFBEDBBA304FC11759EA58064EC88A0AE2EFE2D9BC97E4302E07FA26A56452912C212819EF6D0548BC756AE86A98AFA907D82CE996EE31F8D8CB2736977E8CC3D3C39FFE7291E20ED978C8EE8FBCFC563D92A432282654CC1F9AABD27CDFC08464FAA566EEED3BAC0A550FEABA5193839DA3155802F0BE78298EEBAD3DEC64E8628240223F99F953EAA723E79E7D5A0B48E9BFE0C82307A41DE4FFE84026532D3CAE67155DDB6C5C83DE7C0A95DC5D29762CD3911AA097CA989DFEACF67D3167B2B8C3FCA6D1371C7AD9805FEEA0411B90F565718D06E500AB0EE694F7E735043D6AABD7F8EE1EDC9E6197FFA042D97F31F6932B826FD15D8FBBEBC5AED57D12D921A2C71F73122421E8E15E6475BB96D4956A75F90F26DF32CA4A0A1240C57FC0FB755CD8AD149C6B8C3BA84E0328DAC6FD0673553D0E4E7E7A7534107EB05C9030628C4F8194DA08BFB3075EA45AA7FE5B88C39D55AF2288C5DE6C881D5C1F94994DA9AB27354EC1A3C9046096135CE7FE516C9D45F6DED90DB385D0557ED1C4139E710142A550C72508933489303C4AAFC206EE2032320B4FEC5177FD74CA37FD8D90D35E0CAA04408CCF78012DC55E2B824520B5CF51A371D00DECAA9054F58095614E8E67CDF4E2BCBE72D6A4DB5CD8FB08516A4BCC5FDC1E3A2C008DC1FEFEA61CDC8A9E5A33D4AF6B812E2EC2ACFC0E7A6B8E539C9D71ECEDE99BA1A44D3EECA09950D86B526797A406E83E4BC908D6CDA5DBFFEB9F3F6DB5F5F1C1CA79B653C07AF8CA82CAB3556A7AF2DB0B27E1DFAA661B5BA1FE669863D3A21EA8D21197E2F3652DA9B33CE8007CFD5137A34157149994DC10994725A7A17DBFB3B3D168CC297F4C7EF6783F422BF4E9230E39E19355D2793ADDC06140E32321F27F14F56520607085D4ADE713CEB9D1FA8709CAC3703EC7A837882C9A04212BDD8C0DFBFCEF702CD8FC51D11D7BB1C3095A66610F00AE007B11E4D64D386CD7A6CA62C492C32C61E919CB7170B9D98D0DD66507DBC7F63F509FBE28505946A36DC28BF609BE996E6241290640FF9F393A343093DCDA9932C16B832B318EA3121F0B700FF313C23D6A41E275FAB8ABD4CBF0B7F36C65AEDED17F09CBFC04B044B723BEEBD36B2AF1289CF98BB96BB1CB377F2FE1D25A35DEF6447B1E04EADA63CD9BE0AC83E253FBD2894DA144A5224D9D0EB6D6677DDF97CF42CE2767FC7B763E3E31DBB84FD9ED56975FB2F10521C56BE21A7F6118A2FE7799FB5CD0AA93313C3FA7666B8D30B88D17EF0A7B5730CA9F9FD1FA2EB7B5E5A49C69E907660546A2DA0922D0C0653CDD493616FE469AE18203C3D6F0350B45617726D0CA0E01C42EED501538B3CE6276C2303AFC175DCB6C34F7092B3433F2F041D59E23DBFE0E83332D5B1107157A0CEE0A2428FC8737164AABC91C85F75C49621D4E2DDB5181E65836834E731518FE148E23907D550D8F0279755CE567B9B77BCF2620138E9B119BA17D09163990BF08AE843EAED818893D62BB76631C88D6409A8703D421AD98CDD16896DE4BF1BEF0D8796D3DF95103D3FB41C12668E304385B5508F78198821CC13800DBEC847E23E2DF4721C03295A27F846CA3C95A05D2448263C914A0D0209A2D708FF4835868E48A816EB648BABF1F1D9852B81C4A32C8F45F44ED63AD634EFD6A7BF1B4C018D725C0138CCDC7A3C7502DC2041F1D04A96144B49B2F8EA49E0D3C7DEC820F4CDF9D71D621F3930CA9FA737B084B718B7A56FD0001A0B648FCE10720D9709E3ACB360E76AB985BA662455451778A86A52D16ED8BBDDB82ED141C6F7C9AF651FE4CEB068FF19FA268E2BADC848ECA656B5EA1162B6EEB26D1C5FF164D013D613EC9E21CDDF6E54744425C76E3AD48ED2A6A70BAFC341BE168C9FC7438E3E4C5589E89CC7BF5CFDC199E4A464F53EFCE3937FA31C696FFC7965E11ADD3FEE58E89ACC9D37B8124259C3750FBFCB13F7976CF721D726D712271044E9F1A6C14699947F88FCD669EAB556D9F37B871CCE4FE2B19E7687FD6813B9739192D7A923B6E096CC9963DA590B6DA9D48EFACB9AB1E7B1AACA7827306108FCC72F65486F2B58EE4721CEA6018D7C16ADB86E6110709306EB36C47CA537B89B4F7557C5121B5232F00B4C3DB8C2F86F471D5EF90C48722EF47753B15E58F9835F777909ECB3240CA16AB982AF877E4EF822D59EBB1812B88F44884618039DA99DEBC53701E0921CBC1A011F00F1737CBD7BB4FBF2977F24AE8AEA7EDFE016B279BA8413043BBABD1EC8263A3CB13369C358262EB61754220EC962CC9DA0D4D2544DD0346B36D4D39A7B0356B98DFE1147882FA4C27AF049EEEF59D91F5528221EDFB82AD28AF1F30CE8A9A651430158A27211C5C2F3B20C78A2BF6D39D1CC44215499CAD69093595DF2A8847778ECAE4C83BADB994976B8AA9ECB65D44FF5A56D212F4DF15DEC5D5DAB0F49AC6CAB440893AD2F87296540A29B28C2FDE1045D4C54E122BCF541F9B331F4FE8A1943C351B956E8AD41FA4E43DC3343579F9F52A99962E0200CFD6F8C5A618177A91CD5DE80528D33ADF6BF35112FBA54CE5414B36CA7A6FF1243B4469A86A3F13F5F73869811BBB452CF4130620273B143FA02FE62B0C71CA36DE03C16D002FF37214784E0FE4ED152877433B44A6CF28FCB269D75DB1286E45474D6E31E3BC703058ADF04F38C6A8788D98547BB711844D34A42A705F4B1F04C887D58D66A3C0CBBFF3562B1D5CA1DCD937A9D12D36CC1DDBD5D4A3882F23AED80DACD9A381AF366981B2842F5F220848B39F82F9F7127561780E16D9294D7177858B3F65E99078BAA3C83FAC2D96E9EBADCDD6A09A33634D12C333FA6AD932C15EB1C5490DB616893F1D2D5B5F8724AC0980B35C0ECF294AD44D1BFF19B8D8D0249F335242F36C602AE46FC47E389A2EC34CB342AA56609104073D7917406DAFE2AC93F9B5E8F9BCC25DB36E612D5212C733408768BA7DB899E93A3BB02EBEDD53D1F1D14A616104D41BFC270FF9||
00||
01"

customization:

"km}zS=U8={k/T m"

customization as hex:

"6B6D7D7A533D55383D7B6B2F54206D"

call to cshake:

var result = cshake(
  // newX 
 
  "01880204187B3E43EDA8D51EC181D37DDE5B17ECCDD8BE84C268DC6C950070085F48E98EFA97F400260888ED43A0BBAE5E674BD169631D048A22A6EE7B654B769618BE7D2F32E42A2D804FEADD49A6685758D1E17706211DC518C0DFC1228175F36636EA0813062F49C2D88B81E31D4A016711FDDB3FDCCD02C79036612EEC050CD0CDFB94A8CF68205EF7EBD63E28D5F0C10A68E97CF6B92167D7F682A0FE7D04E9A4DCB707757A289075BBF7E1D692B09D4F22A05D1F6FC86CE1DCAF4120961AEECDADD0D8723FE4ABC508EBBE2602438738A336186BD19675B42A475926EE9861791DE41A19E841853B0C2EB6E47D972835D630B9F7940CB1C8EAC1B3C51FB7F4BFD03CBD2A795D685D2EAC78835F5F75DD0C79DDE25731B5DB0F9477509EB4E784B11544F37435AEB31211294AE7C05D61115179073BA6D4142F325F28117B6E57F65F44BCC0872F868018892C46E50E6FBD88CB0C0BDEE3F6D1047AA4BB2FCC9C3D6F2FF52F1AE9DD5E85E61F9CA619AAD71B58DD17ACC7255574D6AA658E0B673825D25F8FE58A1665E3B8D304396AA8AFE032C79704C210CFBEDBBA304FC11759EA58064EC88A0AE2EFE2D9BC97E4302E07FA26A56452912C212819EF6D0548BC756AE86A98AFA907D82CE996EE31F8D8CB2736977E8CC3D3C39FFE7291E20ED978C8EE8FBCFC563D92A432282654CC1F9AABD27CDFC08464FAA566EEED3BAC0A550FEABA5193839DA3155802F0BE78298EEBAD3DEC64E8628240223F99F953EAA723E79E7D5A0B48E9BFE0C82307A41DE4FFE84026532D3CAE67155DDB6C5C83DE7C0A95DC5D29762CD3911AA097CA989DFEACF67D3167B2B8C3FCA6D1371C7AD9805FEEA0411B90F565718D06E500AB0EE694F7E735043D6AABD7F8EE1EDC9E6197FFA042D97F31F6932B826FD15D8FBBEBC5AED57D12D921A2C71F73122421E8E15E6475BB96D4956A75F90F26DF32CA4A0A1240C57FC0FB755CD8AD149C6B8C3BA84E0328DAC6FD0673553D0E4E7E7A7534107EB05C9030628C4F8194DA08BFB3075EA45AA7FE5B88C39D55AF2288C5DE6C881D5C1F94994DA9AB27354EC1A3C9046096135CE7FE516C9D45F6DED90DB385D0557ED1C4139E710142A550C72508933489303C4AAFC206EE2032320B4FEC5177FD74CA37FD8D90D35E0CAA04408CCF78012DC55E2B824520B5CF51A371D00DECAA9054F58095614E8E67CDF4E2BCBE72D6A4DB5CD8FB08516A4BCC5FDC1E3A2C008DC1FEFEA61CDC8A9E5A33D4AF6B812E2EC2ACFC0E7A6B8E539C9D71ECEDE99BA1A44D3EECA09950D86B526797A406E83E4BC908D6CDA5DBFFEB9F3F6DB5F5F1C1CA79B653C07AF8CA82CAB3556A7AF2DB0B27E1DFAA661B5BA1FE669863D3A21EA8D21197E2F3652DA9B33CE8007CFD5137A34157149994DC10994725A7A17DBFB3B3D168CC297F4C7EF6783F422BF4E9230E39E19355D2793ADDC06140E32321F27F14F56520607085D4ADE713CEB9D1FA8709CAC3703EC7A837882C9A04212BDD8C0DFBFCEF702CD8FC51D11D7BB1C3095A66610F00AE007B11E4D64D386CD7A6CA62C492C32C61E919CB7170B9D98D0DD66507DBC7F63F509FBE28505946A36DC28BF609BE996E6241290640FF9F393A343093DCDA9932C16B832B318EA3121F0B700FF313C23D6A41E275FAB8ABD4CBF0B7F36C65AEDED17F09CBFC04B044B723BEEBD36B2AF1289CF98BB96BB1CB377F2FE1D25A35DEF6447B1E04EADA63CD9BE0AC83E253FBD2894DA144A5224D9D0EB6D6677DDF97CF42CE2767FC7B763E3E31DBB84FD9ED56975FB2F10521C56BE21A7F6118A2FE7799FB5CD0AA93313C3FA7666B8D30B88D17EF0A7B5730CA9F9FD1FA2EB7B5E5A49C69E907660546A2DA0922D0C0653CDD493616FE469AE18203C3D6F0350B45617726D0CA0E01C42EED501538B3CE6276C2303AFC175DCB6C34F7092B3433F2F041D59E23DBFE0E83332D5B1107157A0CEE0A2428FC8737164AABC91C85F75C49621D4E2DDB5181E65836834E731518FE148E23907D550D8F0279755CE567B9B77BCF2620138E9B119BA17D09163990BF08AE843EAED818893D62BB76631C88D6409A8703D421AD98CDD16896DE4BF1BEF0D8796D3DF95103D3FB41C12668E304385B5508F78198821CC13800DBEC847E23E2DF4721C03295A27F846CA3C95A05D2448263C914A0D0209A2D708FF4835868E48A816EB648BABF1F1D9852B81C4A32C8F45F44ED63AD634EFD6A7BF1B4C018D725C0138CCDC7A3C7502DC2041F1D04A96144B49B2F8EA49E0D3C7DEC820F4CDF9D71D621F3930CA9FA737B084B718B7A56FD0001A0B648FCE10720D9709E3ACB360E76AB985BA662455451778A86A52D16ED8BBDDB82ED141C6F7C9AF651FE4CEB068FF19FA268E2BADC848ECA656B5EA1162B6EEB26D1C5FF164D013D613EC9E21CDDF6E54744425C76E3AD48ED2A6A70BAFC341BE168C9FC7438E3E4C5589E89CC7BF5CFDC199E4A464F53EFCE3937FA31C696FFC7965E11ADD3FEE58E89ACC9D37B8124259C3750FBFCB13F7976CF721D726D712271044E9F1A6C14699947F88FCD669EAB556D9F37B871CCE4FE2B19E7687FD6813B9739192D7A923B6E096CC9963DA590B6DA9D48EFACB9AB1E7B1AACA7827306108FCC72F65486F2B58EE4721CEA6018D7C16ADB86E6110709306EB36C47CA537B89B4F7557C5121B5232F00B4C3DB8C2F86F471D5EF90C48722EF47753B15E58F9835F777909ECB3240CA16AB982AF877E4EF822D59EBB1812B88F44884618039DA99DEBC53701E0921CBC1A011F00F1737CBD7BB4FBF2977F24AE8AEA7EDFE016B279BA8413043BBABD1EC8263A3CB13369C358262EB61754220EC962CC9DA0D4D2544DD0346B36D4D39A7B0356B98DFE1147882FA4C27AF049EEEF59D91F5528221EDFB82AD28AF1F30CE8A9A651430158A27211C5C2F3B20C78A2BF6D39D1CC44215499CAD69093595DF2A8847778ECAE4C83BADB994976B8AA9ECB65D44FF5A56D212F4DF15DEC5D5DAB0F49AC6CAB440893AD2F87296540A29B28C2FDE1045D4C54E122BCF541F9B331F4FE8A1943C351B956E8AD41FA4E43DC3343579F9F52A99962E0200CFD6F8C5A618177A91CD5DE80528D33ADF6BF35112FBA54CE5414B36CA7A6FF1243B4469A86A3F13F5F73869811BBB452CF4130620273B143FA02FE62B0C71CA36DE03C16D002FF37214784E0FE4ED152877433B44A6CF28FCB269D75DB1286E45474D6E31E3BC703058ADF04F38C6A8788D98547BB711844D34A42A705F4B1F04C887D58D66A3C0CBBFF3562B1D5CA1DCD937A9D12D36CC1DDBD5D4A3882F23AED80DACD9A381AF366981B2842F5F220848B39F82F9F7127561780E16D9294D7177858B3F65E99078BAA3C83FAC2D96E9EBADCDD6A09A33634D12C333FA6AD932C15EB1C5490DB616893F1D2D5B5F8724AC0980B35C0ECF294AD44D1BFF19B8D8D0249F335242F36C602AE46FC47E389A2EC34CB342AA56609104073D7917406DAFE2AC93F9B5E8F9BCC25DB36E612D5212C733408768BA7DB899E93A3BB02EBEDD53D1F1D14A616104D41BFC270FF90001",
  // customization
  "km}zS=U8={k/T m",
  // capacity
  512,
  // functionName
  "KMAC",
  // macLengthBitsRequested
  1024
)

mac w/ customization:

"8506FC9B1CF4B1DE588507B4AB8C5E9C4B1E09D4940B0F84C5F54368566E12978D4070908B4A7BE2F07A7C781F3E3D697E88AB6AC03673A7A3AF854C816ED2DC6F113D35FEA0C4A100300FA22F5017AE37E685E2FD3A187362A37AF6227D96F3B7A42D8F1B82C628927C10F2524CC4898BB2707832B4B3694566C26D5414E5B5"

Running the same kmac test, this time w/o a customization:

"168851349BDA1E6800651F3086C2E74B1D4090767E9CE9334FF7D9BD0D15E137263971F78396FEEDF5299B8D0CB12EB6E0D424B03A14D3313E7B2DE169B5863438EC9BD5A23C54E48C9F4349D5810FC954243E60127D330E905289B9FECC3D273A91649BBE0301CE95A5C48308B4F5D51A6540B81170616B356655E64A7382CB"

I'm hoping we disagree at some point prior to actually entering the cshake invoke, if not I can drill down further.

I have not yet looked into the differences between the hexCustomization or !hexCustomization.

This couldn't be related to usnistgov/ACVP#912 could it? I'm still not clear on what is special about the byte lengths provided in the issue to easily see which test cases it applies to.

[dghgit]

Found it! It was us, the section of KMAC doing bytePad() also had the extra padding issue. Sigh. Good guess on ACVP#912. I'll add our new vector tests - in this case the issue gets triggered by the key size passed to KMAC, so it's separate to the supporting cSHAKE function, but still the same thing.

I've done a few more runs of KMAC with hexCustomization=false. All work. Thanks!

I checked hexCustomization=true again. If hexCustomization is true I still get errors.

Two interesting vectors for:
"testType":"AFT","xof":true,"hexCustomization":true

This one passes:
{"tcId":113,"key":"DEB2E497870DBDAE128E4E6F34AC201279E46502AFF0CB6389474245420D00230FC30B6086B8B2DCDA0A9C796C4932D8888714F87A7BD2EA136B4F3AEA98B9AF08BAB96784A8998DDCE07E82DADFCFD9812CDEFCBC09DEC82B900920468FBD588D77593B22B4046683D0015943FE7A1D8B96C9A7F1462662F6DC1C880F1D5EDE39C18054FC5A704BD3CF42E36BD4A44718183EA56EF61A0ECB569692FFD836C6B36C2EE8459335C93B379E0C54F2B88C6F1C6045E4124B67AAC6E73224AB8372DB97D523611363795E22B7B95EC4D4B388BA3E834A125E89908FFBA90C93ECF3F1FDD029A64047AD9FAB58429BDBF7A56FF4C6109F746FACC07B64A8C10224B6645395566A44A84ECBDEF7B9BF1F04F4","keyLen":2176,"msg":"BAB00C8681CC817E726F57B896E55366E145AF4916DDCD72EB0F96A011197FF2C0358894FA71D59F8E5F40743FCE77D75BACAD08BFEB95424406F8566B0F","msgLen":496,"macLen":3560,"customizationHex":"6741BA9B9489CD2A60"},

We both get:
{"tcId":113,"mac":"1b1800b1035ccb42bb058485317b5c191fe09177c5b2012428acb7815d27100d777d0cfdfa1ec1c907c881c1fa1fdeee9699c78c6f8e49d1515c786631568b591ddf9103cee7fb624b79f3602e282d133be145a6d32dd198eff51057d4ada3a8e47cb30673abc656f2c40b97dfbd1695d8fd6f0d7e0d0d0f75b9527e17b60bf50cad4f7e52426cc1bc7cb8f58e7e8903c3e11fc414a81926f24fadb77e51c65cddc8fed1512ccb20d0a43210364dbf0f769128ed870146e5e67e6afba9a597a13117a197eeb0e80c8769c53a25fd0813634b6d37ca53817a67738369d56fd9c5525bcc5a126223a7f586cbd73a94610ace67885240d5ec6c2a15b62434b5f6049fbdf1211e0cf5b1561196880625a788e3c9037a603a2edc550d486b2585d3e4cfdc300949086507319a478ba4557ad020a12d907e52e9bee64e7319dd6c5e2cc1aaf466aac025ca86ba4b1c88ca82ad3a302bc789ec099c4838c2a7acd20eaa83e12237015cea302b4027155abf0caec7b7aa0e072b7abb5e063777fd5a4b4e5c6e6995ffe6174344ebc90bf180c44169879449d5ddc305b3caf1f4bbfceb97cbaed8408f190a7b0a3bf8637e833a62ecd13abf77079bc86e0ee86e28"},

This one is different though (as are most of the others in the test group, seems to be about 1 in 10 pass):

{"tcId":102,"key":"650CF430C32EBAD40B284B9D0FE3D631412EA94DB5DB7E9C46513C6CAF5D7F3D3E4EAA5789AF2DCCC5BA1B03FBA66F3024BAEC9FC3808A0F647DE0F0FB7F8F272C46437F14FBEB2ECD6CCD40EBE21A264237EB3E84AB8482CDBE5374DDFA3992FB06E615CCB16810F8714B470358CEB7F1BDEB35128A2D125C10307D0CE49F18B4D098B99A0BE42D6DFEC8B2BEE193FE3B5F587C13EE6393C08121B0DFD50FC1230A2996560A174A3AFE23761DEE947EC110521BA5C222DE67EEC4B1561E2C9075E6A0535F540E8E7AFE3F35CCCB81843377CF0B6684A7C45D89EA649991487A34F0419F72C7F6911CB99F4D877F6751525D439E996080B685131A56E62A48194BC7B5331DD4E27616696BE73A2DD878DFD08FE1AB9442E9245E73553E0CFA88A0DA7A5FF746","keyLen":2352,"msg":"0E240A787671A8158E0DB1DB5157F2A13EC681586DFE2278053F855E601DF4B80A8E1F85423C","msgLen":304,"macLen":952,"customizationHex":"E1D26B20"}

I get:
{"tcId":102,"mac":"82bb9e22c34a846f5efe008ee2ed98a2f1d88a9ab8be54261d18ad763175
b92c6cfca58a0da68b87eda97197601bde64c5c2c11c03f0899f48d8d5a9a5f1216942e6f1c379234d4591bca30158ec8dc68541ccdc3b610c6ec208360a747e42dd30b52890a38cfdb97918b6945f53bce67c011230407340"}

The server expects:
{"tcId":102,"mac":"E65F71A1BB1DCE6CB653105CC208383B19E9C7910C3B1D738639B5A8CA61A809E5EBFCDD19D51246B518D7DE3F067CBA78EF37AD126CE0A6C001CBC79D43580936B2941850318EFD690629359EBB9152E46CEE7C7F05C6BDC76F8456651DE6536022EE2BF688950A42FB0AB7F11D5BDB5AF133E48AE531"}

Failures only in AFT tests with hexCustomization = true. One thing that also surprised me, even though hexCustomisation is set to true there are still vectors in the file with hexCustomization = false. That seems a bit odd.

The URLs were:
"/acvp/v1/testSessions/118847/vectorSets/344478",
"/acvp/v1/testSessions/118847/vectorSets/344479"

[Kritner]

Found it! It was us, the section of KMAC doing bytePad() also had the extra padding issue. Sigh. Good guess on ACVP#912. I'll add our new vector tests - in this case the issue gets triggered by the key size passed to KMAC, so it's separate to the supporting cSHAKE function, but still the same thing.

I've done a few more runs of KMAC with hexCustomization=false. All work. Thanks!

Awesome, good to hear!

---

One thing that also surprised me, even though hexCustomisation is set to true there are still vectors in the file with hexCustomization = false. That seems a bit odd.

The intention was that hexCustomization is an additional "feature" of cshake and its derivatives, where the "string customization" is assumed. We are always going to be testing against a "string customization", but optionally against a hexCustomization. I know they're more or less equivalent as you just get the byte[] from the string customization.

---

I checked hexCustomization=true again. If hexCustomization is true I still get errors.
This one is different though (as are most of the others in the test group, seems to be about 1 in 10 pass)

I believe I've found the reasoning behind this. We were randomly picking "lengths" to use per test case for either the hexCustomization, or the stringCustomization. "length" as it applies to "strings" is just a character, length of "12" means 12 characters. "length" as it applied to hexCustomization was a number of bits. So in the case that you linked above (tcId 102), we had actually generated a hexCustomization of 28 bits; but we only send bitstrings in "full bytes" or 32 bits. We had an additional 4 bits of padding in what we sent you, whereas the crypto was run against only the 28 bits, rather than the full 32, and we weren't sending the "number of bits to take" from the customization.

If you run the above test case with only the 28 most significant bits, I think we'll agree. Additionally, we do agree if I run the crypto against the full 32 bits of the hexCustomization.

That also explains why it's about "1 in 10" that were passing, only about that many test cases were generating hexCustomizations on the byte boundary.

I'll be putting in a change so that hexCustomizations are always generated on the byte boundary. It should be in the next release.

This oversight around hexCustomization also looks to be the case for ParallelHash, CSHAKE, and TupleHash. I'm not sure if you had test hexCustomization against those algorithms already or not, but I'm assuming not since the problem should have been there as well; and I don't think we've discussed those issues, so I believe they've not yet been hit.

[dghgit]

Ah, so the zero was padding... Yes, that's correct, I've stayed away from hexCustomization outside of KMAC. I'll do a full round of testing with hexCustomization set to true and false when the next update is done.

[Kritner]

This change is on demo in release v1.1.0.13.

[dghgit]

I've given it a run, with both hexCustomozation true and false, over the full set of XOF functions. All seems to be working now. Thanks. Marking this one closed.

[Kritner]

This change is now on prod in release https://github.com/usnistgov/ACVP-Server/releases/tag/v1.1.0.13