Invalid size in RSA/sigGen SHAKE-128 in PSS/mgf1

[sandor-szendro-i4p]

environment
Demo

testSessionId
428747

vsId
1774510

Algorithm registration
{
"revision":"FIPS186-5",
"algorithm":"RSA",
"mode":"sigGen",
"capabilities":[
{
"sigType":"pkcs1v1.5",
"properties":[
{
"modulo":2048,
"hashPair":[
{
"hashAlg":"SHA2-224"
},
{
"hashAlg":"SHA2-256"
},
{
"hashAlg":"SHA2-384"
},
{
"hashAlg":"SHA2-512"
},
{
"hashAlg":"SHA3-224"
},
{
"hashAlg":"SHA3-256"
},
{
"hashAlg":"SHA3-384"
},
{
"hashAlg":"SHA3-512"
}
]
},
{
"modulo":3072,
"hashPair":[
{
"hashAlg":"SHA2-224"
},
{
"hashAlg":"SHA2-256"
},
{
"hashAlg":"SHA2-384"
},
{
"hashAlg":"SHA2-512"
},
{
"hashAlg":"SHA3-224"
},
{
"hashAlg":"SHA3-256"
},
{
"hashAlg":"SHA3-384"
},
{
"hashAlg":"SHA3-512"
}
]
},
{
"modulo":4096,
"hashPair":[
{
"hashAlg":"SHA2-224"
},
{
"hashAlg":"SHA2-256"
},
{
"hashAlg":"SHA2-384"
},
{
"hashAlg":"SHA2-512"
},
{
"hashAlg":"SHA3-224"
},
{
"hashAlg":"SHA3-256"
},
{
"hashAlg":"SHA3-384"
},
{
"hashAlg":"SHA3-512"
}
]
}
]
},
{
"sigType":"pss",
"properties":[
{
"modulo":2048,
"maskFunction":[
"mgf1"
],
"hashPair":[
{
"hashAlg":"SHA2-224",
"saltLen":28
},
{
"hashAlg":"SHA2-256",
"saltLen":32
},
{
"hashAlg":"SHA2-384",
"saltLen":48
},
{
"hashAlg":"SHA2-512",
"saltLen":64
},
{
"hashAlg":"SHA3-224",
"saltLen":28
},
{
"hashAlg":"SHA3-256",
"saltLen":32
},
{
"hashAlg":"SHA3-384",
"saltLen":48
},
{
"hashAlg":"SHA3-512",
"saltLen":64
},
{
"hashAlg":"SHAKE-128",
"saltLen":16
},
{
"hashAlg":"SHAKE-256",
"saltLen":32
}
]
},
{
"modulo":3072,
"maskFunction":[
"mgf1"
],
"hashPair":[
{
"hashAlg":"SHA2-224",
"saltLen":28
},
{
"hashAlg":"SHA2-256",
"saltLen":32
},
{
"hashAlg":"SHA2-384",
"saltLen":48
},
{
"hashAlg":"SHA2-512",
"saltLen":64
},
{
"hashAlg":"SHA3-224",
"saltLen":28
},
{
"hashAlg":"SHA3-256",
"saltLen":32
},
{
"hashAlg":"SHA3-384",
"saltLen":48
},
{
"hashAlg":"SHA3-512",
"saltLen":64
},
{
"hashAlg":"SHAKE-128",
"saltLen":16
},
{
"hashAlg":"SHAKE-256",
"saltLen":32
}
]
},
{
"modulo":4096,
"maskFunction":[
"mgf1"
],
"hashPair":[
{
"hashAlg":"SHA2-224",
"saltLen":28
},
{
"hashAlg":"SHA2-256",
"saltLen":32
},
{
"hashAlg":"SHA2-384",
"saltLen":48
},
{
"hashAlg":"SHA2-512",
"saltLen":64
},
{
"hashAlg":"SHA3-224",
"saltLen":28
},
{
"hashAlg":"SHA3-256",
"saltLen":32
},
{
"hashAlg":"SHA3-384",
"saltLen":48
},
{
"hashAlg":"SHA3-512",
"saltLen":64
},
{
"hashAlg":"SHAKE-128",
"saltLen":16
},
{
"hashAlg":"SHAKE-256",
"saltLen":32
}
]
}
]
}

Endpoint in which the error is experienced
https://demo.acvts.nist.gov/acvp/v1/testSessions GET

Expected behavior
For RSA/sigGen when sigType is "pss" and maskFunction is "mgf1" and hashAlg is SHAKE-128 the expected results returned by ACVP server only the first 16 bytes of the 32 bytes of the SHAKE-128 output is used in the mask generation function.

Additional context
According to FIPS 186-5 5.4.1 Mask Generation Functions in RSASSA-PSS refers to B.2.1 of RFC 8017. B.2.1 of RFC 8017 contains the steps for using the mask generation function, where step 3 is:
"For counter from 0 to \ceil (maskLen / hLen) - 1, do the following:"
By examination of the expected test vectors we think in this step instead of hLen, hLen / 2 is used.
This means for SHAKE-128 instead of maskLen / 32 -1, maskLen / 16 -1 is used.
We think maskLen / 32 -1 should be used.

In case of SHAKE-256 32 bytes are used instead of 64.

[jbrock24]

@sandor-szendro-i4p Looking into this

[livebe01]

Thanks for letting us know. I think I see where the issue is. I'm working on it now. I'll give you an update when it's been fixed.

[livebe01]

We have a fix in for this. It will go out in the next release.

[szendros]

Thanks for the fix, I will be able to test it on monday.

[livebe01]

Thank you!

[livebe01]

The fix for this is on Demo in release v1.1.0.31.

[szendros]

I tested it on Demo and it works for me.
Thank you, this can be closed.

[livebe01]

Thanks. Appreciate the confirmation!

[livebe01]

The fix for this is on Prod in release v1.1.0.31.