NXP CAAM RSA Driver Inconsisently Failing on IMX6UL

[Zephyrus29]

Hi, I'm running the current tip of the master branch on my IMX6UL and have some problems with the RSA driver.

When I run xtest with tests 4006, it will fail on either test 20 or 32 most of the time.
Sometimes it passed them all, but it was fairly inconsistent.
It looks to be a problem with signing, no other tests seem to use that set of functions.
Does anyone have any suggestions for something I could try that might fix these problems?

I've copied in a bunch of trace logs below

Test 4006.20

E/TC:? 0 crypto_acipher_alloc_rsa_public_key:53 RSA Public Key (1024 bits) alloc ret = 0x0
2949
E/TC:? 0 crypto_acipher_alloc_rsa_keypair:32 RSA Keypair (1024 bits) alloc ret = 0x0
E/TC:? 0 crypto_acipher_alloc_rsa_keypair:32 RSA Keypair (1024 bits) alloc ret = 0x0
E/TC:? 0 rsassa_pss_sign:749 modBits = 1024, hence EM Length = 128
E/TC:? 0 emsa_pss_encode:353 PS Len = 74, DB Len = 95, M' Len = 60
E/TC:? 0 emsa_pss_encode:388 Get salt of 20 bytes (ret = 0x00000000)
E/TC:? 0 drvcrypt_hash_alloc_ctx:18 hash alloc_ctx algo 0x50000004
E/TC:? 0 drvcrypt_hash_alloc_ctx:27 hash alloc_ctx ret 0x0
E/TC:? 0 emsa_pss_encode:407 H = hash(M') returned 0x00000000
E/TC:? 0 emsa_pss_encode:411 H = hash(M') @0xbe0752a7: 32
E/TC:? 0 emsa_pss_encode:429 DB @0xbe0751e0: 95
E/TC:? 0 drvcrypt_rsa_mgf1:25 Generate Mask (95 bytes) with seed of 32 bytes
E/TC:? 0 drvcrypt_hash_alloc_ctx:18 hash alloc_ctx algo 0x50000004
E/TC:? 0 drvcrypt_hash_alloc_ctx:27 hash alloc_ctx ret 0x0
E/TC:? 0 drvcrypt_rsa_mgf1:45 Nb Loop (32 bytes) = 2, last Block = 31 bytes
E/TC:? 0 drvcrypt_rsa_mgf1:72 Last Block = 31 bytes
E/TC:? 0 drvcrypt_rsa_mgf1:103 return 0x00000000
E/TC:? 0 emsa_pss_encode:450 dbMask = MGF(H, emLen - hLen - 1) returned 0x00000000
E/TC:? 0 emsa_pss_encode:455 dbMask @0xbe075248: 95
E/TC:? 0 emsa_pss_encode:470 maskedDB = DB xor dbMask returned 0x00000000
E/TC:? 0 emsa_pss_encode:474 maskedDB @0xbe075248: 95
E/TC:? 0 emsa_pss_encode:489 EM @0xbe075248: 128
E/TC:? 0 rsassa_pss_sign:754 EMSA PSS Encode returned 0x00000000
E/TC:? 0 crypto_acipher_rsassa_sign:382 Encode signature algo (0x70414930) returned 0x0
E/TC:? 0 crypto_acipher_alloc_rsa_public_key:53 RSA Public Key (1024 bits) alloc ret = 0x0
E/TC:? 0 rsassa_pss_verify:801 In pss_verify
E/TC:? 0 rsassa_pss_verify:822 modBits = 1024, hence EM Length = 128
E/TC:? 0 emsa_pss_verify:544 PS Len = 74, DB Len = 95, M' Len = 60
E/TC:? 0 emsa_pss_verify:568 Error leftmost octet maskedDB not 0's
E/TC:? 0 emsa_pss_verify:569 EM[0] = 0xB9 - EM Len = 128, emBits = 1023
E/TC:? 0 rsassa_pss_verify:854 EMSA PSS Verify returned 0xffff3072
E/TC:? 0 crypto_acipher_rsassa_verify:454 Signature verif algo (0x70414930) returned 0xffff3072

E/TC:? 0 drvcrypt_hash_alloc_ctx:27 hash alloc_ctx ret 0x0
to case 33 algo 0x70414930 line 2949
E/TC:? 0 crypto_acipher_alloc_rsa_public_key:53 RSA Public Key (1024 bits) alloc ret = 0x0
E/TC:? 0 crypto_acipher_alloc_rsa_keypair:32 RSA Keypair (1024 bits) alloc ret = 0x0
E/TC:? 0 crypto_acipher_alloc_rsa_keypair:32 RSA Keypair (1024 bits) alloc ret = 0x0
E/TC:? 0 rsassa_pss_sign:749 modBits = 1024, hence EM Length = 128
E/TC:? 0 emsa_pss_encode:353 PS Len = 74, DB Len = 95, M' Len = 60
E/TC:? 0 emsa_pss_encode:388 Get salt of 20 bytes (ret = 0x00000000)
E/TC:? 0 drvcrypt_hash_alloc_ctx:18 hash alloc_ctx algo 0x50000004
E/TC:? 0 drvcrypt_hash_alloc_ctx:27 hash alloc_ctx ret 0x0
E/TC:? 0 emsa_pss_encode:407 H = hash(M') returned 0x00000000
E/TC:? 0 emsa_pss_encode:411 H = hash(M') @0xbe0752a7: 32
E/TC:? 0 emsa_pss_encode:429 DB @0xbe0751e0: 95
E/TC:? 0 drvcrypt_rsa_mgf1:25 Generate Mask (95 bytes) with seed of 32 bytes
E/TC:? 0 drvcrypt_hash_alloc_ctx:18 hash alloc_ctx algo 0x50000004
E/TC:? 0 drvcrypt_hash_alloc_ctx:27 hash alloc_ctx ret 0x0
E/TC:? 0 drvcrypt_rsa_mgf1:45 Nb Loop (32 bytes) = 2, last Block = 31 bytes
E/TC:? 0 drvcrypt_rsa_mgf1:72 Last Block = 31 bytes
E/TC:? 0 drvcrypt_rsa_mgf1:103 return 0x00000000
E/TC:? 0 emsa_pss_encode:450 dbMask = MGF(H, emLen - hLen - 1) returned 0x00000000
E/TC:? 0 emsa_pss_encode:455 dbMask @0xbe075248: 95
E/TC:? 0 emsa_pss_encode:470 maskedDB = DB xor dbMask returned 0x00000000
E/TC:? 0 emsa_pss_encode:474 maskedDB @0xbe075248: 95
E/TC:? 0 emsa_pss_encode:489 EM @0xbe075248: 128
E/TC:? 0 rsassa_pss_sign:754 EMSA PSS Encode returned 0x00000000
E/TC:? 0 crypto_acipher_rsassa_sign:382 Encode signature algo (0x70414930) returned 0x0
E/TC:? 0 crypto_acipher_alloc_rsa_public_key:53 RSA Public Key (1024 bits) alloc ret = 0x0
E/TC:? 0 rsassa_pss_verify:801 In pss_verify
E/TC:? 0 rsassa_pss_verify:822 modBits = 1024, hence EM Length = 128
E/TC:? 0 emsa_pss_verify:544 PS Len = 74, DB Len = 95, M' Len = 60
E/TC:? 0 emsa_pss_verify:568 Error leftmost octet maskedDB not 0's
E/TC:? 0 emsa_pss_verify:569 EM[0] = 0xA7 - EM Len = 128, emBits = 1023
E/TC:? 0 rsassa_pss_verify:854 EMSA PSS Verify returned 0xffff3072
E/TC:? 0 crypto_acipher_rsassa_verify:454 Signature verif algo (0x70414930) returned 0xffff3072

Test 4006.32:

E/TC:? 0 crypto_acipher_alloc_rsa_public_key:53 RSA Public Key (1024 bits) alloc ret = 0x0
3059
E/TC:? 0 crypto_acipher_alloc_rsa_keypair:32 RSA Keypair (1024 bits) alloc ret = 0x0
E/TC:? 0 crypto_acipher_alloc_rsa_keypair:32 RSA Keypair (1024 bits) alloc ret = 0x0
E/TC:? 0 rsassa_pss_sign:749 modBits = 1024, hence EM Length = 128
E/TC:? 0 emsa_pss_encode:353 PS Len = 74, DB Len = 95, M' Len = 60
E/TC:? 0 emsa_pss_encode:388 Get salt of 20 bytes (ret = 0x00000000)
E/TC:? 0 drvcrypt_hash_alloc_ctx:18 hash alloc_ctx algo 0x50000004
E/TC:? 0 drvcrypt_hash_alloc_ctx:27 hash alloc_ctx ret 0x0
E/TC:? 0 emsa_pss_encode:407 H = hash(M') returned 0x00000000
E/TC:? 0 emsa_pss_encode:411 H = hash(M') @0xbe07526f: 32
E/TC:? 0 emsa_pss_encode:429 DB @0xbe0751a8: 95
E/TC:? 0 drvcrypt_rsa_mgf1:25 Generate Mask (95 bytes) with seed of 32 bytes
E/TC:? 0 drvcrypt_hash_alloc_ctx:18 hash alloc_ctx algo 0x50000004
E/TC:? 0 drvcrypt_hash_alloc_ctx:27 hash alloc_ctx ret 0x0
E/TC:? 0 drvcrypt_rsa_mgf1:45 Nb Loop (32 bytes) = 2, last Block = 31 bytes
E/TC:? 0 drvcrypt_rsa_mgf1:72 Last Block = 31 bytes
E/TC:? 0 drvcrypt_rsa_mgf1:103 return 0x00000000
E/TC:? 0 emsa_pss_encode:450 dbMask = MGF(H, emLen - hLen - 1) returned 0x00000000
E/TC:? 0 emsa_pss_encode:455 dbMask @0xbe075210: 95
E/TC:? 0 emsa_pss_encode:470 maskedDB = DB xor dbMask returned 0x00000000
E/TC:? 0 emsa_pss_encode:474 maskedDB @0xbe075210: 95
E/TC:? 0 emsa_pss_encode:489 EM @0xbe075210: 128
E/TC:? 0 rsassa_pss_sign:754 EMSA PSS Encode returned 0x00000000
E/TC:? 0 crypto_acipher_rsassa_sign:382 Encode signature algo (0x70414930) returned 0x0
E/TC:? 0 crypto_acipher_alloc_rsa_public_key:53 RSA Public Key (1024 bits) alloc ret = 0x0
E/TC:? 0 rsassa_pss_verify:801 In pss_verify
E/TC:? 0 rsassa_pss_verify:822 modBits = 1024, hence EM Length = 128
E/TC:? 0 emsa_pss_verify:544 PS Len = 74, DB Len = 95, M' Len = 60
E/TC:? 0 emsa_pss_verify:556 rigthmost octet != 0xbc (0x51)
E/TC:? 0 rsassa_pss_verify:854 EMSA PSS Verify returned 0xffff3072
E/TC:? 0 crypto_acipher_rsassa_verify:454 Signature verif algo (0x70414930) returned 0xffff3072

[clementfaure]

Hello @Zephyrus29,
I just tested on imx6ulevk and I get a xtest 4006 failure on the fourth loop ..
Thanks for raising the issue, I will investigate

[Zephyrus29]

I've run it 10 or so times with that patch applied and it passes everytime.
Looks to have fixed the issue, thankyou.