Curve point evaluation output should be validated. It can be infinity

[ilitteri]

Context: P256VERIFY.yul#L606

Description:

According to ecdsa algorithm description after Step 5 it we should validate that the result is not infinity. Otherwise we should return invalid signature.
Current implementation does not do it which results passing 0 to montgomeryModularInverse. This is unexpected input value for the function which results endless loop.

This happens for case when i.e. r == hash and Q == -G

Recommendation:

Add additional check on zr value and handle it returning 0 (invalid signature)
0x5ad83880e16658d7521d4e878521defaf6b43dec1dbd69e514c09ab8f1f2ffe25ad83880e16658d7521d4e878521defaf6b43dec1dbd69e514c09ab8f1f2ffe2871c518be8c56e7f5c901933fdab317efafc588b3e04d19d9a27b29aad8d9e696b17d1f2e12c4247f8bce6e563a440f277037d812deb33a0f4a13945d898c296b01cbd1c01e58065711814b583f061e9d431cca994cea1313449bf97c840ae0a.

diff --git a/precompiles/P256VERIFY.yul b/precompiles/P256VERIFY.yul
index c75be80..c650a35 100644
--- a/precompiles/P256VERIFY.yul
+++ b/precompiles/P256VERIFY.yul
@@ -604,6 +604,10 @@ object "P256VERIFY" {
             let t1 := outOfMontgomeryForm(montgomeryMul(r, s1, N(), N_PRIME()), N(), N_PRIME())
 
             let xr, yr, zr := shamirLinearCombination(x, y, z, t0, t1)
+            if iszero(zr) {
+                mstore(0, 0)
+                return(0, 32)
+            }
 
             // As we only need xr in affine form, we can skip transforming the `y` coordinate.
             let z_inv := montgomeryModularInverse(zr, P(), R2_MOD_P())

[ilitteri]

Resolved in #200