feat(ux): out-of-place syntactic sugar (#413)

ARCHITECTURE.md

@@ -30,6 +30,24 @@ and prefer mutable out parameters.
    on large inputs like `Fp12[BLS12_381]` 48\*12 bytes = 576 bytes.
 4. As we sometimes return SecretBool or status code, this keeps the API consistent.
 
+Syntactic sugar through out-of-place arithmetic functions like `+` and `*`
+is available for rapid prototyping, testing and debugging.
+
+For elliptic curves, variable time functions are prefixed with a tilde like
+`~+` `~-` and `~*`.
+
+They SHOULD NOT be use in performance-critical or stack space critical
+subroutines.
+- They should be tagged {.inline, noInit.} and just forward to the in-place function
+  to guarantee copy elision. (and even then it's not guaranteed)
+- Issues:
+  - Extremely inefficient codegen in Constantine itself https://github.com/mratsim/constantine/issues/145
+    with useless moves instead of in-place construction.
+  - In other languages like Rust, users have seen a dramatic 20% increase in performance by moving from out-of-place to in-place mutation: https://www.reddit.com/r/rust/comments/kfs0oe/comment/ggc0dui/
+    - And they are struggling with GCE (Guarenteed Copy Elision) and NRVO/RVO(Named) Return Value Optimization
+      - https://github.com/rust-lang/rust/pull/76986
+      - https://github.com/rust-lang/rfcs/pull/2884
+
 ## Code organization
 
 TBD

PLANNING.md

@@ -12,20 +12,20 @@ Other tracks are stretch goals, contributions towards them are accepted.
 <!-- TOC -->
 
 - [Constantine's planning](#constantines-planning)
-    - [Table of Contents](#table-of-contents)
-    - [Tracks](#tracks)
-        - [Tech debt track](#tech-debt-track)
-        - [Ethereum Consensus Track](#ethereum-consensus-track)
-        - [Ethereum Execution Track](#ethereum-execution-track)
-        - [Proving Ethereum track](#proving-ethereum-track)
-        - [Optimization track](#optimization-track)
-        - [User Experience track](#user-experience-track)
-        - [Technical marketing track](#technical-marketing-track)
-        - [ZK and proof systems track](#zk-and-proof-systems-track)
-        - [Multi-party computation MPC track](#multi-party-computation-mpc-track)
-        - [Core crypto track](#core-crypto-track)
-        - [Fully-Homomorphic encryption FHE track](#fully-homomorphic-encryption-fhe-track)
-        - [Post-Quantum cryptography PQC track](#post-quantum-cryptography-pqc-track)
+  - [Table of Contents](#table-of-contents)
+  - [Tracks](#tracks)
+    - [Tech debt track](#tech-debt-track)
+    - [Ethereum Consensus Track](#ethereum-consensus-track)
+    - [Ethereum Execution Track](#ethereum-execution-track)
+    - [Proving Ethereum track](#proving-ethereum-track)
+    - [Optimization track](#optimization-track)
+    - [User Experience track](#user-experience-track)
+    - [Technical marketing track](#technical-marketing-track)
+    - [ZK and proof systems track](#zk-and-proof-systems-track)
+    - [Multi-party computation (MPC) track](#multi-party-computation-mpc-track)
+    - [Core crypto track](#core-crypto-track)
+    - [Fully-Homomorphic encryption (FHE) track](#fully-homomorphic-encryption-fhe-track)
+    - [Post-Quantum cryptography (PQC) track](#post-quantum-cryptography-pqc-track)
 
 <!-- /TOC -->
 
@@ -102,21 +102,7 @@ Other tracks are stretch goals, contributions towards them are accepted.
 
 ### User Experience track
 
-- Provide curves_sugar and fields_sugar higher-level with
-  out-of-place functions like `+` and `*`.
-    - They should be tagged {.inline, noInit.} and just forward to the in-place function
-      to guarantee copy elision.
-    - Mention lack of control over stack space
-    - Guaranteed optimization issue
-        - with even up to a dramatic 20% perf:
-            - https://www.reddit.com/r/rust/comments/kfs0oe/comment/ggc0dui/
-            - past Constantine large copy bug:
-                - out-of-place function lead to bad codegen:
-                  - https://github.com/mratsim/constantine/issues/145
-                - https://github.com/nim-lang/Nim/issues/16897
-    - Mention Rust GCE and NRVO / RVO
-    - https://github.com/rust-lang/rust/pull/76986
-    - https://github.com/rust-lang/rfcs/pull/2884
+Create a "Constantine book" to introduce Constantine concepts and walkthrough available protocols.
 
 ### Technical marketing track
 

constantine/hash_to_curve/hash_to_curve.nim

@@ -54,11 +54,11 @@ func mapToCurve_svdw[F, G](
   tv1 *= h2cConst(F.Name, svdw, G, curve_eq_rhs_Z)
   tv2 = tv1
   when F is Fp:
-    tv2 += F(mres: F.getMontyOne())
-    tv1.diff(F(mres: F.getMontyOne()), tv1)
+    tv2 += F.getOne()
+    tv1.diff(F.getOne(), tv1)
   else:
-    tv2.c0 += Fp[F.F.Name](mres: Fp[F.F.Name].getMontyOne())
-    tv1.c0.diff(Fp[F.F.Name](mres: Fp[F.F.Name].getMontyOne()), tv1.c0)
+    tv2.c0 += Fp[F.F.Name].getOne()
+    tv1.c0.diff(Fp[F.F.Name].getOne(), tv1.c0)
     tv1.c1.neg()
   tv3.prod(tv1, tv2)
   tv3.inv()

constantine/lowlevel_elliptic_curves.nim

@@ -85,3 +85,16 @@ export zoo_subgroups.isInSubgroup
 # ------------------------------------------------------------
 
 export hash_to_curve.hash_to_curve
+
+# Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+# tend to generate useless memory moves or have difficulties to minimize stack allocation
+# and our types might be large (Fp12 ...)
+# See: https://github.com/mratsim/constantine/issues/145
+#
+# They are intended for rapid prototyping, testing and debugging.
+export ec_shortweierstrass.`+`
+export ec_shortweierstrass.`-`
+export ec_shortweierstrass.`~+`
+export ec_shortweierstrass.`~-`
+export ec_shortweierstrass.`*`
+export ec_shortweierstrass.`~*`

constantine/lowlevel_fields.nim

@@ -73,6 +73,10 @@ export arithmetic.setZero
 export arithmetic.setOne
 export arithmetic.setMinusOne
 
+export arithmetic.getZero
+export arithmetic.getOne
+export arithmetic.getMinusOne
+
 export arithmetic.neg
 export arithmetic.sum
 export arithmetic.`+=`
@@ -102,3 +106,14 @@ export arithmetic.sqrt_invsqrt_if_square
 export arithmetic.sqrt_if_square
 export arithmetic.invsqrt_if_square
 export arithmetic.sqrt_ratio_if_square
+
+
+# Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+# tend to generate useless memory moves or have difficulties to minimize stack allocation
+# and our types might be large (Fp12 ...)
+# See: https://github.com/mratsim/constantine/issues/145
+#
+# They are intended for rapid prototyping, testing and debugging.
+export arithmetic.`+`
+export arithmetic.`-`
+export arithmetic.`*`

constantine/math/arithmetic/finite_fields.nim

@@ -150,7 +150,6 @@ func setMinusOne*(a: var FF) =
   #       Check if the compiler optimizes it away
   a.mres = FF.getMontyPrimeMinus1()
 
-
 func neg*(r: var FF, a: FF) {.meter.} =
   ## Negate modulo p
   when UseASM_X86_64 and a.mres.limbs.len <= 6: # TODO: handle spilling
@@ -872,3 +871,53 @@ func batchInv_vartime*[F](dst: var openArray[F], source: openArray[F]) {.inline.
 
 func batchInv_vartime*[N: static int, F](dst: var array[N, F], src: array[N, F]) =
   batchInv_vartime(dst.asUnchecked(), src.asUnchecked(), N)
+
+# ############################################################
+#
+#                 Out-of-Place functions
+#
+# ############################################################
+#
+# Except from the constant function getZero, getOne, getMinusOne
+# out-of-place functions are intended for rapid prototyping, debugging and testing.
+#
+# They SHOULD NOT be used in performance-critical subroutines as compilers
+# tend to generate useless memory moves or have difficulties to minimize stack allocation
+# and our types might be large (Fp12 ...)
+# See: https://github.com/mratsim/constantine/issues/145
+
+func getZero*(T: type FF): T {.inline.} =
+  discard
+
+func getOne*(T: type FF): T {.noInit, inline.} =
+  cast[ptr T](T.getMontyOne().unsafeAddr)[]
+
+func getMinusOne*(T: type FF): T {.noInit, inline.} =
+  cast[ptr T](T.getMontyPrimeMinus1().unsafeAddr)[]
+
+func `+`*(a, b: FF): FF {.noInit, inline.} =
+  ## Finite Field addition
+  ##
+  ## Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+  ## tend to generate useless memory moves or have difficulties to minimize stack allocation
+  ## and our types might be large (Fp12 ...)
+  ## See: https://github.com/mratsim/constantine/issues/145
+  result.sum(a, b)
+
+func `-`*(a, b: FF): FF {.noInit, inline.} =
+  ## Finite Field addition
+  ##
+  ## Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+  ## tend to generate useless memory moves or have difficulties to minimize stack allocation
+  ## and our types might be large (Fp12 ...)
+  ## See: https://github.com/mratsim/constantine/issues/145
+  result.diff(a, b)
+
+func `*`*(a, b: FF): FF {.noInit, inline.} =
+  ## Finite Field substraction
+  ##
+  ## Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+  ## tend to generate useless memory moves or have difficulties to minimize stack allocation
+  ## and our types might be large (Fp12 ...)
+  ## See: https://github.com/mratsim/constantine/issues/145
+  result.prod(a, b)

constantine/math/elliptic/ec_scalar_mul.nim

@@ -291,3 +291,55 @@ func scalarMul*[EC; Ecaff: not EC](R: var EC, scalar: Fr or BigInt, P: ECaff) {.
   ## Those will be assumed to maintain constant-time property
   R.fromAffine(P)
   R.scalarMul(scalar)
+
+# ############################################################
+#
+#                 Out-of-Place functions
+#
+# ############################################################
+#
+# Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+# tend to generate useless memory moves or have difficulties to minimize stack allocation
+# and our types might be large (Fp12 ...)
+# See: https://github.com/mratsim/constantine/issues/145
+
+func `*`*[EC: EC_ShortW_Jac or EC_ShortW_Prj or EC_TwEdw_Prj](
+      scalar: Fr or BigInt, P: EC): EC {.noInit, inline.} =
+  ## Elliptic Curve Scalar Multiplication
+  ##
+  ##   R <- [k] P
+  ##
+  ## Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+  ## tend to generate useless memory moves or have difficulties to minimize stack allocation
+  ## and our types might be large (Fp12 ...)
+  ## See: https://github.com/mratsim/constantine/issues/145
+  result.scalarMul(scalar, P)
+
+func `*`*[F, G](
+      scalar: Fr or BigInt,
+      P: EC_ShortW_Aff[F, G],
+      T: typedesc[EC_ShortW_Jac[F, G] or EC_ShortW_Prj[F, G]]
+      ): T {.noInit, inline.} =
+  ## Elliptic Curve Scalar Multiplication
+  ##
+  ##   R <- [k] P
+  ##
+  ## Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+  ## tend to generate useless memory moves or have difficulties to minimize stack allocation
+  ## and our types might be large (Fp12 ...)
+  ## See: https://github.com/mratsim/constantine/issues/145
+  result.scalarMul(scalar, P)
+
+func `*`*[F, G](
+      scalar: Fr or BigInt,
+      P: EC_ShortW_Aff[F, G]
+      ): EC_ShortW_Jac[F, G] {.noInit, inline.} =
+  ## Elliptic Curve Scalar Multiplication
+  ##
+  ##   R <- [k] P
+  ##
+  ## Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+  ## tend to generate useless memory moves or have difficulties to minimize stack allocation
+  ## and our types might be large (Fp12 ...)
+  ## See: https://github.com/mratsim/constantine/issues/145
+  result.scalarMul(scalar, P)

constantine/math/elliptic/ec_scalar_mul_vartime.nim

@@ -11,6 +11,8 @@ import
   ./ec_shortweierstrass_affine,
   ./ec_shortweierstrass_jacobian,
   ./ec_shortweierstrass_projective,
+  ./ec_twistededwards_affine,
+  ./ec_twistededwards_projective,
   ./ec_endomorphism_accel,
   ./ec_shortweierstrass_batch_ops,
   ./ec_twistededwards_batch_ops,
@@ -409,3 +411,63 @@ func scalarMul_vartime*[EC; Ecaff: not EC](R: var EC, scalar: Fr or BigInt, P: E
   ## Those conditions will be assumed.
   R.fromAffine(P)
   R.scalarMul_vartime(scalar)
+
+# ############################################################
+#
+#                 Out-of-Place functions
+#
+# ############################################################
+#
+# Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+# tend to generate useless memory moves or have difficulties to minimize stack allocation
+# and our types might be large (Fp12 ...)
+# See: https://github.com/mratsim/constantine/issues/145
+
+func `~*`*[EC: EC_ShortW_Jac or EC_ShortW_Prj or EC_TwEdw_Prj](
+      scalar: Fr or BigInt, P: EC): EC {.noInit, inline.} =
+  ## Elliptic Curve variable-time Scalar Multiplication
+  ##
+  ##   R <- [k] P
+  ##
+  ## Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+  ## tend to generate useless memory moves or have difficulties to minimize stack allocation
+  ## and our types might be large (Fp12 ...)
+  ## See: https://github.com/mratsim/constantine/issues/145
+  result.scalarMul_vartime(scalar, P)
+
+func `~*`*[F, G](
+      scalar: Fr or BigInt,
+      P: EC_ShortW_Aff[F, G],
+      T: typedesc[EC_ShortW_Jac[F, G] or EC_ShortW_Prj[F, G]]
+      ): T {.noInit, inline.} =
+  ## Elliptic Curve variable-time Scalar Multiplication
+  ##
+  ##   R <- [k] P
+  ##
+  ## This MUST NOT be used with secret data.
+  ##
+  ## This is highly VULNERABLE to timing attacks and power analysis attacks.
+  ##
+  ## Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+  ## tend to generate useless memory moves or have difficulties to minimize stack allocation
+  ## and our types might be large (Fp12 ...)
+  ## See: https://github.com/mratsim/constantine/issues/145
+  result.scalarMul_vartime(scalar, P)
+
+func `~*`*[F, G](
+      scalar: Fr or BigInt,
+      P: EC_ShortW_Aff[F, G],
+      ): EC_ShortW_Jac[F, G] {.noInit, inline.} =
+  ## Elliptic Curve variable-time Scalar Multiplication
+  ##
+  ##   R <- [k] P
+  ##
+  ## This MUST NOT be used with secret data.
+  ##
+  ## This is highly VULNERABLE to timing attacks and power analysis attacks.
+  ##
+  ## Out-of-place functions SHOULD NOT be used in performance-critical subroutines as compilers
+  ## tend to generate useless memory moves or have difficulties to minimize stack allocation
+  ## and our types might be large (Fp12 ...)
+  ## See: https://github.com/mratsim/constantine/issues/145
+  result.scalarMul_vartime(scalar, P)