Initial Shplonk prover/verifier implementation #301
Conversation
There was a problem hiding this comment.
Thanks a lot. I think we can have an initial side-by-side implementation in this PR for review, and once we're happy with the status of the PR, we can add a last commit that moves the changes from shplonk to hyperkzg.
As long as we're taking this approach, would you mind adding shplonk to the PCS benchmarks?
src/provider/hyperkzg.rs
Outdated
| @@ -61,18 +61,20 @@ where | |||
| E::Fr: TranscriptReprTrait<E::G1>, | |||
| E::G1Affine: TranscriptReprTrait<E::G1>, // TODO: this bound on DlogGroup is really unusable! | |||
| { | |||
| fn compute_challenge( | |||
| /// TODO: write doc | |||
| pub fn compute_challenge( | |||
| com: &[E::G1Affine], | |||
| transcript: &mut impl TranscriptEngineTrait<NE>, | |||
| ) -> E::Fr { | |||
| transcript.absorb(b"c", &com.to_vec().as_slice()); | |||
There was a problem hiding this comment.
You can remove to_vec().as_slice(), the transcript already takes slices.
| com: &[E::G1Affine], | ||
| transcript: &mut impl TranscriptEngineTrait<NE>, | ||
| ) -> E::Fr { | ||
| transcript.absorb(b"c", &com.to_vec().as_slice()); | ||
| transcript.squeeze(b"c").unwrap() | ||
| } | ||
|
|
||
| /// TODO: write doc |
There was a problem hiding this comment.
I like the existing comment already :)
There was a problem hiding this comment.
Yes, so I do not focus on this only because we are going to merge Shplonk with HyperKZG eventually. The followup PR or "final" commit that you mentioned will for sure include these TODO resolvings
src/provider/hyperkzg.rs
Outdated
| @@ -88,14 +90,16 @@ where | |||
| transcript.squeeze(b"r").unwrap() | |||
| } | |||
|
|
|||
| fn batch_challenge_powers(q: E::Fr, k: usize) -> Vec<E::Fr> { | |||
| /// TODO: write doc | |||
There was a problem hiding this comment.
The inner comment should perhaps just be moved as a doc comment of this function. See also crate::spartan::powers, which could easily replace this function and its uses. We just need to move that function somewhere more accessible (the math module comes to mind).
src/provider/shplonk.rs
Outdated
| <NE::CE as CommitmentEngineTrait<NE>>::commit(ck, &polys[i]) | ||
| .comm |
There was a problem hiding this comment.
In general, you don't want to use to_affine() in a loop. Consider using batch_normalize to post process the projective points, see #290 for examples.
There was a problem hiding this comment.
Happy to do that, but can you please explain the reason? As batch_normalize seems to iterate over vector of G1 elements and invoke to_affine for each similarly:
fn batch_normalize(p: &[Self], q: &mut [Self::AffineRepr]) {
assert_eq!(p.len(), q.len());
for (p, q) in p.iter().zip(q.iter_mut()) {
*q = p.to_affine();
}
}
There was a problem hiding this comment.
batch_normalize is part of the Curve trait, and what you're showing is the default implementation that kicks in if the implementer choses to not optimize it. In fact, in most cases, the implementer does implement something better, and so it does not work that way for the curves that interest us. In fact it's optimized in pasta:
https://github.com/zcash/pasta_curves/blob/df67e299e6644c035a213dbad369be279276919c/src/curves.rs#L173-L207
and for halo2curves:
https://github.com/privacy-scaling-explorations/halo2curves/blob/9fff22c5f72cc54fac1ef3a844e1072b08cfecdf/src/derive/curve.rs#L583-L614
| E::Fr: Serialize + DeserializeOwned, | ||
| E::G1Affine: Serialize + DeserializeOwned, | ||
| E::G2Affine: Serialize + DeserializeOwned, | ||
| E::G1: DlogGroup<ScalarExt = E::Fr, AffineExt = E::G1Affine>, | ||
| <E::G1 as Group>::Base: TranscriptReprTrait<E::G1>, // Note: due to the move of the bound TranscriptReprTrait<G> on G::Base from Group to Engine | ||
| E::Fr: PrimeFieldBits, // TODO due to use of gen_srs_for_testing, make optional | ||
| E::Fr: TranscriptReprTrait<E::G1>, | ||
| E::G1Affine: TranscriptReprTrait<E::G1>, |
There was a problem hiding this comment.
It's really a pity we have to go through this declaration slog 😦 The good news is those bounds are auto-satisfied (e.g. E::Fr : Serialize comes through NE: NovaEngine<Scalar = E::Fr>), which makes the type rather usable.
There was a problem hiding this comment.
yes, our trait bounds are indeed quite complicated, so I just copy-pasted them from hyperkzg with minimal adaptations to satisfy compiler I think
src/provider/shplonk.rs
Outdated
| #[allow(clippy::needless_range_loop)] | ||
| Pi.par_iter_mut().enumerate().for_each(|(j, Pi_j)| { | ||
| *Pi_j = | ||
| point[point.len() - i - 1] * (polys[i][2 * j + 1] - polys[i][2 * j]) + polys[i][2 * j]; | ||
| }); |
There was a problem hiding this comment.
aka:
(0..Pi_len).into_par_iter().map(|j| {
point[ell - i - 1] * (polys[i][2 * j + 1] - polys[i][2 * j]) + polys[i][2 * j]
}).collect_into_vec(&mut Pi);There was a problem hiding this comment.
Thanks. Indeed cleaner and doesn't require adding clippy exception
| // K(x) = P(x) - Q(x) * D(a) - R(a), note that R(a) should be subtracted from a free term of polynomial | ||
| let K_x = Self::compute_k_polynomial(&batched_Pi, &Q_x, &D, &R_x, a); | ||
|
|
||
| // TODO: since this is a usual KZG10 we should use it as utility instead |
There was a problem hiding this comment.
Note the very important definition:
https://github.com/lurk-lab/arecibo/blob/7a5d7bf0e9be80b8b5c4d47bb99c67c2b058b35e/src/provider/non_hiding_kzg.rs#L224-L225
IOW, you're invoking this as a UVKZGPoly, but really, you can use UniPoly everywhere instead. I should perhaps not have made this type alias public.
There was a problem hiding this comment.
I assume this is true for Zeromorph as well, right?
src/provider/shplonk.rs
Outdated
| let q = HyperKZG::<E, NE>::get_batch_challenge(&pi.evals, transcript); | ||
| let q_powers = HyperKZG::<E, NE>::batch_challenge_powers(q, pi.comms.len()); | ||
|
|
||
| let R_x = UVKZGPoly::new(pi.R_x.clone()); |
There was a problem hiding this comment.
Same remark, this is a UniPoly morally speaking.
src/provider/shplonk.rs
Outdated
| let mut batched_eval = E::Fr::ZERO; | ||
| evals_i | ||
| .iter() | ||
| .zip_eq(q_powers.iter()) | ||
| .for_each(|(eval, q_i)| { | ||
| batched_eval += *eval * q_i; | ||
| }); |
There was a problem hiding this comment.
I believe this is UniPoly::ref_cast(evals_i).evaluate(q), eliminating the need for q_powers.
src/provider/shplonk.rs
Outdated
| let C_P = q_powers | ||
| .iter() | ||
| .zip_eq(pi.comms.iter()) | ||
| .fold(E::G1::identity(), |acc, (q_i, C_i)| acc + *C_i * q_i); |
There was a problem hiding this comment.
This looks like pi_comms.iter().rlc(q) I think.
|
My draft benchmarking experiments (with "hacked" parallelism - I'm looking forward review comments from @adr1anh about that) show that with Shplonk we should have slightly better prover and comparable verifier (to current HyperKZG): Despite the MSM elimination at verifier side, in shplonk we still have heavy |
I have an upcoming branch that may help with that (https://github.com/huitseeker/arecibo/tree/rayon-parscan : warning, very WIP) |
This PR adds Shplonk-based optimisation to HyperKZG PCS. For simplifying the reviewing process and focusing on correctness at first, It is implemented as a separate PCS (one more implementation of
EvaluationEngineTrait) and contains most of @adr1anh's suggestions kindly provided in both private discussions and during intermediate reviewing of draft version.Once this PR is merged, I'm planning to have some more follow-up PRs to solve #302, #304 and #303.