Add frost-secp256k1-tr crate (BIP340/BIP341)

[zebra-lucky]

• add frost-secp256k1-tr crate with support of BIP0340, BIP0341

[conradoplg]

Thank you, this is great. I'll take a closer look soon and may make some adjustments.

[codecov]

Codecov Report

Attention: Patch coverage is 74.30684% with 139 lines in your changes are missing coverage. Please review.

Project coverage is 81.07%. Comparing base (5f4ac6e) to head (b360496).
Report is 4 commits behind head on main.

❗ Current head b360496 differs from pull request most recent head c63a3ca. Consider uploading reports for the commit c63a3ca to get more accurate results

Files	Patch %	Lines
frost-core/src/traits.rs	16.92%	54 Missing ⚠️
frost-secp256k1-tr/src/lib.rs	85.20%	46 Missing ⚠️
frost-secp256k1-tr/src/keys/dkg.rs	0.00%	21 Missing ⚠️
frost-secp256k1-tr/src/keys/repairable.rs	59.09%	18 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #584      +/-   ##
==========================================
- Coverage   82.18%   81.07%   -1.11%     
==========================================
  Files          31       34       +3     
  Lines        3188     3694     +506     
==========================================
+ Hits         2620     2995     +375     
- Misses        568      699     +131     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[conradoplg]

After a closer look, here are some things that we'd prefer to change before merging this PR:

• The current approach changes the key (to ensure a even Y in the public key) during the signing operation. This makes sense if you follow BIP-340 directly. However, with FROST, I think it would be better and simpler to change the key during share generation, i.e. if the public key as an odd Y, negate all the shares, verifying shares, etc. This would make the PR much less invasive since the signing code would not change. You can take a look at this PR from the reddsa crate which does basically the same thing
• The PR also tweaks the key per BIP-341 with an empty merkle root. As I understand this wouldn't work in practice since the merkle root needs to be specified. I think this can be accomplished separately, using the frost-rerandomized crate - I think that the Taproot tweak is basically equal to rerandomization, but with a specific tweak rather than a random one.

I realize these might be big changes. Let me know if this makes sense and if you are able to work on this. Otherwise we can eventually incorporate these changes to the PR, but might not be able to get that done shortly.

Also - what did you use to generate the test vectors? Do you have a separate reference implementation for this? I'd like to take a look if possible.

[zebra-lucky]

I realize these might be big changes. Let me know if this makes sense and if you are able to work on this. Otherwise we can eventually incorporate these changes to the PR, but might not be able to get that done shortly.

I'll try to add changes to generation process to use only even VerifyingKey

On the "emtpy merkle root" -- need to do more investigations, but current code is working. As a proof I'll add a link on the repo with testing code to sign tx (in the next post).

Also - what did you use to generate the test vectors? Do you have a separate reference implementation for this? I'd like to take a look if possible.

On testing vectors -- I just added the values from the working code so the tests wouldn't fail.

[zebra-lucky]

Some proofs that the current code works

Testing code on the Testnet sign-tx-frost https://github.com/zebra-lucky/sign-tx-frost

To use there is a need to fix paths in the Cargo.toml

frost-core = { version = "1.0.0-rc.0", features = ["serde"], path = "../frost.copy/frost-core" }
...
frost-secp256k1-tr = { version = "1.0.0-rc.0", features = ["serde"], path = "../frost.copy/frost-secp256k1-tr"  }

• generate command outputs to stdout and this can be saved in testdata.json
• address command show address on generated VerifyinKey in the testdata.json
• sendtoaddress allow sign and get hex of output transaction with arguments in the form:

sendtoaddress tb1pqkgsz274gjnkdxp7v9rpzwqtqtjacjp5t2mz2vaqu2r6qln8qsesve6uez 1000 02000000000101bf2d2d426...

Where last part is a hex of transaction which makes output on generated address.

Some dirty but working code.

Latest transactions on the Testnet:
https://live.blockcypher.com/btc-testnet/address/tb1pqkgsz274gjnkdxp7v9rpzwqtqtjacjp5t2mz2vaqu2r6qln8qsesve6uez/

[mimoo]

Just FYI we're testing this library on Bitcoin and getting flaky results (sometimes signature fails). I'm investigating on our side to understand if it's us or if there's some issue here (my current intuition is that the issue is here).

Maybe the issue that I'm seeing is because the negation happens based on key_package.verifying_key().y_is_odd(), but it should happen based on the tweaked verifying/public key instead (i.e. tweaked_public_key(key_package.verifying_key(), &[]).y_is_odd()).

In addition, I had another issue which led me to spend quite some time reviewing this PR, so here are some notes and some reverse-engineering notes (for others who are trying to understand this like me):

1. it seems like there's no way to tweak the merkle tree from public facing function
2. half of the changes seems to be about conditional negation of "stuff". IIUC, it is needed to ensure that when the tweaked public key starts with 0x03 it is forced to start with 0x02 (as 0x03 prefix is not supported by taproot).
3. Now I'm wondering, in Bitcoin how do they ensure that every tweak leads to a pubkey that starts with 0x02? -> ah, BIP 341 says "Note that because tweaks are applied to 32-byte public keys, taproot_tweak_seckey may need to negate the secret key before applying the tweak." so that makes sense with my previous comment!
4. the term "tweak" is overused throughout the codebase (even sometimes in tests to mean tampering). IMO it shouldn't be used in most of these places as it collides with "tweak" of pubkey in taproot (it confused me a lot when I started reviewing this PR). Same comment with is_need_tweaking which imo should be renamed (e.g. bitcoin_spec, bitcoin_compat, etc.)
5. I initially thought that each participant would need to tweak their private share, but no need! The aggregator can do it at the end (z = k + s * c becomes z = k + (s + tweak) * c = (k + s * c) + (tweak * c) so tweak * c is added at the end)
6. It's not clear how to use the taproot lib with the current types, as we don't know if a pubkey is tweaked or untweaked. Things just work magically because verify will end up calling tweaked_public_key. I had to understand all the code in this PR to manage to fix another issue I had: I didn't get I had to tweak the pubkey on my side if I wanted to verify a signature using a different library than frost). A such I would really recommend not inserting magic logic in the verify function but rather getting a tweaked pubkey out (maybe verifying_key() could fail with the taproot version, and you'd be forced to use a tweaked_verifying_key(tweak: &[u8]) which would be unimplemented in the trait?

Nevermind, I wrote my understanding here: https://hackmd.io/li6ByhmfQUucZOqdboF1-g?view

I'm still unsure as to why we're seeing issues on our side atm.

[mimoo]

I fixed all the comments I pointed out in zebra-lucky#1 and everything now works on our end!

(bonus, the art that helped me find the issues)

[zebra-lucky]

Nevermind, I wrote my understanding here: https://hackmd.io/li6ByhmfQUucZOqdboF1-g?view

Thanks!, I'll read it to realize.

[zebra-lucky]

I am looking at the changes proposed in zebra-lucky#1

[zebra-lucky]

Merged fix on tweaked public key processing from @mimoo.
zebra-lucky#1

I'll add DKG/trusted dealer code which generates only
even VerifyingKey in a few hours.

[zebra-lucky]

Some problems on tests::batch::check_batch_verify arised, need more time to resolve.

[mimoo]

I'll add DKG/trusted dealer code which generates only even VerifyingKey

actually that might be overkill since all the checks must happen anyway no?

[zebra-lucky]

actually that might be overkill since all the checks must happen anyway no?

yep... thinking on that...

currently stuck on batch test... signing with SigningKey...

[zebra-lucky]

Added:

• Some more fixes on tweaked public key use after fix taproot crate zebra-lucky/frost#1 merge
• Renamed fn with more consistent names
• add DKG vector test for frost-secp256k1-tr (as in recent commits for other curves)

There is a question about only even VerifyingKey generation with DKG/trusted dealer.
This code is done, but tweaked public key can independently be even or odd.
I'll try to analyse tomorrow, but seems no overall code simplification can be done.
But maybe some refactoring on current code should be done.

Note about merkle root (as in BIP341 code example):

def taproot_sign_key(
    script_tree,
    internal_seckey,
    hash_type,
    bip340_aux_rand
):
    if script_tree is None:
        h = bytes()
...

For key path spending script_tree is absent, so empty bytes value set as h

[zebra-lucky]

p.s.: rebased PR branch on current main branch

[0xBEEFCAF3]

Should the merkel root be empty here?
i.e does this code assume that we only spend from the default key path? Not a tapscript path?

[zebra-lucky]

Possibly my understanding is wrong, but in BIP341, in
code example for taproot_sign_key I've seen next:

    if script_tree is None:
        h = bytes()  # empty bytes for merkle root

In the case when script path is not used (used only key path),
seems empty bytes for merkle root is correct.

[mimoo]

yeah this code only works with empty commitment, and it works fine for us (https://github.com/sigma0-xyz/zkbitcoin). I think it's a good first step, but it'd be nice to expose that feature later

[zebra-lucky]

Thank you. I'll look to code in zkbitcoin in a few days.

[0xBEEFCAF3]

In the case when script path is not used (used only key path),

Ah yes you're correct. In the case of a script path spend (non-frost spend) the spender can tweak the verifying key outside the context of this library.

[zebra-lucky]

Thank you. I'll look to code in zkbitcoin in a few days.

Only looking in the current day...

[Polybius93]

Hey all!

I hope I it's okay to post this here, but let me know if I should ask this somewhere else, or make an issue. My question about signing Schnorr that I think might be related to this PR.

I'm encountering an issue when broadcasting a PSBT transaction signed using FROST for aggregated Schnorr signatures. While individual Schnorr signatures work correctly, the aggregated signature fails with the following error:

"sendrawtransaction RPC error: {"code":-26,"message":"non-mandatory-script-verify-flag (Invalid Schnorr signature)"}

We generate a hash from the PSBT's unsigned transaction and sign it using a keypair. This process completes without errors.

let hash = SighashCache::new(&psbt.unsigned_tx.clone())
            .taproot_script_spend_signature_hash(
                0,
                &Prevouts::All(&[tx_out.clone()]),
                ScriptPath::with_defaults(&multisig_script),
                SchnorrSighashType::Default,
            )
            .unwrap_throw();

let sig = secp.sign_schnorr(
            &Message::from_slice(&hash).unwrap_throw(),
            &keypair,
);

When using FROST to sign the transaction, we provide the same hash as the message parameter for the SigningPackage.

let signing_package =
                frost::SigningPackage::new(signing_state.signing_commitments.clone(), hash);

The process creates an aggregated signature, but broadcasting the transaction results in the above error.

Given the transition from successful individual signing to the failure of the FROST aggregated signature, is there a specific aspect of message handling or signature aggregation with FROST that needs to be adjusted for these transactions? Any insights or suggestions on resolving the "Invalid Schnorr signature" error would be greatly appreciated.

Also I'm wondering if I need the _tr library for the signing process I'm doing, or if it would be enough to just use the frost_secp256k1 library?

Thanks!

[zebra-lucky]

I'm encountering an issue when broadcasting a PSBT transaction signed using FROST for aggregated Schnorr signatures. While individual Schnorr signatures work correctly, the aggregated signature fails with the following error:

Hello!

Need to clarify details. On the first commits to the this PR there was
a problems with combinations of pubkey/tweaked pubkey,
which lead to invalid sig. Currently it's seems to be fixed with @mimoo help.

Do you use the latest commit of frost-secp256k1-tr b360496?

There is some testing code on the testnet to check how it's works:
https://github.com/zebra-lucky/sign-tx-frost

[zebra-lucky]

Also I'm wondering if I need the _tr library for the signing process I'm doing, or if it would be enough to just use the frost_secp256k1 library?

frost-secp256k1 is a general on a curve, frost-secp256k1-tr adds
some specifics from BIP340/341

[MatthewLM]

@conduition @zebra-lucky I've not experienced any further issues using 20c2c98, thank you for the fix. I've been able to create successful Taproot key-path and script-path spends. I look forward to seeing this finished and merged in.

Given Taproot support, I'm curious about deriving unhardened BIP32 keys. Clearly hardened derivation is not possible, except perhaps by modifying the scheme somehow. A straight-forward method for unhardened derivation might be to first derive a master chaincode from the participant's public key shares. Then the same tweaks can be applied to the group key, public share and private shares.

I wonder if there are any security implications to deriving BIP32 keys this way though. I don't see how there would be but I know there might be some nuance that I'm unaware of.

[conduition]

@MatthewLM Intriguing idea. I hadn't considered BIP32 support. Yes, hardened derivation wouldn't be possible without some special DKG-like ritual, which would probably be more trouble than it's worth (At that point you could just run the DKG again to make a fresh FROST key).

A straight-forward method for unhardened derivation might be to first derive a master chaincode from the participant's public key shares. Then the same tweaks can be applied to the group key, public share and private shares.

This approach should be safe. The chain code is public data (public in the sense of "it reveals nothing that would help an observer forge signatures"), so deriving it from the FROST group's public verification shares should be acceptable. Group members would not be concerned that the chaincode was chosen unfairly, because each member is presumed to have participated in the DKG process which produced the verification shares (and thus the chain code).

There are down-sides to that approach though. If the full set of verification shares must be exposed for any reason (e.g to prove each signer's membership in the group), this would also result in exposing the full extended public key for the group.

Also, this approach would not be practically compatible with Verifiable Secret Resharing (VSR) or a Repairable Threshold Scheme, because those techniques involve adding or otherwise changing the set of shares. Doing so would also change the chain code derived from those shares, and thus the group extended pubkey would need to change. I imagine many use cases would prefer that not to happen.

I have a few alternative suggestions for how to combine FROST and BIP32, but i think for the sake of scope-reduction we should keep this PR limited to a single static group key. Additive tweaks on signing shares are easy for downstream callers to add themselves, by using the internals flag to access and modify shares directly. This PR is a lower-level prereq for taproot compatibility, so we should avoid biting off more than we (or the repository maintainers) can chew here.

[MatthewLM]

@conduition Thanks for your response. I agree with what you say. The risk of leaking all of the public shares could perhaps be mitigated if membership can be proved with zero-knowledge. Deriving the chaincode from the public shares would be trivial but I agree in some (perhaps many) cases it would not be desirable. A chain-code could be separately generated between all participants through an extension of the DKG.

I agree that it complicates this PR, so maybe I could create a discussion on this repo for BIP32 considerations?

[conduition]

I agree that it complicates this PR, so maybe I could create a discussion on this repo for BIP32 considerations?

I think that would be a swell idea 😄

[MatthewLM]

@conduition I started a discussion here: #636

[conduition]

@zebra-lucky I realized we were still serializing taproot signatures as 65-byte arrays with compressed R points. To be compatible with BIP340, callers would need to serialize and manually trim that leading byte. For usability, i have submitted a PR which fixes this, ensuring signatures are always computed with even R points, and serialized as BIP340-compatible 64-byte arrays with x-only R points. zebra-lucky#5

[zebra-lucky]

I'm sorry for a delayed reaction.
Lookin to code changes. Need to run tests.

[conduition]

Not delayed at all, man. Thanks very much for your quick attention 😄

[sosaucily]

Hi @zebra-lucky and @conduition, I'm just following up on this thread as it's been quiet for a while. I see This library released a 2.0.0-rc recently, and I wonder if this feature might be part of the full 2.0.0 release. We're not really blocked as we're working off the branch, but it would be great to get merged into the upstream code.

Thanks for all the hard work on this one!

[mpguerra]

I see This library released a 2.0.0-rc recently, and I wonder if this feature might be part of the full 2.0.0 release.
@conradoplg @natalieesk what do you think?

[conduition]

@sosaucily I would love to get this into the 2.0.0 release. Unfortunately i'm not sure when this PR will be merged - I imagine the maintainers of the frost repo will probably want to request changes, such as a different API, additional documentation coverage or tests.

i also have an additional feature branch, taproot-bip32, based off of zebra-lucky:add-secp256k1-tr which adds BIP32 support to the frost-secp256k1-tr crate. See this discussion for more background. It would be nice to be able to PR that into this repo as well, so that FROST keys can be used in hierarchical and descriptor wallets.

[zebra-lucky]

Hi!

At the first I very thankful to the people who gave a ideas on the PR:

• @akhavr on the plan
• @mimoo on the perfect math and graphics
• @conduition on the perfect knowledge of Rust and the bictcoin
• developers of ZcashFoundation on the attention

need to think how to transfer PR code, to not do a new PR

[zebra-lucky]

need to think how to transfer PR code, to not do a new PR

my productivity currently is at very low, I'm sorry

[conradoplg]

Thanks everyone, I really appreciate the work that went on this.

I want to get another look into this and I might do some refactorings to avoid changing the frost_core code so much. I think it's unlikely that this will make into the 2.0.0 release, but we'll probably do a 2.1.0 release not much time after that and this might be included then.

Don't worry about the conflicts, I'll solve them.

[ChristopherA]

We are hosting our send FROST library implementors workshop on September 18th https://developer.blockchaincommons.com/frost/meeting2/

So far scheduled are some discussion from bitcoin devs on their FROST implementations in C++. One of the critical issues to me are the issues of x-only pubkey compatibility and the tweaked signatures for use on-chain. All of Blockchain Common's own code base has been moving from C++ to Rust and we'd prefer a rust implementation.

Could we get someone from this team to participate, or even present a brief status update / roadmap on this PR? Contact me at ChristopherA@LifeWithAlacrity.com if you are interested.

Thanks!

[conduition]

@ChristopherA Thanks for organizing that 🙂 I've just sent you an email

@zebra-lucky It looks like it's not possible to transfer a pull request on github unfortunately, we'd have to close this PR and open a new one if we wanted to change the PR owner. Would you prefer I open a new PR from my fork of frost? that way you wouldn't need to be involved reviewing/merging stuff anymore

[zebra-lucky]

@zebra-lucky It looks like it's not possible to transfer a pull request on github unfortunately, we'd have to close this PR and open a new one if we wanted to change the PR owner. Would you prefer I open a new PR from my fork of frost? that way you wouldn't need to be involved reviewing/merging stuff anymore

Hi! Make it in your preferred way.

[conduition]

I have re-opened this PR here under my ownership: #730 - All further changes should be made to that PR's branch. CC @sosaucily

[mergify]

⚠️ The sha of the head commit of this PR conflicts with #730. Mergify cannot evaluate rules on this PR. ⚠️