Update libsecp256k1 and python-bitcointx

[kristapsk]

See Simplexum/python-bitcointx#75.

This updates libsecp256k1 to latest v0.3.2 release and python-bitcointx to current master. Currently for testing purposes only, for merging into master we will need to wait for a new python-bitcointx release.

[kristapsk]

Rebased

[kristapsk]

Updated libsecp256k1 to v0.4.0

Simplexum/python-bitcointx#75 (comment)

[kristapsk]

Rebased

[kristapsk]

We should be more active in testing this and giving feedback. Nobody else will do and then next python-bitcointx release will never happen. I'm reckless and running this code on mainnet now. Even things like - "runned this for a while with yield generator, noticed no issues" would be helpful.

[roshii]

Testing libsecp256k1@v0.4.0 built with musl libc, python-bitcointx@master as of now. I shall report if anything out of the ordinary.

[kristapsk]

Rebased against current master, so that it's easy for people to test against other recent changes.

[roshii]

Testing libsecp256k1@v0.4.0 built with musl libc, python-bitcointx@master as of now. I shall report if anything out of the ordinary.

I have seen successful transactions since then, no issue to report so far.

[kristapsk]

python-bitcointx 1.1.4 has been released. https://github.com/Simplexum/python-bitcointx/releases/tag/python-bitcointx-v1.1.4

[roshii]

tACK 9410b9c

[AdamISZ]

Interesting, I'm installing right now and the libsecp tests are taking multiple minutes. I think they might have changed it. (I know we have the option to switch that off but just thought it was worth noting, did anyone else see that?)

[AdamISZ]

tACK 9410b9c

(Earlier deleted comment - I forgot to do pip install .[test], sorry)

Pretty minimal testing, just relevant parts of test suite, full installation (see comment above about secp tests), and starting a yieldgen script, but I believe it's enough.

[AdamISZ]

It might be worth adding some "flavour" comments here:

The leftover dependency in python-bitcointx, on openssl, was related to DER encoding of ECDSA signatures, i.e. consensus in bitcoin for a long time depended on applying the rules to what is and is not a valid serialized ECDSA signature, using the rules as applied in openSSL. OpenSSL dependency in Bitcoin Core was removed in 2019, it had been used for other stuff than ECDSA, e.g. BIP70 and some randomness stuff(?), but the important thing here is that, after BIP66 got activated, the only "non-strict" encoding of ECDSA signatures that Core would ever have to deal with, would be the historical ones pre-BIP66 activation, so they used a chunk of parsing code that they knew worked with that specific set of historical signatures (by testing it). GMax explains this much better than I could here. And python-bitcointx includes the same logic as of Simplexum/python-bitcointx#77.

Does it matter for Joinmarket? Kinda no, actually. Part of python-bitcointx's idiosyncracies, which are explained in detail in the README file of the repo, is that its VerifyScript function is not at all guaranteed to be compatible with the consensus layer of Bitcoin Core. We took the .. uhh .. "executive decision" to not actively deal with that issue (i.e. we call VerifyScript as a sanity check, more than anything - the input could still get rejected), and the reasoning is something like "there is no adversarial threat from invalid input signatures in coinjoins" - at least, not the kind of coinjoins we implement here. Coinjoins represent a smart contract with no intermediate state, there is just a 0 -> 1 state transition, with validity decided by the Bitcoin network. Well, something like that.

What we could do is call testmempoolaccept on the whole coinjoin, but it seems a little pointless given we're just about to broadcast it at that point, which includes the same validity-testing behaviour. On the other hand, there is no similar call for a specific input (is there?).

The openSSL dependency we have that still remains, is the one in twisted that is used for certificates. Presumably that can use openSSL 3, but I'm not sure of some details here - pyOpenSSL apparently supports both openSSL 1.1 and openSSL 3, but I'm not sure if what twisted uses, is a function of the twisted version, or if it should somehow be manually set (and indeed, whether it matters).

[jonasnick]

Interesting, I'm installing right now and the libsecp tests are taking multiple minutes. I think they might have changed it. (I know we have the option to switch that off but just thought it was worth noting, did anyone else see that?)

If you ran the libsecp make check tests then roughly more than doubling of time is expected due to the addition of "noverify_tests" and the ellswift module (which is turned on by default).