Segwit stands for segregated witness and is a backwards-compatible upgrade or "soft fork" that activated on the Bitcoin network in August of 2017. While the activation was controversial, the features of this technology require some explanation. In this chapter, we’ll explore how Segwit works, why it’s backwards compatible and what Segwit enables.
As a brief overview, Segwit did a multitude of things:
-
Block size increase
-
Transaction malleability fix
-
Introduced Segwit versioning for clear upgrade paths
-
Quadratic hashing fix
-
Offline wallet fee calculation security
It’s not entirely obvious what Segwit is without looking at how it’s implemented. We’ll start by examining the most basic type of Segwit transaction, pay-to-witness-pubkey-hash.
Pay to witness pubkey hash (p2wpkh) is one of four types of Scripts defined by Segwit in BIP0141 and BIP0143. This is a smart contract that acts a lot like pay-to-pubkey-hash and is named similarly for that reason. The main change from p2pkh is that the data for the ScriptSig is now in the Witness field. The rearrangement is to fix transaction malleability.
Transaction Malleability is the ability to change the transaction’s ID without altering the transaction’s meaning. Mt. Gox CEO Mark Karpeles cited transaction malleability as the reason why his exchange was not allowing withdrawals back in 2013.
Malleability of the ID is an important consideration when creating Payment Channels, which are the atomic unit of the Lightning Network. A malleable transaction ID makes much more difficult the safe creation of Payment Channel transactions.
The reason why transaction malleability is a problem at all is because the transaction ID is calculated from the entire transaction. The ID of the transaction is the hash256 of the transaction. Most of the fields in a transaction cannot be changed without invalidating the transaction’s signature (and thus the transaction itself), so from a malleability standpoint, these fields are not a problem.
The one field that does allow for some manipulation without invalidating the Signature is the ScriptSig field on each input. The ScriptSig is emptied before creating the signature hash (see [chapter_tx]), so it’s possible to change the ScriptSig without invalidating the signature. Also, as we learned in [chapter_elliptic_curve_cryptography], signatures contain a random component. This means that two different ScriptSigs can essentially mean the same thing but be different byte-wise.
This makes the ScriptSig field malleable, that is, able to be changed without changing the meaning, and means that the entire transaction, and the Transaction ID is malleable. A malleable transaction ID means that any dependent transactions (that is, a transaction spending one of the malleable transaction’s outputs), cannot be constructed in a way to guarantee validity. The previous transaction hash is uncertain, so the dependent transaction’s transaction input field cannot be guaranteed to be valid.
This is not usually a problem as once a transaction enters the blockchain, the transaction ID is fixed and no longer malleable (at least without finding a proof-of-work!). However, with Payment Channels, there are dependent transactions created before the funding transaction is added to the blockchain.
Transaction malleability is fixed by emptying the ScriptSig field and putting the data in another field that’s not used for ID calculation. For p2wpkh, the signature and pubkey are the items from ScriptSig, so those get moved to the Witness field, which is not used for ID calculation. This way, the transaction ID stays stable as the malleability vector disappears. The Witness field and the whole Segwit serialization of a transaction is only sent to nodes that ask for it. In other words, old nodes that haven’t upgraded to Segwit don’t receive the Witness field and don’t verify the pubkey and signature.
If this sounds familiar, it should. This is similar to how p2sh works ([chapter_p2sh]) in that newer nodes do additional validation that older nodes do not and is the basis for why Segwit is a soft fork (backwards-compatible upgrade) and not a hard fork (backwards-incompatible upgrade).
To understand Segwit, it helps to look at what the transaction looks like when sent to an old node (Pay-to-witness-pubkey-hash as seen by pre-BIP0141 software) versus a new node (Pay-to-witness-pubkey-hash as seen by post-BIP0141 software):
The difference between these two serializations is that the latter transaction (Segwit serialization) has the marker, flag and Witness fields. Otherwise, the two transactions look similar. The reason the transaction ID not malleable is because first serialization is used for calculating the transaction ID.
The Witness in p2wpkh has the Signature and Pubkey as its two elements. These will be used for validation for upgraded nodes only.
The ScriptPubKey for a p2wpkh is OP_0 <20-byte hash>.
The ScriptSig, as seen in both serializations is empty.
The combined Script is Pay-to-witness-pubkey-hash (p2wpkh) ScriptPubKey:
The processing of the combined Script starts like p2wpkh Start:
OP_0 pushes a 0 on the stack (p2wpkh Step 1):
The 20-byte hash is an element, so it’s pushed to the stack (p2wpkh Step 2):
At this point, older nodes will stop as there are no more Script instructions to be processed.
Since the top element is non-zero, this will be counted as a valid Script.
This is very similar to p2sh ([chapter_p2sh]) in that older nodes cannot validate further.
Newer nodes, however, have a special Segwit rule much like the special rule for p2sh (see [chapter_p2sh]).
Recall that with p2sh, the exact Script sequence of <redeem script> OP_HASH160 <hash> OP_EQUAL trigger a special rule.
In the case of p2wpkh, the Script sequence is OP_0 <20-byte hash>.
When that Script sequence is encountered, the pubkey and signature from the Witness field and the 20-byte hash are added to the instruction set in exactly the same sequence as p2pkh.
Namely: <signature> <pubkey> OP_DUP OP_HASH160 <20-byte hash> OP_EQUALVERIFY OP_CHECKSIG.
p2wpkh Step 3 shows the state that is encountered next:
The rest of the processing of p2wpkh is the same as the processing of p2pkh as seen in [chapter_script].
The end state is a single 1 on the stack if and only if the 20-byte hash is the hash160 of the pubkey and the signature is valid (p2wpkh Step 4):
For an older node, processing stops at <20-byte hash> 0, as older nodes don’t know the special Segwit rule.
Only upgraded nodes do the rest of the validation, much like p2sh.
Note that less data is sent over the network to older nodes.
Also, nodes are given the option of not having to download (and hence not verify) transactions that are X blocks old if they don’t want to.
In a sense, the signature has been witnessed by a bunch of people and a node can choose to trust that this is valid instead of validating directly if it so chooses.
Furthermore, this is a special rule for Segwit Version 0.
Segwit Version 1 can have a completely different processing path.
<20-byte hash> 1 could be the special Script sequence that triggers a different rule.
Upgrades of Segwit can introduce Schnorr Signatures, Graftroot or even a different scripting system altogether like Simplicity.
Segwit gives us a clear upgrade path.
Software that understands how to validate Segwit Version X will validate such transactions, but software that isn’t aware of Segwit Version X simply processes only up to the point of the special rule.
P2wpkh is great, but unfortunately, this is a new type of Script and older wallets cannot send Bitcoins to p2wpkh ScriptPubKeys. P2wpkh uses a new address format called bech32, whose ScriptPubKeys older wallets don’t know how to create.
The Segwit authors found an ingenious way to make Segwit backwards compatible by using p2sh. We can "wrap" p2wpkh inside a p2sh. This is called "nested" Segwit as the Segwit Script is nested in a p2sh RedeemScript.
A p2sh-p2wpkh address is a normal p2sh address, but the RedeemScript is OP_0 <20-byte hash>, or the ScriptPubKey of p2wpkh.
As with p2wpkh, different transactions are sent to older nodes (Pay-to-script-hash-pay-to-witness-pubkey-hash (p2sh-p2wpkh) to pre-BIP0141 software) versus newer nodes (p2sh-p2wpkh to post-BIP0141 software):
The difference versus p2wpkh is that the ScriptSig is no longer empty. The ScriptSig has a RedeemScript which is equal to the ScriptPubkey in p2wpkh. As this is a p2sh, the ScriptPubKey is the same as any other p2sh. The combined Script looks like p2sh-p2wpkh ScriptPubKey is the same as a normal p2sh ScriptPubKey:
We start the Script evaluation like p2sh-p2wpkh Start:
Notice that the instructions to be processed are exactly what triggers the p2sh Special rule. The RedeemScript goes on the stack (p2sh-p2wpkh Step 1):
The OP_HASH160 will turn the RedeemScript’s hash (p2sh-p2wpkh Step 2):
The hash will go on the stack and we then get to OP_EQUAL (p2sh-p2wpkh Step 3):
At this point, if the hashes are equal, pre-BIP0016 nodes will simply mark the input as valid as they are unaware of the p2sh validation rules.
However, post-BIP0016 nodes recognize the special Script sequence for p2sh, so the RedeemScript will then be evaluated as Script instructions.
The RedeemScript is OP_0 <20-byte hash>, which is the same as the ScriptPubKey for p2wpkh.
This makes the Script state look like p2sh-p2wpkh Step 4:
This should look familiar as this is the state that p2wpkh starts with.
After OP_0 and the 20-byte hash we are left with p2sh-p2wpkh Step 5:
At this point, pre-Segwit nodes will mark this input as valid as they are unaware of the Segwit validation rules. However, post-Segwit nodes will recognize the special Script sequence for p2wpkh. The signature and pubkey from the Witness field along with the 20-byte hash will add the p2pkh instructions (p2sh-p2wpkh Step 6):
The rest of the processing is the same as p2pkh ([chapter_script]). Assuming the signature and pubkey are valid, we are left with p2sh-p2wpkh End:
As you can see, a p2sh-p2wpkh transaction is backwards compatible all the way to before BIP0016. A node pre-BIP0016 would consider the Script valid once the RedeemScripts were equal and a post-BIP0016, pre-Segwit node would consider the Script valid at the 20-byte hash. Both would not do the full validation and would accept the transaction. A post-Segwit node would do the complete validation, including checking the signature and pubkey.
|
Note
|
Can Anyone Spend Segwit Outputs?
Detractors to Segwit have referred to Segwit outputs as "anyone can spend". This would be true if the Bitcoin community rejected Segwit. In other words, if an economically significant part of the Bitcoin community refused to do the Segwit validation and actively split from the network by accepting a transaction that’s not Segwit valid, the outputs would have been anyone can spend. Due to a variety of economic incentives, Segwit was activated on the network, there was no network split, a lot of Bitcoins are now locked in Segwit outputs and Segwit transactions are validated per the soft-fork rules by the vast economic majority of nodes. We can now say confidently that the detractors were wrong. |
The first change we’re going to make is to the Tx class where we need to mark whether the transaction is Segwit or not:
link:code-ch13/tx.py[role=include]Next, we change the parse method depending on the serialization we receive.
class Tx:
...
link:code-ch13/tx.py[role=include]-
To determine whether we have a Segwit transaction or not, we look at the fifth byte. The first four are version, the fifth is the Segwit marker.
-
The fifth byte being 0 is how we tell that this transaction is Segwit (this is not fool-proof, but is what we’re going to use). We use different parsers depending on whether it’s Segwit.
-
Put the stream back to the position before we examined the first 5 bytes.
We’ve moved the old parse method to parse_legacy.
Here’s a parser for the Segwit serialization.
class Tx:
...
link:code-ch13/tx.py[role=include]-
There are two new fields, one of them is the Segwit marker.
-
The last new field is Witness, which contains items for each input.
We code the corresponding changes to the serialization methods:
class Tx:
...
link:code-ch13/tx.py[role=include]-
What used to be called
serializeis nowserialize_legacy. -
The Segwit serialization adds the markers.
-
The Witness is serialized at the end.
In addition we have to change the hash method to use the legacy serialization, even for Segwit transactions as that will keep the transaction ID stable.
class Tx:
...
link:code-ch13/tx.py[role=include]The verify_input method requires a different z for Segwit transactions.
The Segwit transaction signature hash calculation is specified in BIP0143.
In addition, the Witness field is passed through to the Script evaluation engine.
class Tx:
...
def verify_input(self, input_index):
tx_in = self.tx_ins[input_index]
script_pubkey = tx_in.script_pubkey(testnet=self.testnet)
if script_pubkey.is_p2sh_script_pubkey():
instruction = tx_in.script_sig.instructions[-1]
raw_redeem = int_to_little_endian(len(instruction), 1) + instruction
redeem_script = Script.parse(BytesIO(raw_redeem))
if redeem_script.is_p2wpkh_script_pubkey(): # (1)
z = self.sig_hash_bip143(input_index, redeem_script) # (2)
witness = tx_in.witness
else:
z = self.sig_hash(input_index, redeem_script)
witness = None
else:
if script_pubkey.is_p2wpkh_script_pubkey(): # (3)
z = self.sig_hash_bip143(input_index) # (2)
witness = tx_in.witness
else:
z = self.sig_hash(input_index)
witness = None
combined_script = tx_in.script_sig + tx_in.script_pubkey(self.testnet)
return combined_script.evaluate(z, witness) # (4)-
This handles the p2sh-p2wpkh case.
-
BIP0143 signature hash generation code is in
tx.pyof this chapter’s code. -
This handles the p2wpkh case.
-
The Witness passes through to the evaluation engine so that p2wpkh can construct the right instructions.
We also define what a p2wpkh Script looks like in script.py.
link:code-ch13/script.py[role=include]
...
link:code-ch13/script.py[role=include]-
This is
OP_0 <20-byte-hash>. -
This checks if the current script is a p2wpkh ScriptPubKey.
Lastly, we need to implement the special rule in the evaluate method.
class Script:
...
def evaluate(self, z, witness):
...
while len(instructions) > 0:
...
else:
stack.append(instruction)
...
link:code-ch13/script.py[role=include]-
This is where we execute Witness Program version 0 for p2wpkh. We make a p2pkh combined Script from the 20-byte hash, signature and pubkey and evaluate.
While p2wpkh takes care of a major use case, we need something more flexible if we want more complicated Scripts like multisig. This is where p2wsh comes in. Pay-to-witness-script-hash is like p2sh, but with all the ScriptSig data in the Witness field instead.
As with p2wpkh, we send different data to pre-BIP0141 software (Pay-to-witness-script-hash as seen by pre-BIP0141 software) vs post-BIP0141 software (Pay-to-witness-script-hash as seen by post-BIP0141 software):
The ScriptPubKey for a p2wsh is OP_0 <32-byte hash>.
This sequence triggers another special rule.
The ScriptSig, as with p2wpkh, is empty.
When p2wsh is being spent, the combined Script looks like Pay-to-witness-script-hash (p2wsh) ScriptPubKey:
The processing of this Script starts similarly to p2wpkh (Figure #p2wsh_start and #p2wsh_step_1):
The 32-byte hash is an element, so is pushed to the stack (p2wsh Step 2):
As with p2wpkh, older nodes will stop as there are no more Script instructions to be processed. Newer nodes will recognize the special sequence and do additional validation by looking at the Witness field.
The Witness field for p2wsh in our case is a 2-of-3 multisig (p2wsh Witness):
The last item of the Witness is called the WitnessScript and must sha256 to the 32-byte hash from the ScriptPubKey. Note this is sha256, not hash256. Once the WitnessScript is validated by having the same hash value, the WitnessScript is interpreted as Script instructions put into the instruction set. The Witness Script looks like p2wsh Witness Script:
The rest of the Witness field is put on top to produce this instruction set (p2wsh Step 3):
As you can see, this is a 2-of-3 multisig much like what was explored in [chapter_p2sh] (p2wsh Step 4).
If the signatures are valid, we end like p2wsh Step 5:
The WitnessScript is very similar to the RedeemScript in that the sha256 of the serialization is addressed in the ScriptPubKey, but only revealed when being spent. Once the sha256 of the WitnessScript is found to be the same as the 32-byte hash, the WitnessScript is interpreted as Script instructions and added to the instruction set. The rest of the Witness is then put on the instruction set as well, producing the final set of instructions to be evaluated. p2wsh is particularly important as unmalleable multisig transactions are required for creating bi-directional Payment Channels for the Lightning Network.
Like p2sh-p2wpkh, p2sh-p2wsh is a way to make p2wsh backward-compatible. These transactions are sent to older nodes (Pay-to-script-hash-pay-to-witness-script-hash (p2sh-p2wsh) to pre-BIP0141 software) vs newer nodes (p2sh-p2wsh to post-BIP0141 software):
As with p2sh-p2wpkh, the ScriptPubKey is indistinguishable from any other p2sh and the ScriptSig is only the RedeemScript (p2sh-p2wsh ScriptPubKey):
We start the p2sh-p2wsh in exactly the same way that p2sh-p2wpkh starts (p2sh-p2wsh Start).
The RedeemScript is pushed to the stack (p2sh-p2wsh Step 1):
The OP_HASH160 will return the RedeemScript’s hash (p2sh-p2wsh Step 2):
The hash is pushed to the stack and we then get to OP_EQUAL (p2sh-p2wsh Step 3):
As with p2sh-p2wpkh, if the hashes are equal, pre-BIP0016 nodes will mark the input as valid as they are unaware of the p2sh validation rules.
However, post-BIP0016 nodes will recognize the special Script sequence for p2sh, so the RedeemScript will be interpreted as new Script instructions.
The RedeemScript is OP_0 <32-byte hash>, which is the same as the ScriptPubKey for p2wsh (p2sh-p2wsh RedeemScript).
This makes the Script state look like p2sh-p2wsh Step 4:
Of course, this is the exact same starting state as p2wsh (p2sh-p2wsh Step 5).
The 32-byte hash is an element, so is pushed to the stack (p2sh-p2wsh Step 6):
At this point, pre-Segwit nodes will mark this input as valid as they are unaware of the Segwit validation rules. However, post-Segwit nodes will recognize the special Script sequence for p2wsh. The Witness field (p2sh-p2wsh Witness) contains the Witness Script (p2sh-p2wsh Witness Script). The sha256 of the WitnessScript is checked against 32-byte hash and if equal, the WitnessScript is interpreted as Script and put into the instruction set (p2sh-p2wsh Step 7):
The Witness Script is a 2-of-3 multisig which is put into the instruction set (p2sh-p2wsh Step 7):
As you can see, this is a 2-of-3 multisig as in [chapter_p2sh]. If the signatures are valid, we end like p2sh-p2wsh End:
This makes p2wsh backwards compatible, allowing older wallets to send to p2sh ScriptPubKeys which they can handle.
The parsing and serialization are exactly the same as before.
The main changes have to do with verify_input in tx.py and evaluate in script.py.
class Tx:
...
def verify_input(self, input_index):
tx_in = self.tx_ins[input_index]
script_pubkey = tx_in.script_pubkey(testnet=self.testnet)
if script_pubkey.is_p2sh_script_pubkey():
instruction = tx_in.script_sig.instructions[-1]
raw_redeem = int_to_little_endian(len(instruction), 1) + instruction
redeem_script = Script.parse(BytesIO(raw_redeem))
if redeem_script.is_p2wpkh_script_pubkey():
z = self.sig_hash_bip143(input_index, redeem_script)
witness = tx_in.witness
elif redeem_script.is_p2wsh_script_pubkey(): # (1)
instruction = tx_in.witness[-1]
raw_witness = encode_varint(len(instruction)) + instruction
witness_script = Script.parse(BytesIO(raw_witness))
z = self.sig_hash_bip143(input_index, witness_script=witness_script)
witness = tx_in.witness
else:
z = self.sig_hash(input_index, redeem_script)
witness = None
else:
if script_pubkey.is_p2wpkh_script_pubkey():
z = self.sig_hash_bip143(input_index)
witness = tx_in.witness
elif script_pubkey.is_p2wsh_script_pubkey(): # (2)
instruction = tx_in.witness[-1]
raw_witness = encode_varint(len(instruction)) + instruction
witness_script = Script.parse(BytesIO(raw_witness))
z = self.sig_hash_bip143(input_index, witness_script=witness_script)
witness = tx_in.witness
else:
z = self.sig_hash(input_index)
witness = None
combined_script = tx_in.script_sig + tx_in.script_pubkey(self.testnet)
return combined_script.evaluate(z, witness)-
This takes care of p2sh-p2wsh
-
This takes care of p2wsh
We code a way to identify p2wsh in script.py:
link:code-ch13/script.py[role=include]
...
class Script:
...
link:code-ch13/script.py[role=include]-
OP_0 <32-byte script>is what we expect
Lastly, we handle the special rule for p2wsh:
class Script:
...
def evaluate(self, z, witness):
...
while len(instructions) > 0:
...
else:
stack.append(instruction)
...
link:code-ch13/script.py[role=include]-
The top element is the sha256 hash of the WitnessScript.
-
The second element is the Witness version, 0.
-
Everything but the WitnessScript is added to the instruction set.
-
The WitnessScript is the last item of the Witness.
-
The WitnessScript must hash to the sha256 that was in the stack.
-
Parse the WitnessScript and add to the instruction set.
Other improvements to Segwit include fixing the quadratic hashing problem through a different calculation of the signature hash.
A lot of the calculations for signature hash, or z, can be reused instead of requiring a new hash256 hash for each input.
The signature hash calculation is detailed in BIP0143 and can be seen in code-ch13/tx.py.
Another improvement is that by policy, uncompressed SEC pubkeys are now forbidden and only compressed SEC pubkeys are used for Segwit, saving space.