fix unconfirmed change note tracking in transactions

[borngraced]

This fixes the issue where change outputs weren't being tracked properly in the unconfirmed state, which could cause incorrect balance calculations and note availability issues KomodoPlatform/komodo-defi-framework#2273

[dimxy]

As I can see, the change output is added in KDF code. I hope this won't affect that code.
Maybe some test is needed in KDF to ensure that the change is calculated correctly

[borngraced]

I see. Thanks a lot..

[dimxy]

Without this code we would need to wait until the block is scanned, right?

[borngraced]

right

[borngraced]

with the case above, we need this fix in kdf directly after building txs..

[dimxy]

Looks like this store_sent_tx call stores the created transaction output in the walletdb.
Wouldn't the decrypt_and_store_transaction call do a similar thing?

[borngraced]

store_sent_tx - stores notes we created to send value to others, while decrypt_and_store_transaction calls store_received_tx(stores notes we can spend) internally

librustzcash/zcash_extras/src/wallet.rs

Lines 24 to 60 in d78f4df

	/// Scans a [`Transaction`] for any information that can be decrypted by the accounts in
	/// the wallet, and saves it to the wallet.
	pub async fn decrypt_and_store_transaction<N, E, P, D>(
	params: &P,
	data: &mut D,
	tx: &Transaction,
	) -> Result<(), E>
	where
	E: From<Error<N>>,
	P: consensus::Parameters,
	D: WalletWrite<Error = E>,
	{
	// Fetch the ExtendedFullViewingKeys we are tracking
	let extfvks = data.get_extended_full_viewing_keys().await?;
	let max_height = data.block_height_extrema().await?.map(|(_, max)| max + 1);
	let height = data
	.get_tx_height(tx.txid())
	.await?
	.or(max_height)
	.or_else(|| params.activation_height(NetworkUpgrade::Sapling))
	.ok_or(Error::SaplingNotActive)?;
	let outputs = decrypt_transaction(params, height, tx, &extfvks);
	if outputs.is_empty() {
	Ok(())
	} else {
	data.store_received_tx(&ReceivedTransaction {
	tx,
	outputs: &outputs,
	})
	.await?;
	Ok(())
	}
	}

[dimxy]

Aha, we need to add the change as a received output, to fix the balance

[shamardy]

Thanks for the fix! I have only one question!

[shamardy]

For unconfirmed transactions which this PR is for, will using the max height cause problems in decrypt_transaction? get_tx_height will return nothing as it's not in db so max height will be used.

[borngraced]

So using max_height would only cause issues during the Canopy network upgrade window and we're way past the Canopy grace period (ended late 2020) and all new transactions now uses 0x02 note version

librustzcash/zcash_primitives/src/sapling/note_encryption.rs

Lines 322 to 338 in e92443a

	if params.is_nu_active(Canopy, height) {
	let grace_period_end_height =
	params.activation_height(Canopy).unwrap() + ZIP212_GRACE_PERIOD;
	if height < grace_period_end_height && leadbyte != 0x01 && leadbyte != 0x02 {
	// non-{0x01,0x02} received after Canopy activation and before grace period has elapsed
	false
	} else if height >= grace_period_end_height && leadbyte != 0x02 {
	// non-0x02 received past (Canopy activation height + grace period)
	false
	} else {
	true
	}
	} else {
	// return false if non-0x01 received when Canopy is not active
	leadbyte == 0x01
	}

[shamardy]

As discussed in DM, it makes sense to use the max height since it's the next anticipated block which we wish the transaction to be in. We should remove this though

librustzcash/zcash_extras/src/wallet.rs

Lines 44 to 45 in 6aa3917

	.or_else(|| params.activation_height(NetworkUpgrade::Sapling))
	.ok_or(Error::SaplingNotActive)?;

and use unwrap_or_else instead of map here

librustzcash/zcash_extras/src/wallet.rs

Line 39 in 6aa3917

let max_height = data.block_height_extrema().await?.map(|(_, max)| max + 1);

[borngraced]

We should remove this though

Why? I mean height will be optional which isn't what we want.

[shamardy]

That's why I suggested to use unwrap_or_else, we use max height when it's none not NetworkUpgrade::Sapling. We should return an error if we can't get block_height_extrema, what do you think?

[borngraced]

ok that makes sense..we assume block_height_extrema shouldn't return an error except some internal error with db or syncing so it make sense to return an error

[dimxy]

So using max_height would only cause issues during the Canopy network upgrade window

AFAIK we do not support zcash privacy transactions (yet), right?
Only supporting PIRATE which is still on Sapling.

[borngraced]

Yep.. you mean shielded transactions.

[shamardy]

LGTM! Only 1 nit!

[shamardy]

@dimxy are you planning to do another review round? If not, I will merge this.

[dimxy]

@dimxy are you planning to do another review round? If not, I will merge this.

LGTM