Extend the way of verifying the transaction in the transaction pool

[liuchengxu]

Currently, when an extrinsic is submitted to the transaction pool, there is only one way to verify the transaction before actually importing it to the pool, which is using the runtime API validate_transaction.

substrate/client/transaction-pool/src/api.rs

Line 232 in ce449a5

runtime_api.validate_transaction(at, source, uxt, block_hash)

In Subspace, we have an extrinsic for submitting a fraud proof(FraudProof can have a non-trivial StorageProof), the execution proof check is an std only function, in that way, we have to add a new host function, but calling into the runtime to verify this function imposes some constraints:

• The size and complexity of the fraud proof(StorageProof + execution_proof) are naturally limited by the runtime/wasm environment and transaction/block size.
• By adding a new host function, you have to cross the runtime and client boundary to invoke the actual implementation under the hood, which is unnecessary for the verification purpose.

For this specific use case, when an extrinsic is submitted, we can first check if it's about submitting a fraud proof, if true, verify the fraud proof directly using the underlying verification function in the transaction pool, if the proof is invalid, we just drop it without having to call the runtime API validate_transaction, then the heaviest part of fraud proof verification can happen on the client only. So, nice to have some kind of hook around the current verify_transaction, which will be invoked before calling the runtime API validate_transaction.

Overall, it's beneficial to have an additional way to verify the transaction besides the only way of existing runtime API validate_transaction, which can be a universal enhancement to the transaction pool IMHO. The alternative way is to make a wrapper around the existing transaction pool to satisfy our need, but nice to have your opinion about this. If you think it's generally useful and can be integrated into Substrate, I'm happy to help and push it forward :D.

[bkchr]

Wrapping the validate_transaction of FullPool should directly give you support for what you want here:

substrate/client/transaction-pool/src/api.rs

Lines 129 to 164 in 73d1fc6

	fn validate_transaction(
	&self,
	at: &BlockId<Self::Block>,
	source: TransactionSource,
	uxt: graph::ExtrinsicFor<Self>,
	) -> Self::ValidationFuture {
	let (tx, rx) = oneshot::channel();
	let client = self.client.clone();
	let at = *at;
	let validation_pool = self.validation_pool.clone();
	let metrics = self.metrics.clone();
	async move {
	metrics.report(|m| m.validations_scheduled.inc());
	validation_pool
	.lock()
	.await
	.send(
	async move {
	let res = validate_transaction_blocking(&*client, &at, source, uxt);
	let _ = tx.send(res);
	metrics.report(|m| m.validations_finished.inc());
	}
	.boxed(),
	)
	.await
	.map_err(|e| Error::RuntimeApi(format!("Validation pool down: {:?}", e)))?;
	match rx.await {
	Ok(r) => r,
	Err(_) => Err(Error::RuntimeApi("Validation was canceled".into())),
	}
	}
	.boxed()
	}

[liuchengxu]

Yeah, I'm trying this alternative approach right now, thanks for the link! Just wondering if there is a chance to add a more natural way as an enhancement to the current transaction pool :(.

[bkchr]

What do you mean by more natural way? I mean it is already generic using this method.

[liuchengxu]

I mean it's nicer to have a built-in Hook in the TransactionPool trait, maybe a new type ValidateTransaction: ValidationTransaction;? something like that

substrate/client/transaction-pool/api/src/lib.rs

Line 177 in ce449a5

pub trait TransactionPool: Send + Sync {

// TBD
trait ValidateTransaction {
    fn pre_validate_transaction(&self);
    fn post_validate_transaction(&self);
}

That being said, people can easily inject their custom validation logic before and after running the current way of calling into the runtime api validate_transaction, without having to make a bunch of wrappers around the existing data structures like this https://github.com/subspace/subspace/blob/d496f959e5/crates/subspace-service/src/pool.rs (it's not yet finished, but compiles, you can see a lot of boilerplate code)

[burdges]

Any individual transaction is either valid or invalid, but we never demand this be the whole story. In fact, we multiple unfinished "transaction" types in Polkadot itself, like on-chain multi-sigs, and parachain candidate backing. Approval assignments too, and some places in approval & parachain rewards too, except they stay off chain because we do not require consensus upon the exact approval checker set.

[nazar-pc]

In Substrate consensus is typically validated in the client. The transaction Liu-Cheng is talking about is of a similar kind: it is finished and needs to be included in the block, but its validity can only be properly checked by the client, not by the runtime.

[nazar-pc]

I need to clarify that we don't technically need a client, but storage overlay and other things needed to verify fraud proofs do not compile to WASM right now, so we can't just call it in the runtime directly.

[burdges]

Alright, so the transaction itself is valid, but only makes (somewhat) falsifiable claims, and if disputed then later valid transactions can enact penalties or whatever.

We should not imho provide an abstraction for claims and disputes because afaik they're not cleanly abstractable, and really need consensus level optimizations.

As an crazy example, I've designed our approval rewards scheme to literally approximate the correct rewards under consensus, but without ever reaching consensus upon the inputs to the approximation. I doubt this specific idea works recursively using the different notion of consensus that the relay chain exports to parachains, but other ideas work, and a parachain could do its own internal consensus.

[nazar-pc]

It is not falsifiable. In Polkadot validators execute the whole block with PoV, in Subspace farmers execute just one extrinisc from the block and only if there is a fraud proof presented (so they don't execute anything in optimistic case and do very limited computation when fraud proof is presented). It is either correct (state transition for a particular extrinsic) or it isn't. This is a non-interactive process.