CIP-0051? | Preserve submitter's ordering of transaction inputs

[szg251]

This CIP proposes a small change to the ledger specification allowing for optimisation in certain script validator executions.

[SebastienGllmt]

There was some talk about switching inputs from a set to a list for Babbage because set is messy to work with for Plutus. I'm not sure if this ended up being included in the list of changes that will make it into the update though

[WhatisRT]

The script has to properly validate the information it uses anyway, which should always require traversing the full input set. Relying on the ordering of the inputs seems very dangerous, since an attacker could use this to hide information from a script that isn't carefully validating its inputs. And checking that no such hidden information is present is pretty much exactly what is required to find the relevant inputs.

[michaelpj]

because the scripts in those transactions do not rely on pattern matching by position

This is false. They perfectly well can do, it's weird, but people write lots of weird stuff.

This CIP changes the semantics of the transaction context given to scripts in an observable fashion, and as such it need a new Plutus language version, I'm afraid (see #215).

[GeorgeFlerovsky]

When is the next Plutus language version planned to be released? Will the June hardfork be Plutus V1 or Plutus V2?

[michaelpj]

June HF will be V2, but it is too late for additional changes to go into that.

[GeorgeFlerovsky]

So, this CIP would have to wait for V3?

[michaelpj]

Yes.

[JaredCorduan]

There was some talk about switching inputs from a set to a list for Babbage because set is messy to work with for Plutus. I'm not sure if this ended up being included in the list of changes that will make it into the update though

No, this is not a part of the Babbage spec, for the reasons @WhatisRT mentioned above. At the very least, in needs more careful thought.

[michaelpj]

Oh, also it's wrong to describe this as "deterministic" transaction input ordering. The ordering is definitely deterministic, it's just not the same as the one in the transaction format. "Fixed" transaction ordering might be a better way to describe it.

[GeorgeFlerovsky]

The script has to properly validate the information it uses anyway, which should always require traversing the full input set. Relying on the ordering of the inputs seems very dangerous, since an attacker could use this to hide information from a script that isn't carefully validating its inputs. And checking that no such hidden information is present is pretty much exactly what is required to find the relevant inputs.

As @michaelpj mentioned (#231 (comment)), script inputs are already ordered and scripts can already rely on that ordering if they so choose. This CIP does not change that fact.

This CIP merely seeks to preserve the order of inputs that the transaction submitter sets when constructing the transaction, instead of the current approach of ordering transactions lexicographically by (1) their originating transaction's ID and (2) by their output index within that transaction. This would allow the transaction submitter to order the inputs in a way that better aligns to script logic, in the same way that this can currently be done with transaction outputs.

For example, consider a protocol with a transaction class that zips a variable-length (i.e. batchable) list of "pairs" [(A id,B id)] of inputs into a list of outputs [C id]. If input ordering is preserved, then the submitter can submit a transaction with the following inputs and outputs:

inputs = [A 1, B 1, A 3, B 3, A 5, B 5]
outputs = [C 1, C 3, C 5]

The script could iterate in parallel through the outputs (one element at a time) and inputs (two elements at a time), and verify that:

1. They are relevant to the protocol – i.e. they contain a token, datum, and/or script address that indicates that they belong to the protocol.
2. That they match each other – e.g. (A 1, B 1, C 1) should match, but both (A 1, B 2, C 1) and (A 1, A 1, C 1) shouldn't match.

This is quite straightforward and efficient to achieve if the script and transaction submitter can coordinate via the input/output ordering. It is much harder and inefficient (O(n log(n))?) if the inputs are ordered according to their original (TxId, OutputIndex), which is irrelevant in most contexts.

Your point about hidden information is well taken. However, I think this risk can be mitigated with a cheap, linear traversal through the rest of the inputs/outputs to check that they are not "relevant" to the transaction at hand. In any case, I don't think that this should prevent us from allowing the transaction submitter to set an input ordering that gets preserved.

[GeorgeFlerovsky]

Oh, also it's wrong to describe this as "deterministic" transaction input ordering. The ordering is definitely deterministic, it's just not the same as the one in the transaction format. "Fixed" transaction ordering might be a better way to describe it.

I suggest "Preserve submitter's ordering of transaction inputs"

[michaelpj]

Relying on the ordering of the inputs seems very dangerous, since an attacker could use this to hide information from a script that isn't carefully validating its inputs.

I also don't think this is too much of a concern: many applications don't care about what's going on with the other inputs, and can safely ignore them. And if you do care, well, you already have to look at them all, and that won't change.

[michaelpj]

Not advised, required. There is no way around it. It's a non-backwards-compatible change, it must go in a new version.

[JaredCorduan]

The ledger has to be able to translate the wire format from previous eras into new ones, so the CDDL will need to be:

transaction_body =
 { 0 : set<transaction_input> / [* transaction_input] ; inputs

And therefore the construction of the redeemers will need to be aware of which serialization option was used, so that they can point to the correct use sites (implicit lexicographic ordering for the first, explicit ordering in the second), assuming we don't want multiple types of redeemer pointers.

When transaction inputs which come across the wire as set<transaction_input> are deserialized, they will need to be sorted into the in-memory list representation so that the pointers can be ignorant of the wire format. This change should be mostly invisible to Plutus, since the translation from the ledger state to the Plutus transaction context already supplies a sorted list. You could, though, cook up weird scripts. The obvious contrived example is the script that returns true iff the inputs are sorted. This script would always return true for the old style transactions, but could be false for new ones.

[KtorZ]

@JaredCorduan I am not sure to follow your point about backward compatible when it comes to the CDDL. The format surely is backward compatible w.r.t the bytes on-the-wire, because in the end, a set<a> is merely a [ * a]. It is only the interpretation / semantic of that binary data which changes.

Or, are you thinking that there should be a way in the binary serialization to distinguish lexicographically-ordered inputs from submitter-ordered inputs?

[JaredCorduan]

Or, are you thinking that there should be a way in the binary serialization to distinguish lexicographically-ordered inputs from submitter-ordered inputs?

Correct. I feel an obligation to not change the semantics of previous era wire formats, since we will still be supporting them.

[WhatisRT]

I think there is a slippery slope with this CIP. Let's say this is implemented, what about for example preservation of 'minting order', 'withdrawal order' or even the order a Value has? All of those things are inherently ordered in the serialization and it's not unthinkable that one would want to preserve them for similar reasons. Maybe the argument 'it's much more useful for inputs' is already sufficient, but I'd like to have this addressed a bit more.

[WhatisRT]

I have another concern about this, which is interoperability. If this CIP would be implemented, I'd expect that most scripts (at some point in the future) would require the inputs that script cares about to be at the beginning of the list of inputs, simply because of convenience. This means that people would only be able to execute multiple scripts in the same transaction if those scripts were explicitly designed with that in mind. This is a significant change from how things work today: scripts that don't know and care about each other can be run in the same transaction, even if they were never intended to be executed simultaneously with other scripts.

[michaelpj]

That same issue is present today with, say, outputs. If scripts A and B both care about outputs O_A and O_B, it's easy for A to be lazily written to require O_A to be the first output, and likewise for B. So I'm not sure this makes things worse.

[KtorZ]

Suggested change

	This CIP proposes a way to improve efficiency of handling transaction inputs in Plutus scripts by changing the way they are representend in the ledger.
	This CIP proposes a way to improve efficiency of handling transaction inputs in Plutus scripts by changing the way they are representend in the ledger.represented

[KtorZ]

I am not sure to understand the point being made here and in particular, why transaction integrity is something that helps mitigating an adversary changing the order of inputs?

[KtorZ]

which could defeat the purpose of this optimization with it's own fees.

I am not sure that framing this as an optimization to save on fees is the best justification. Surely, it improves the developer experience and makes easier to work with transaction inputs. I think one of the most common problem with that is calculation of redeemer pointers and how it conflicts with transaction balancing (adding new inputs to cover for fee may invalid all existing pointers on inputs). If the argument is made about saving on fees, then it should come backed with numbers showing how much to help weigh whether the change is worth it or not.

[WhatisRT]

I happened to be thinking about this again, and I think what @michaelpj's comment suggests is that the order of the outputs should not have been preserved either. We could have just sorted them as well, which would have gives us better interoperability defaults. We can't change that anymore since people might rely on it somewhere, but we could strongly discourage using that information in our APIs. So while I'm not sure this CIP would make things worse, I'm also not sure that it doesn't make it worse.

[michaelpj]

I happened to be thinking about this again, and I think what @michaelpj's comment suggests is that the order of the outputs should not have been preserved either.

I think this is true if what we primarily cared about was encouraging scripts not to make bad assumptions. But in fact the aim was mostly just to accurately represent the structure of the transaction, and I still think that's a good default.

[WhatisRT]

I might have said what I meant in a way that's a bit confusing: I wouldn't want to throw away ledger information, I think that the ledger should reorder them and that should be accurately represented. In an alternate universe we could've had a version of Shelley that does sort outputs, and in hindsight I'd have preferred that.

[rphair]

This is up for Last Check at the CIP meeting today: although without response from the author about some issues posted above, I believe this has been vetted by IOG according to this announcement: input-output-hk/Developer-Experience-working-group#20

So (I hope @KtorZ will correct me if I'm wrong in this thinking) this is fait accompli because it's already being implemented... and therefore should be merged unless the explanations are misleading or not understandable, or the document is improperly formatted somehow (it looks OK to me).

[rphair]

(last label application was my operator error)

[michaelpj]

So this is fait accompli because it's already being implemented

This is not being implemented AFAIK.

[WhatisRT]

Yes, no plans on the ledger side. I also still believe that this is a step in the wrong direction, since it doesn't actually simplify much as scripts still have to check that the inputs are what they expect, see above. Instead it adds another pitfall (forgetting to properly check that the order is correct) and it may worsen interoperability.

[rphair]

@gege251 closing because of lack of support from Ledger & Plutus team(s) indicated above.

[colll78]

Please reopen this. If you guys saw the codebases of any of the DEXs you would understand why this is such a big deal. The crazy non-sense they all do (that introduces a ton of complexity and room for vulnerabilities) to reconstruct the necessary of inputs within the transaction is actually revolting, and the DEXs that don't do this ugly reconstruction can only fit roughly half as many requests in a batch. In addition to the absolutely massive efficiency benefits, it actually solves the double satisfaction issue, because you can match the corresponding outputs based on their indices (ie input at index 1 must correspond to output at index 1).

[rphair]

@colll78 (@euonymos) I hear your statement (and support) but this is not an issue for implementation: it's a technical document which particularly the Plutus team decided not to support. It would make no difference if we reopened or even merged this pull request as a CIP if it's not something they're going to do.

@michaelpj @WhatisRT on the off-chance that the last 7 months have led to the IOG Plutus or Ledger team implementing such a thing, or anything else that would mitigate the problem reported especially by DEXes, then can you post here? If so we can add it to the CIP meeting agenda in 3 days' time when Plutus reps are expected to be present, and maybe reopen this.

@colll78 the general problem of DEXes sounds like a good subject for a CPS (Cardano Problem Statement) if it remains unaddressed by any other CIP or development plan. Can you check first to see if any of these are applicable? https://github.com/cardano-foundation/CIPs#cardano-problem-statements-cps-1

... especially this one, which I think covered the "double satisfaction" problem? https://github.com/cardano-foundation/CIPs/tree/master/CPS-0005

[WhatisRT]

On the ledger side there are no plans to support this (or anything like it), and even if we wanted to support it there's no way this would happen any time soon. From what I hear, the Plutus team is also incredibly busy with things that have higher priority.

I've pointed out how to emulate this feature using redeemers a bunch of times, and it's not all that big of a deal to implement (something like 5 lines of Haskell code, maybe more when you need something extremely optimized). I'm open to discussing this issue further, but it should be motivated by good real-world examples (note that the example given in this CIP would be a horrible idea in the real world).

[rphair]

Historical note: this effort is continuing through candidate CIP-0128: #758