[suggestion] Sort query results by metadata key

[appetrosyan]

Feature request

Solving #2501 via #2515 rerquires us to store a timestamp for all assets. This doubles the memory overhead and increases the time complexity of all queries that can return multiple Assets or AssetIds by $O ( \log n)$, where $n$ is the (usually large) number of assets over which we're paginating.

Motivation

Instead, a better solution would be to allow sorting of arbitrary query results based off of the metadata in each element. We still get the near-doubling of RAM usage, and time complexity for the queries which request sorting is also affected, however:

1. Providing the timestamp is optional. If you don't need it in an asset, you don't have it. RAM impact is minimal.
2. Sorting is optional. Most queries with pagination won't need it.
3. Sorting is more flexible: you can sort by timestamp or by other metadata keys.
4. The instability described in [FEATURE] Pagination in the order of insertion #2501 can be resolved separately without introducing a timestamp, instead by making pagination a resumable operation, that takes a snapshot of the data in the blockchain. Currently all results that aren't part of the pagination are re-computed during each query, and data is often re-created.
5. [fix] #2501: Fixes the order of assets in queries with pagination #2515 requires minimal modifications to work.

Who can help?

@pesterev @Anrdey

[mversic]

Instead, a better solution would be to allow sorting of arbitrary query results based off of the metadata in each element

yes, that's better IMO

4. instead by making pagination a resumable operation, that takes a snapshot of the data in the blockchain

de facto providing the client with a server side forward cursor that points somewhere within the snapshot? Any modifications made post query, but during the lifetime of the cursor won't be observed. I'm not sure this resolves the issue described

[mversic]

I'm not sure this resolves the issue described

actually, I've looked it up again. That should do it, it's a problem on our side that pagination isn't resumable. It should be resumable

[pesterev]

Overall looks good, but:

One client can affect the sorting order of another client if we let clients generate a timestamp. My suggestion is to allow the client to only set a flag in the metadata that the new asset should contain a timestamp, but the timestamp will be generated on the server and inserted into the metadata.

[appetrosyan]

We should let clients define custom timestamps. This should be handled by the smartcontract infra. Letting people define different timestamps is a permission issue, not a problem of our algorithm. It is possible (likely) that in real cases being able to sub-sort by different parameters (other than timestamp) is desirable behaviour!