Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[suggestion] Sort query results by metadata key #2553

Closed
appetrosyan opened this issue Jul 28, 2022 · 4 comments
Closed

[suggestion] Sort query results by metadata key #2553

appetrosyan opened this issue Jul 28, 2022 · 4 comments
Assignees
Labels
Enhancement New feature or request iroha2-dev The re-implementation of a BFT hyperledger in RUST

Comments

@appetrosyan
Copy link
Contributor

appetrosyan commented Jul 28, 2022

Feature request

Solving #2501 via #2515 rerquires us to store a timestamp for all assets. This doubles the memory overhead and increases the time complexity of all queries that can return multiple Assets or AssetIds by $O ( \log n)$, where $n$ is the (usually large) number of assets over which we're paginating.

Motivation

Instead, a better solution would be to allow sorting of arbitrary query results based off of the metadata in each element. We still get the near-doubling of RAM usage, and time complexity for the queries which request sorting is also affected, however:

  1. Providing the timestamp is optional. If you don't need it in an asset, you don't have it. RAM impact is minimal.
  2. Sorting is optional. Most queries with pagination won't need it.
  3. Sorting is more flexible: you can sort by timestamp or by other metadata keys.
  4. The instability described in [FEATURE] Pagination in the order of insertion #2501 can be resolved separately without introducing a timestamp, instead by making pagination a resumable operation, that takes a snapshot of the data in the blockchain. Currently all results that aren't part of the pagination are re-computed during each query, and data is often re-created.
  5. [fix] #2501: Fixes the order of assets in queries with pagination #2515 requires minimal modifications to work.

Who can help?

@pesterev @Anrdey

@appetrosyan appetrosyan added Enhancement New feature or request iroha2-dev The re-implementation of a BFT hyperledger in RUST labels Jul 28, 2022
@mversic
Copy link
Contributor

mversic commented Jul 28, 2022

Instead, a better solution would be to allow sorting of arbitrary query results based off of the metadata in each element

yes, that's better IMO

  1. instead by making pagination a resumable operation, that takes a snapshot of the data in the blockchain

de facto providing the client with a server side forward cursor that points somewhere within the snapshot? Any modifications made post query, but during the lifetime of the cursor won't be observed. I'm not sure this resolves the issue described

@mversic
Copy link
Contributor

mversic commented Jul 28, 2022

I'm not sure this resolves the issue described

actually, I've looked it up again. That should do it, it's a problem on our side that pagination isn't resumable. It should be resumable

@pesterev
Copy link
Contributor

pesterev commented Jul 29, 2022

Overall looks good, but:

One client can affect the sorting order of another client if we let clients generate a timestamp. My suggestion is to allow the client to only set a flag in the metadata that the new asset should contain a timestamp, but the timestamp will be generated on the server and inserted into the metadata.

@appetrosyan
Copy link
Contributor Author

We should let clients define custom timestamps. This should be handled by the smartcontract infra. Letting people define different timestamps is a permission issue, not a problem of our algorithm. It is possible (likely) that in real cases being able to sub-sort by different parameters (other than timestamp) is desirable behaviour!

pesterev added a commit to pesterev/iroha that referenced this issue Aug 1, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 1, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 1, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 1, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 2, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 3, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 3, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 5, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 8, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 8, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 9, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 9, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 9, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 10, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 10, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 11, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 11, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 11, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 11, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
pesterev added a commit to pesterev/iroha that referenced this issue Aug 12, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
appetrosyan pushed a commit to pesterev/iroha that referenced this issue Aug 14, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
appetrosyan pushed a commit that referenced this issue Aug 14, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
s8sato pushed a commit to s8sato/iroha that referenced this issue Aug 15, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
mversic pushed a commit to mversic/iroha that referenced this issue Sep 6, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
mversic pushed a commit to mversic/iroha that referenced this issue Sep 6, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
mversic pushed a commit to mversic/iroha that referenced this issue Sep 6, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
mversic pushed a commit to mversic/iroha that referenced this issue Sep 7, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
BAStos525 pushed a commit to mversic/iroha that referenced this issue Sep 7, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
Signed-off-by: BAStos525 <jungle.vas@yandex.ru>
mversic pushed a commit to mversic/iroha that referenced this issue Sep 8, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
mversic pushed a commit to mversic/iroha that referenced this issue Sep 9, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
mversic pushed a commit to mversic/iroha that referenced this issue Sep 9, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
BAStos525 pushed a commit to mversic/iroha that referenced this issue Sep 9, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
mversic pushed a commit to mversic/iroha that referenced this issue Sep 9, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
BAStos525 pushed a commit to mversic/iroha that referenced this issue Sep 9, 2022
Signed-off-by: Vladimir Pesterev <pesterev@pm.me>
@Erigara Erigara mentioned this issue Jul 15, 2024
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement New feature or request iroha2-dev The re-implementation of a BFT hyperledger in RUST
Projects
None yet
Development

No branches or pull requests

4 participants