-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Maintain sorted operation indexes #438
Conversation
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## development #438 +/- ##
===============================================
+ Coverage 90.05% 90.19% +0.13%
===============================================
Files 87 87
Lines 8439 8514 +75
===============================================
+ Hits 7600 7679 +79
+ Misses 839 835 -4
☔ View full report in Codecov by Sentry. |
Great!!
That's half working actually, but with this PR we have everything in place I think to help us with it: #441 |
* development: (23 commits) Implement `dialer` behaviour (#444) Sort expected results in strategy tests Update CHANGELOG Replicate operations in topo order (#442) Maintain sorted operation indexes (#438) Use fork of `asynchronous-codec` (#440) Ingest check for duplicate entries (#439) Reverse lookup for pinned relations in dependency task (#434) Remove unnecessary exact version pinning in Cargo.toml Make `TaskInput` an enum and other minor clean ups in materialiser (#429) Use `libp2p` `v0.52.0` (#425) Fix race condition when check for existing view ids was too early (#420) Reduce logging verbosity CI: Temporary workaround for Rust compiler bug (#417) Fix early document view insertion (#413) Handle duplicate document view insertions (#410) Decouple p2panda's authentication data types from libp2p's (#408) Remove dead_code attribute in lib Integrate replication manager with networking stack (#387) Implement naive replication protocol (#380) ...
We want to persist and maintain sorted position indexes for all operations which have been materialized into a document. This index is the operations position once topological sorting has occurred and can be understood as the
sorted_index
of a particular operation, or also the documents "depth" at that point in it's history.This is done by introducing the following:
sorted_index
column to theoperations_v1
tableupdate_operation_index
method to theOperationStore
reduce
task update thesorted_index
of all operations in a documentNote, there is a period of time when an operation has been inserted into the store, but it's
sorted_index
isNone
because materialization has not been performed yet. As we have a "retry" system already in-place to account for a node shutting down unexpectedly before tasks could be completed, then we can be confident all operations will be processed and have theirsorted_index
set eventually.📋 Checklist
CHANGELOG.md