Fix algorithmic complexity of on-demand scheduler with regards to number of cores. #3190
Conversation
Does not yet typecheck.
|
New profile looks good 🚀 |
Reintroduces on_initialize hook to update the spot traffic even when there are no on demand orders being placed, allowing for a price decrease. Add Ord tests for QueueIndex and Reverse
polkadot/runtime/parachains/src/assigner_on_demand/migration.rs
Outdated
Show resolved
Hide resolved
polkadot/runtime/parachains/src/assigner_on_demand/migration.rs
Outdated
Show resolved
Hide resolved
|
@antonva Command |
|
bot bench polkadot-pallet --runtime=westend --pallet=runtime_parachains::assigner_on_demand |
|
@antonva https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5561742 was started for your command Comment |
Co-authored-by: ordian <write@reusable.software>
…=westend --target_dir=polkadot --pallet=runtime_parachains::assigner_on_demand
|
@antonva Command |
polkadot/runtime/parachains/src/assigner_on_demand/migration.rs
Outdated
Show resolved
Hide resolved
polkadot/runtime/parachains/src/assigner_on_demand/migration.rs
Outdated
Show resolved
Hide resolved
| /// The actual queue is implemented via multiple priority queues. One for each core, for entries | ||
| /// which currently have a core affinity and one free queue, with entries without any affinity yet. | ||
| /// | ||
| /// The design aims to have most queue accessess be O(1) or O(log(N)). Absolute worst case is O(N). |
There was a problem hiding this comment.
a note on complexity: when you have a map like AffinityEntries from core index to BinaryHeap, the read and write to it would still be O(len(BinaryHeap)). Having BinaryHeap instead of a Vec only saves on in-memory operations, but I assume these will be dominated by the actual db writes/reads + allocation + encoding/decoding + hashing unless you have a lot of in-memory ops for one read/write
There was a problem hiding this comment.
You mean, because the data still needs to be fetched? Yes indeed, I considered this, but decided to not make it part of the complexity analysis, because the fetching of data is unavoidable and not really part of the algorithm. I was also hoping that the batch data fetching is fast. In any case, we are also usually fetching less data than before. I was tempted to optimize not accessing the free list, in those cases where we don't need data from there, by keeping track of its status, but decided the complexity is not worth it.
| #[cfg(not(test))] | ||
| debug_assert_ne!( | ||
| affinity, None, | ||
| "Decreased affinity for a para that has not been served on a core?" | ||
| ); |
There was a problem hiding this comment.
first time seeing debug assert for non test environment. is the intention to run this assert on testnets and benchmarks only?
There was a problem hiding this comment.
I guess it breaks some edge case tests?
|
The CI pipeline was cancelled due to failure one of the required jobs. |
polkadot/runtime/westend/src/weights/runtime_parachains_assigner_on_demand.rs
Show resolved
Hide resolved
antonva
added
the
T8-polkadot
…ber of cores. (paritytech#3190) We witnessed really poor performance on Rococo, where we ended up with 50 on-demand cores. This was due to the fact that for each core the full queue was processed. With this change full queue processing will happen way less often (most of the time complexity is O(1) or O(log(n))) and if it happens then only for one core (in expectation). Also spot price is now updated before each order to ensure economic back pressure. TODO: - [x] Implement - [x] Basic tests - [x] Add more tests (see todos) - [x] Run benchmark to confirm better performance, first results suggest > 100x faster. - [x] Write migrations - [x] Bump scale-info version and remove patch in Cargo.toml - [x] Write PR docs: on-demand performance improved, more on-demand cores are now non problematic anymore. If need by also the max queue size can be increased again. (Maybe not to 10k) Optional: Performance can be improved even more, if we called `pop_assignment_for_core()`, before calling `report_processed` (Avoid needless affinity drops). The effect gets smaller the larger the claim queue and I would only go for it, if it does not add complexity to the scheduler. --------- Co-authored-by: eskimor <eskimor@no-such-url.com> Co-authored-by: antonva <anton.asgeirsson@parity.io> Co-authored-by: command-bot <> Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com> Co-authored-by: ordian <write@reusable.software>
We witnessed really poor performance on Rococo, where we ended up with 50 on-demand cores. This was due to the fact that for each core the full queue was processed. With this change full queue processing will happen way less often (most of the time complexity is O(1) or O(log(n))) and if it happens then only for one core (in expectation).
Also spot price is now updated before each order to ensure economic back pressure.
TODO:
Optional: Performance can be improved even more, if we called
pop_assignment_for_core(), before callingreport_processed(Avoid needless affinity drops). The effect gets smaller the larger the claim queue and I would only go for it, if it does not add complexity to the scheduler.