-
Notifications
You must be signed in to change notification settings - Fork 151
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add /blocks
that enforces range
query param.
#954
Conversation
Some benchmarks for querying 1000 blocks at a time against a polkadot archive node:
|
…into tarik-blocks-range Merge master
/blocks
that enforces range
query param./blocks
that enforces range
query param.
const min = Number(splitRange[0]); | ||
const max = Number(splitRange[1]); | ||
|
||
if (!this.verifyInt(min) || !this.verifyInt(max)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I find this logic a little bit hard to understand.
As it will accept both 0-100
and 100-0
, I kind assumed that that "reverse range" should not work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i.e,
if !(this.verifyUInt(min) {
throw new BadRequest('Inputted `min` invalid UInt.');
)
if !(this.verifyNonZeroUInt(min) {
throw new BadRequest('Inputted `max` invalid NonZeroUInt.');
)
// not sure if range(1,1) should be valid seems pointless but could work :)
if (min > max ) {
throw new BadRequest('Invalid range: max must be bigger than min');
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea this is totally correct, will add that check.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would rather a have two different functions because the code becomes easier to understand as I suggested but your choice :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You mean with verifying UInt and a NonZeroUInt correct? If so, sounds totally reasonable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, or something even better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea i agree, it will allow for concise behavior between the min and max as well. Will put that in now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay yea i cleaned it up to handle Uint correctly and Non zero uints correctly. I think its way cleaner and concise about what the error messages will return.
// How many tasks are currently running concurrently? | ||
#runningTasks = 0; | ||
// The queued tasks waiting to run | ||
#tasks: LinkedList<() => void>; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm, a double ended queue
should be faster than a LinkedList
here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that's true, the linkedlist was just easier to implement I guess, and with a double ended queue you probably want to make sure it shrinks down again when items are removed, which the linked list thing just handles :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given this notion, do we want to implement a double ended queue, or is a linked list sufficient?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
leave it out this PR, I think the networking is the bottleneck anyway.
but maybe worth investigating :)
private verifyInt(num: Number): boolean { | ||
return Number.isInteger(num) && num > 0; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a random thought; I wonder how many of these sorts of functions could just be standalone utility functions and not be associated with a class in some way (once something is in a class, it has access to everything under this
and so in my mind can be a little harder to reason about what they might do, whereas with standlone functions in a utility file you know exactly all of the data they will have access to).
(not suggesting anything should be done here)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea I am open to pushing this in our util folder. It's newly created with the PromiseQueue
being added to sidecar so it would be nice to have these simpler verification helper functions in there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a start to definitely clean things up.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay cool, so the verifyInt has all been removed and added to its own integers util folder.
// We set a max range to 500 blocks. | ||
const rangeOfNums = this.parseRangeOfNumbersOrThrow(range, 500); | ||
const rangeOfNumsToHash = await Promise.all( | ||
rangeOfNums.map(async (n) => await this.getHashForBlock(n.toString())) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rangeOfNums.map(async (n) => await this.getHashForBlock(n.toString())) | |
rangeOfNums.map((n) => this.getHashForBlock(n.toString())) |
I guess this is identical anyway?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yea this is correct we dont need the async await
Looks great! |
Feature
Endpoint:
/blocks
Query params:
range
(required) ex./blocks?range=0-999
0-999
, or for something more dynamic such as1,2,3,10-100,123,1234
Response:
co-authored by @jsdw (Implemented the PromiseQueue)
closes: #460