Turn a paged http request into a stream of pages. Focuses on simplicity and modularity. Doesn't try to assume anything. You must implement a 'next' function that returns the next request based upon the previous one.
npm install paged-http-stream
Options:
method
: defaults to GET
uri
: the url to query. includes query string.
... and anything else that can be passed to a typical node.js http request (uses got)
You need to implement the next
function, which be able to interpret the data from the previous request, passed in as an argument, into a new request.
For example, figshare returns each of their search pages like this:
{
"items_found" : 91,
"page_nr": 1,
"items": []
}
So, I can write a function next
that takes the page_nr
and adds one, and then returns null
when the items list is empty:
function next (data) {
// data will be JSON parsed already.
if (data.error) throw new Error(data.error)
if (data.items && data.items.length === 0) return null // we are done here
var query = {
search_for: 'this is my query',
page: parseInt(data.page_nr + 1) || 1 // get the next page
}
return getOpts(query)
}
function getOpts (query) {
return {
method: 'GET',
uri: 'http://api.figshare.com/v1/articles/search?' + qs.stringify(query)
}
}
Then, pass to the pager:
var pageStream = pager(startingOpts, next)
You can then get access to each page like this (as JSON):
pageStream.on('data', funciton (page) {
console.log(page.items)
})
- timeout, or wait time between requests