-
Notifications
You must be signed in to change notification settings - Fork 591
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
core: created streaming methods #1665
core: created streaming methods #1665
Conversation
I think we should go with |
* this.end(); | ||
* }); | ||
*/ | ||
Dataset.prototype.getTableStream = common.paginator.streamify('getTables'); |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
* @param {function} originalMethod - The original method. | ||
* @return {function} wrapped - The wrapped method. | ||
*/ | ||
paginator.wrap_ = function(originalMethod) { |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
@@ -34,48 +34,70 @@ var util = require('./util.js'); | |||
|
|||
/*! Developer Documentation | |||
* | |||
* streamRouter is used to extend `nextQuery`+callback methods with stream | |||
* functionality. | |||
* paginator is used to auto paginated `nextQuery` methods as well as |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
* @param {function} originalMethod - The original method. | ||
* @return {function} wrapped - The wrapped method. | ||
*/ | ||
paginator.wrap_ = function(originalMethod) { |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
We can definitely do this, but it makes me a little nervous that it'll either require additional method renaming since the private version could exist or in the future it could lead to some confusion if we're overwriting private methods without realizing it. |
I'm not too concerned about that, but you can always name it something more unique. |
@callmehiphop @jgeewax wdyt re: |
IIRC we had settled on smushing the s's. I closest thing to written confirmation I could find was in #1142 (comment). Personally I like the smushed naming better, and I don't think that is something that will confuse the user. |
I meant "smushing" as in pushing them both together, not collapsing / some other word that's not clear. Basically, I prefer |
@stephenplusplus I made a bunch of updates - I think this is in a pretty good place. PTAL! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work, sir! I have a general comment that I didn't want to drop throughout, but I think in some places, this deviates from the alpha-sort pattern of methods. Please double check and try to keep it sorted in the source and tests as well, as it makes jumping into a bug fix / test cycle much easier. If the whole file was out of sort already, don't worry about taking on any extra work to clean it up, but I think most were sorted.
Other than that, just a few notes on naming / docs.
* See {module:bigquery#createQueryStream} for full documentation of this | ||
* method. | ||
*/ | ||
Dataset.prototype.createQueryStream = function(options) { |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
* this.end(); | ||
* }); | ||
*/ | ||
Dataset.prototype.getTablesStream = common.paginator.streamify('getTables'); |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
* | ||
* @example | ||
* table.getRowsStream() |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
@@ -425,7 +425,8 @@ Table.prototype.getMetadata = function(options, callback) { | |||
}; | |||
|
|||
/** | |||
* Get Row objects for the rows currently in your table. | |||
* Get {module:bigtable/row} objects for the rows currently in your table as a | |||
* readable object stream. |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
@@ -34,48 +34,70 @@ var util = require('./util.js'); | |||
|
|||
/*! Developer Documentation | |||
* | |||
* streamRouter is used to extend `nextQuery`+callback methods with stream | |||
* functionality. | |||
* paginator is used to auto paginated `nextQuery` methods as well as |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
* | ||
* Before: | ||
* | ||
* search.query('done=true', function(err, results, nextQuery) {}); | ||
* search.query('done=true', function(err, results, nextQuery) { | ||
* search.query(nextQuery, function(err, results, nextQuery) { ... }); |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
*/ | ||
Query.prototype.run = function() { | ||
Query.prototype.stream = function() { |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
* query. You can pass that object back to this method to see if more results | ||
* exist. | ||
* | ||
* @param {module:datastore/query} q - Query object. |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
|
||
'use strict'; | ||
|
||
var Module = require('./helpers').Module; |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
@stephenplusplus I made some updates based on your feedback, PTAL! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
assert.strictEqual(bq.getDatasetsStream, 'getDatasets'); | ||
assert.strictEqual(bq.getJobsStream, 'getJobs'); | ||
assert.strictEqual(bq.createQueryStream, 'query'); | ||
assert.strictEqual(bq.getDatasetsStream, 'getDatasets'); |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
@stephenplusplus not trying to be annoying :) - but where do we stand on this? |
Just a few comments we didn't discuss: #1665 (comment) Mobile now, but there may have been more PTAL |
Actually I resolved all of those, not sure why the comments are still in the feed.. maybe a quirk of starting a review? |
Oh, that's frustrating. Well, while I dig back into the changes, it looks like there's a build bug: https://ci.appveyor.com/project/GoogleCloudPlatform/google-cloud-node/build/1.0.298/job/y5hsy2w05p6iqg6c |
related: #1512 I think we should roll out a deprecation plan here:
We will git rid of the warnings in 1.0 stable, but for the beta, I think it's fair to print to the console. Can you also make a map of old -> new methods, so we can include them all at once in the release notes? // cc @richardkazuomiller |
Not a bug per se, but we don't link common on CI, so the
Not sure how we are supposed to do that, the whole intention behind this is to add support for Promises, meaning when a callback is omitted to one of these original methods, we instead return a Promise.
How do you want me to squeeze that in? Technically all the methods will still be usable with callbacks and promises.
Sure can. |
Okay, what can we do to make the CI pass? Do we need to cut a release of changes in common?
I'm thinking of methods such as
See above; these docs would be for the methods that were removed, that I think we should add back. |
Well yes, but this branch was being merged into the Promise support branch.. not master. I figured we'd get that branch working fine and then figure out a roll out plan later (break up commits, etc.)
I'll double check, but I'm pretty confident I didn't actually remove any methods, all the streaming APIs that I refactored had a non-streaming version that should still be intact. |
Ah right, not a big deal then. |
I double checked and I can't find any methods that were outright deleted. |
Okidoke, sorry to derail. No methods were removed, this only adds new methods with Can you list methods where we did that ^ and put them in the OP of this PR? I believe it may only be the example above and something from BigQuery? |
Since we needed a map of what functionality went where, I went ahead and just updated the overview with every method that has changed. |
A beautiful sight. Thanks! imamerge. |
This PR moves all streaming APIs to their own methods.
e.g.
bigquery.getJobs()
->bigquery.getJobsStream()
TODOS
Changes
All callback style methods are left intact, however as mentioned earlier, all streaming methods have been moved to their own methods. Here is a map of where streaming functionality has moved to.
BigQuery
BigQuery#query
->BigQuery#createQueryStream
BigQuery#getDatasets
->BigQuery#getDatasetsStream
BigQuery#getJobs
->BigQuery#getJobsStream
Dataset#query
->Dataset#createQueryStream
Dataset#getTables
->Dataset#getTablesStream
Job#getQueryResults
->Job#getQueryResultsStream
Table#query
->Table#createQueryStream
Table#getRows
->Table#createReadStream
Bigtable
Bigtable#getInstances
->Bigtable#getInstancesStream
Instance#getClusters
->Instance#getClustersStream
Instance#getTables
->Instance#getTablesStream
Table#getRows
->Table#createReadStream
Table#sampleRowKeys
->Table#sampleRowKeysStream
Compute
Compute#getAddresses
->Compute#getAddressesStream
Compute#getAutoscalers
->Compute#getAutoscalersStream
Compute#getDisks
->Compute#getDisksStream
Compute#getInstanceGroups
->Compute#getInstanceGroupsStream
Compute#getFirewalls
->Compute#getFirewallsStream
Compute#getHealthChecks
->Compute#getHealthChecksStream
Compute#getMachineTypes
->Compute#getMachineTypesStream
Compute#getNetworks
->Compute#getNetworksStream
Compute#getOperations
->Compute#getOperationsStream
Compute#getRegions
->Compute#getRegionsStream
Compute#getRules
->Compute#getRulesStream
Compute#getServices
->Compute#getServicesStream
Compute#getSnapshots
->Compute#getSnapshotsStream
Compute#getSubnetworks
->Compute#getSubnetworksStream
Compute#getVMs
->Compute#getVMsStream
Compute#getZones
->Compute#getZonesStream
InstanceGroup#getVMs
->InstanceGroup#getVMsStream
Network#getSubnetworks
->Network#getSubnetworksStream
Network#getFirewalls
->Network#getFirewallsStream
Region#getAddresses
->Region#getAddressesStream
Region#getOperations
->Region#getOperationsStream
Region#getRules
->Region#getRulesStream
Region#getSubnetworks
->Region#getSubnetworksStream
Zone#getAutoscalers
->Zone#getAutoscalersStream
Zone#getDisks
->Zone#getDisksStream
Zone#getInstanceGroups
->Zone#getInstanceGroupsStream
Zone#getMachineTypes
->Zone#getMachineTypesStream
Zone#getOperations
->Zone#getOperationsStream
Zone#getVMs
->Zone#getVMsStream
Datastore
Query#run
->Query#runStream
DatastoreRequest#get
->DatastoreRequest#createReadStream
DatastoreRequest#runQuery
->DatastoreRequest#runQueryStream
DNS
DNS#getZones
->DNS#getZonesStream
Zone#getChanges
->Zone#getChangesStream
Zone#getRecords
->Zone#getRecordsStream
Logging
Logging#getEntries
->Logging#getEntriesStream
Logging#getSinks
->Logging#getSinksStream
Log#getEntries
->Log#getEntriesStream
Prediction
Prediction#getModels
->Prediction#getModelsStream
PubSub
PubSub#getSubscriptions
->PubSub#getSubscriptionsStream
PubSub#getTopics
->PubSub#getTopicsStream
Topic#getSubscriptions
->Topic#getSubscriptionsStream
Resource
Resource#getProjects
->Resource#getProjectsStream
Storage
Storage#getBuckets
->Storage#getBucketsStream
Bucket#getFiles
->Bucket#getFilesStream