datastore: update to v1beta3

[stephenplusplus]

Fixes #872
Fixes #1049
Fixes #1181

Docs

http://stephenplusplus.github.io/gcloud-node/#/docs/master/datastore

Blockers

• datastore: properly parse HTTP responses from the API #913 -- need it for testing test/datastore/request.js
• New release of gRPC
• Datastore v1beta3 release

To Dos

• Update resource links (blocked by v1beta3's release (public docs required to get URLs))
  • Resolution: Removed resource links
• Remove entry from Troubleshooting (docs: explain how Datastore requires userinfo scope #1172)
• Convert to gRPC
• Add support for env vars from cli (Support Datastore default dataset env var #1040):
  • DATASTORE_PROJECT_ID
  • DATASTORE_EMULATOR_HOST (protocol not included)
• Use Service & ServiceObject
• Tests
  • Unit
    • Entity
    • Index (Datastore, formerly Dataset)
    • Query
    • Request
    • Transaction
  • System
• Readme snippet update
• Depend on a new release of protobuf.js (tracking)

⚠️ Big Breaking Changes

Dataset isn't a thing anymore:

var gcloud = require('gcloud')({/*...*/});

Before

var datastore = gcloud.datastore;
var dataset = datastore.dataset();

dataset.get(dataset.key(['Kind', 1]), function(err, results) {});

After

var datastore = gcloud.datastore();
datastore.get(datastore.key(['Kind', 1]), function(err, results) {});

// Use a different project:
var datastore = gcloud.datastore({ projectId: 'grape-spaceship-123' });

[stephenplusplus]

Thanks for the review @pcostell! I'll be making some adjustments throughout the week and will ping if I have any questions.

[pcostell]

No worries -- one more note, the url has changed. So for example for runQuery it is now:

datastore.googleapis.com/v1beta3/projects/<project-id>:runQuery
Instead of www.googleapis.com/datastore/v1beta2/datasets/<project-id>/RunQuery

[stephenplusplus]

Is v1beta3 generally available? I've been using the response code of https://cloud.google.com/datastore/docs/apis/v1beta3 to tell me if it's live or not, and it's still a 404.

[pcostell]

The docs and blog post won't drop until Monday, however the API can be enabled: https://console.cloud.google.com/apis/api/datastore.googleapis.com/overview

[stephenplusplus]

For users that already have Datastore enabled and have been running v1beta2, they won't have to re-enable the API, right?

[pcostell]

Any projects active in the last 7 days will not need to re-enable.

[stephenplusplus]

Cool, thanks. And more generally, thanks for all of the help along the way! 👍

[pcostell]

Thank you for helping us get this out the door!

As a heads up, the cloud SDK will not have a grpc emulator at launch time. So users that want to use gcloud-node will have to use the gcd.sh tool directly, which they can get from here.

[pcostell]

Release notes for Datastore available if you need to link to them: https://cloud.google.com/datastore/release-notes

[stephenplusplus]

@callmehiphop - all squashed.

[callmehiphop]

🎉

[stephenplusplus]

@pcostell any API flakiness going on? We are having inconsistent results querying for data:

Commit:

{
  "mutations": [
    {
      "upsert": {
        "key": {
          "path": [
            {
              "kind": "Person",
              "name": "name"
            }
          ]
        },
        "properties": {
          "f": {
            "stringValue": "Full name"
          }
        }
      }
    }
  ],
  "projectId": "nth-circlet-705",
  "mode": "NON_TRANSACTIONAL"
}

Query:

{
  "readOptions": {},
  "query": {
    "distinctOn": [],
    "kind": [
      {
        "name": "Person"
      }
    ],
    "order": [],
    "projection": [],
    "filter": {
      "compositeFilter": {
        "filters": [
          {
            "propertyFilter": {
              "property": {
                "name": "f"
              },
              "op": "EQUAL",
              "value": {
                "stringValue": "Full name"
              }
            }
          }
        ],
        "op": "AND"
      }
    }
  },
  "projectId": "nth-circlet-705"
}

Sometimes the query returns the entity, other times it doesn't. Is there anything we're doing wrong?

[pcostell]

@stephenplusplus That looks like standard Eventual Consistency. For any e2e tests that require a get and a query you should be using ancestors.

[stephenplusplus]

I'm getting it with that as well:

Commit:

{
  "mutations": [
    {
      "upsert": {
        "key": {
          "path": [
            {
              "kind": "People",
              "name": "US"
            },
            {
              "kind": "Person",
              "name": "name"
            }
          ]
        },
        "properties": {
          "f": {
            "stringValue": "Full name"
          }
        }
      }
    }
  ],
  "projectId": "nth-circlet-705",
  "mode": "NON_TRANSACTIONAL"
}

Query

{
  "readOptions": {},
  "query": {
    "distinctOn": [],
    "kind": [
      {
        "name": "Person"
      }
    ],
    "order": [],
    "projection": [],
    "filter": {
      "compositeFilter": {
        "filters": [
          {
            "propertyFilter": {
              "property": {
                "name": "f"
              },
              "op": "EQUAL",
              "value": {
                "stringValue": "Full name"
              }
            }
          }
        ],
        "op": "AND"
      }
    }
  },
  "projectId": "nth-circlet-705"
}

Maybe I'm not understanding how to save and/or query using ancestors?

[stephenplusplus]

Sorry, I got it using hasAncestor. I think I'm good.

[pcostell]

Cool, let me know if there are any other issues