API requests hang indefinitely

[goshoo]

Hi,

I'm using google datastore API and if I run my application from my local computer then the application connects to the datastore, but if I deploy the application it cannot connect to the datastore.

Environment details

• OS: Debian GNU/Linux 8
• Node.js version: 6.9.4
• npm version: shipped with Node.js 6.9.4
• @google-cloud/datastore: 0.6.1

Steps to reproduce

I ran the following example on AppEngine instance:

const Datastore = require('@google-cloud/datastore');

const datastore = Datastore({
      projectId: '<$projectId>',
      keyFilename: './gcloud-keyfile.json'
});

const query = datastore.createQuery('UserModel')
  .filter('user_name', '=', 'gosho');

datastore.runQuery(query)
  .then((results) => {
    const users = results[0];
    users.forEach((user) => console.log(user));
  });

My question is, why cannot my deployed application connect to the datastore?

Thanks!

[just1689]

+1

[stephenplusplus]

Please try upgrading to the latest version of google-cloud or @google-cloud/datastore.

[goshoo]

Hi stephenplusplus,

I use the latest version of @google-cloud/datastore - 0.6.1

[just1689]

+1 again

[stephenplusplus]

Sorry, I forgot to publish to npm. Please try @google-cloud/datastore@0.7.0.

[Muesly]

I'm getting the same issue, and I've tried 0.7.0 too. Any idea how to resolve this, or the last working version?

[stephenplusplus]

To everybody: Do you get an error? How are you authenticating?

[Muesly]

Hi,

Hope this helps

var datastore = require('@google-cloud/datastore')({ projectId: <Project ID> });
var item = datastore.key(['Key', 'Value']);
datastore.get(item, function(err, result) 
{
...

It doesn't return with an err or result. I don't authenticate at any point. As this is server code I wasn't expecting to need to?

[Muesly]

I'm running this through App Engine. I tried to remove the projectId as advised if running inside Google Cloud. https://www.npmjs.com/package/@google-cloud/datastore

However, then it did call the callback with the error "Error: Sorry, we cannot connect to Google Cloud Services without a project ID. You may specify one with an environment variable named "GCLOUD_PROJECT". See https://googlecloudplatform.github.io/google-cloud-node/#/docs/guides/authentication for a detailed guide on creating an authenticated connectio..."

I then added the GCLOUD_PROJECT variable and I have the same problem as before.

[stephenplusplus]

@jmdobry I'm spinning my wheels a bit trying to get an app running on GAE-- are you able to try deploying a sample Datastore app to GAE to see if you can reproduce these errors?

Error 1: GCLOUD_PROJECT not being set by GAE
Error 2: Hanging after an API request as described above (a runQuery or a get)

[jmdobry]

To debug I need from y'all:

• Your app.yaml file
• Your output of running gcloud --version

Note when running on Google App Engine flexible environment you should instantiate the Cloud Datastore client library without passing in projectId or keyFilename (with the exception of some advanced use cases).

[jmdobry]

Also need to know if you have a yarn.lock file or not.

[Muesly]

Output of gcloud --version:

Google Cloud SDK 141.0.0
app-engine-python 1.9.49
app-engine-python-extras 1.9.49
beta 2016.01.12
bq 2.0.24
bq-nix 2.0.24
core 2017.01.20
core-nix 2016.11.07
gcloud
gsutil 4.22
gsutil-nix 4.18

[Muesly]

I can't see a yarn.lock in my project root, if that's where it should be.
I'll add the GCLOUD_PROJECT back into the app.yaml and remove projectId from the .js call
My app.yaml is:

[START app_yaml]

runtime: nodejs
vm: true
threadsafe: true

automatic_scaling:
min_num_instances: 1
max_num_instances: 5
cool_down_period_sec: 360 # up checks every 300 seconds
cpu_utilization:
target_utilization: 0.5

handlers:

• url: /.*
  script: IGNORED
  secure: always

skip_files:

• ^(node_modules)
• ^(logs)

env_variables:
GCLOUD_TRACE_DISABLE: true
GCLOUD_PROJECT: project-name

[END app_yaml]

[jmdobry]

Note that vm: true has been deprecated, and you should update to env: flex. threadsafe: true has no meaning for Node.js apps. The GCLOUD_PROJECT environment variable is set automatically for apps deployed with env: flex, so after updating to env: flex you can remove GCLOUD_PROJECT from your app.yaml file.

But none of that has much to do with the hanging Datastore API calls. I've been able to reproduce the problem using three different versions of @google-cloud/datastore and I've notified the App Engine team.

[czzarr]

I'm getting the same problem, can't connect. This is a pretty big issue, any fix planned? Fwiw 0.1.1 works...

[simis2626]

I'm getting the same errors (i.e. no errors and hung calls.)
app.yaml:
runtime: nodejs
env: flex

manual_scaling:
instances: 1

health_check:
enable_health_check: False
gcloud --version
Google Cloud SDK 140.0.0
beta 2016.01.12
bq 2.0.24
bq-win 2.0.24
bundled-python 2.7.10
core 2017.01.17
core-win 2016.11.07
gcloud
gsutil 4.22
gsutil-win 4.20
windows-ssh-tools 2016.05.13

[Binarytales]

I think I was seeing this issue (#1954). It seemed to resolve itself, or at least whatever was failing and causing an error (which never was surfaced ) resolved itself by updating to Node v6.9.5. I noticed that @goshoo said he was running on v6.9.4

[jmdobry]

Can anyone comment on whether this issue also appears on Google Compute Engine and/or Google Container Engine?

[LeoIsCoding]

My node server (that exhibits the same issue) is running in the Flexible Environment (GCE based).

[Muesly]

I've tried updating to Node 6.9.5 but it made no difference unfortunately

[one20]

I ran into this issue, was using app engine (flex+nodejs). Problem seems to be the google nodejs docker file, I created my own custom docker (https://nodejs.org/en/docs/guides/nodejs-docker-webapp/) and my app was able to connect to datastore from app engine.

EDIT:
I was running Node 7.4.0, google-cloud 0.46.1. Combo had the same symptoms as mentioned by others, until I decided to use a custom runtime.

[ofrobots]

We suspect that this is due to a recent change in grpc@1.1.1 not dealing properly with the lack of /etc/services. We are working on an updated version of the Node.js docker image with a work-around that should be available in a few hours.

Could folks try deploying again (in a few hours) with the latest runtime image to see if the problem gets resolved?

[just1689]

Would be happy to. I have a number of great test cases.

…

On Tue, 7 Feb 2017, 04:38 Ali Ijaz Sheikh, ***@***.***> wrote: We suspect that this is due to a recent change in ***@***.*** not dealing properly with the lack of /etc/services. We are working on an updated version of the Node.js docker image with a work-around <GoogleCloudPlatform/nodejs-docker#79> that should be available in a few hours. Could folks try deploying again (in a few hours) with the latest runtime image to see if the problem gets resolved? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1955 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADdMei00Do1RsRXqlikcpfprMQqpRiRIks5rZ9kHgaJpZM4L1L4X> .

[LeoIsCoding]

I added the "netbase" patch to my package.json as a workaround to regain access to datastore from express server. The workaround was suggested here (#1946)

[murgatroid99]

grpc@1.1.2 has been published, and it should not have this failure.

[stephenplusplus]

Just released google-cloud version 0.46.2 -- anyone willing to test, please update and let us know how it went.

[just1689]

It's working! It's alive!

Thanks everyone!

[Muesly]

I'd be happy to test, but I only import the datastore specifically so would need to wait until you released 0.7.1 of @google-cloud/datastore

[stephenplusplus]

The sub-modules will automatically include the fix if you un- and re-install them.

[Muesly]

Thanks - looks to work now :)

[stephenplusplus]

Great, thanks for the patience and the help, everyone! If anyone is still having issues after a hard upgrade, let us know.