Intermittent Cloud Datastore Error: Unauthorized: 401 ... Unable to find the server at accounts.google.com

[speedplane]

I am getting regular, but intermittent errors when using GCL to read from the datastore. The stacktrace is below. I am running Python 2.7 on Ubuntu.

I get this error only occasionally when I call the get_multi function. It is intermittent and looks like it could be a network error based on the very last line (Unable to find the server at accounts.google.com.). I am making thousands of calls to the datastore, and while this occurs only a small percent of the time, it occurs with pretty heavy regularity (about once a minute).

 File "/home/speedplane/DA/djangoappengine/db_gcl/compiler.py", line 349, in get_matching_pk
    self.client.get_multi(self.included_pks, **config)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/client.py", line 307, in get_multi
    transaction_id=transaction and transaction.id,
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/client.py", line 124, in _extended_lookup
    transaction_id=transaction_id,
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/_http.py", line 519, in lookup
    lookup_response = self._datastore_api.lookup(project, lookup_request)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/_http.py", line 304, in lookup
    return self._stub.Lookup(request_pb)
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/_http.py", line 260, in _grpc_catch_rendezvous
    raise error_class(exc.details())
Unauthorized: 401 Traceback (most recent call last):.  File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 154, in grpc._cython.cygrpc.plugin_get_metadata (src/python/grpcio/grpc/_cython/cygrpc.c:6918).  File "/usr/local/lib/python2.7/dist-packages/grpc/_plugin_wrapping.py", line 107, in __call__.    AuthMetadataPluginCallback(wrapped_cygrpc_callback)).  File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/grpc.py", line 65, in __call__.    callback(self._get_authorization_headers(), None).  File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/grpc.py", line 51, in _get_authorization_headers.    self._credentials.refresh(self._request).  File "/usr/local/lib/python2.7/dist-packages/google/oauth2/service_account.py", line 313, in refresh.    request, self._token_uri, assertion).  File "/usr/local/lib/python2.7/dist-packages/google/oauth2/_client.py", line 143, in jwt_grant.    response_data = _token_endpoint_request(request, token_uri, body).  File "/usr/local/lib/python2.7/dist-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request.    method='POST', url=token_uri, headers=headers, body=body).  File "/usr/local/lib/python2.7/dist-packages/google_auth_httplib2.py", line 119, in __call__.    raise exceptions.TransportError(exc).TransportError: Unable to find the server at accounts.google.com.

That last line in the stacktrace above has its own stacktrace built-in. I've expanded it below for ease of reading:

Unauthorized: 401 Traceback (most recent call last):.
  File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 154, in grpc._cython.cygrpc.plugin_get_metadata (src/python/grpcio/grpc/_cython/cygrpc.c:6918).
  File "/usr/local/lib/python2.7/dist-packages/grpc/_plugin_wrapping.py", line 107, in __call__.
    AuthMetadataPluginCallback(wrapped_cygrpc_callback)).
  File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/grpc.py", line 65, in __call__.
    callback(self._get_authorization_headers(), None).
  File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/grpc.py", line 51, in _get_authorization_headers.
    self._credentials.refresh(self._request).
  File "/usr/local/lib/python2.7/dist-packages/google/oauth2/service_account.py", line 313, in refresh.
    request, self._token_uri, assertion).
  File "/usr/local/lib/python2.7/dist-packages/google/oauth2/_client.py", line 143, in jwt_grant.
    response_data = _token_endpoint_request(request, token_uri, body).
  File "/usr/local/lib/python2.7/dist-packages/google/oauth2/_client.py", line 104, in _token_endpoint_request.
    method='POST', url=token_uri, headers=headers, body=body).
  File "/usr/local/lib/python2.7/dist-packages/google_auth_httplib2.py", line 119, in __call__.
    raise exceptions.TransportError(exc).TransportError: Unable to find the server at accounts.google.com.

I am running this within a Docker container orchestrated by Kubernetes. My Dockerfile inherits from ubuntu:trusty. Below is version information:

# lsb_release
LSB Version:    core-2.0-amd64:core-2.0-noarch:core-3.0-amd64:core-3.0-noarch:core-3.1-amd64:core-3.1-noarch:core-3.2-amd64:core-3.2-noarch:core-4.0-amd64:core-4.0-noarch:core-4.1-amd64:core-4.1-noarch:security-4.0-amd64:security-4.0-noarch:security-4.1-amd64:security-4.1-noarch
# python --version
Python 2.7.6

[daspecster]

@speedplane, if it's intermittent, I would lean towards a networking flutter. If you retry the request does it succeed?

[speedplane]

@daspecster Normally, I would agree, but this is happening far too often to be a flutter. The image below is a short log of the cases in which I am seeing this, notice how often it occurs. Further, it would be a bit disconcerting if a Google service cannot reach accounts.google.com, I would expect the internal google network to be pretty good. Also, I am getting a 401, not a 503, suggesting that something weird is happening with authentication/credentials, not the network. Finally, I send an email to myself every time I receive this error, so clearly the network is working.

[lukesneeringer]

@speedplane Are you still seeing this error? (I am baffled at what the problem might be.)

[speedplane]

@lukesneeringer I've mitigated it. It turns out I was creating a new datastore client object for each datastore query. I now create a single global object and reuse it and the problem went away. While this was likely my fault, it would be nice if I got some warning on this.

Sadly, although this problem went away, I started getting a new problem: 503 warnings, see #3128

[lukesneeringer]

Got it. Thanks.

[cobookman]

@speedplane One of our appengine flex tutorials generates the datastore client for each incoming request in a http handler: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/flexible/datastore/main.py#L38.

I'll work with that team on fixing the sample code.

[naoko]

I'm actually getting this intermittent problem although I instantiate Client object only one time per life of application. I do get intermittent 503 too but per Google Doc suggestion, I've handled with back off strategy and retry but I do not want to add 401 as part of back off strategy...