BigTable: Table.read_row: StatusCode.INTERNAL, GOAWAY received

[pyk]

Hey, I have application running on Google App Engine. When executing read_row
It raises grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INTERNAL, GOAWAY received).

Full stack trace:

Traceback (most recent call last):
  File "/home/vmagent/app/server.py", line 186, in alias
    user_alias_id = api.handler.alias(user_id, user_alias_id)
  File "/home/vmagent/app/handler.py", line 97, in alias
    existing_alias = self.bt.get_identifier(user_id)
  File "/home/vmagent/app/bigtable.py", line 99, in get_identifier
    row = self.identifier_table.read_row(row_key)
  File "/env/lib/python3.6/site-packages/google/cloud/bigtable/table.py", line 267, in read_row
    rows_data.consume_all()
  File "/env/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 327, in consume_all
    self.consume_next()
  File "/env/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 261, in consume_next
    response = six.next(self._response_iterator)
  File "/env/lib/python3.6/site-packages/grpc/_channel.py", line 344, in __next__
    return self._next()
  File "/env/lib/python3.6/site-packages/grpc/_channel.py", line 338, in _next
    raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INTERNAL, GOAWAY received)>

Any idea why this is happen? I can't reproduce the exception (the exception happen only once).

another question: Is it okay if I set GOOGLE_CLOUD_DISABLE_GRPC=true in production environment?

Thanks!

OS type and version

Google app engine custom runtime

FROM gcr.io/google-appengine/python

Python version

python3.6

Packages

gapic-google-cloud-datastore-v1==0.15.3
gapic-google-cloud-error-reporting-v1beta1==0.15.3
gapic-google-cloud-logging-v2==0.91.3
google-api-core==0.1.3
google-api-python-client==1.6.4
google-auth==1.2.1
google-cloud==0.32.0
google-cloud-bigquery==0.28.0
google-cloud-bigquery-datatransfer==0.1.0
google-cloud-bigtable==0.28.1
google-cloud-container==0.1.0
google-cloud-core==0.28.0
google-cloud-datastore==1.4.0
google-cloud-dns==0.28.0
google-cloud-error-reporting==0.28.0
google-cloud-firestore==0.28.0
google-cloud-language==1.0.0
google-cloud-logging==1.4.0
google-cloud-monitoring==0.28.0
google-cloud-pubsub==0.30.1
google-cloud-resource-manager==0.28.0
google-cloud-runtimeconfig==0.28.0
google-cloud-spanner==0.29.0
google-cloud-speech==0.30.0
google-cloud-storage==1.6.0
google-cloud-trace==0.17.0
google-cloud-translate==1.3.0
google-cloud-videointelligence==1.0.0
google-cloud-vision==0.29.0
google-gax==0.15.16
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
grpc-google-iam-v1==0.11.4
grpcio==1.8.3

[sduskis]

The server-side disconnects every so often and sends a GOAWAY. I'm quite surprised that it results in a StatusCode.INTERNAL. I'll have to ask about that.

The solution here is automatic retries by the client, which is a long standing Feature Request: #2255. We are actively working on this issue.

[pyk]

@sduskis thanks for the explanation, i hope the feature is implemented soon 👍