Thread-ConsumeBidirectionalStream caught unexpected exception 'NoneType' object has no attribute 'Call' and will exit

[orby]

Environment details

• OS type and version:
  • FROM python:3.7.9-slim-stretch
• Python version: python --version
  • 3.7.9
• pip version: pip --version
  • 20.2
• google-api-core version: pip show google-api-core
  • 2.2.2

Steps to reproduce

1. Make a Subscriber using google-cloud-pubsub 2.9.0
2. await new messages
3. After 30 minutes you get the unexpected exception

Stack trace

Thread-ConsumeBidirectionalStream caught unexpected exception 'NoneType' object has no attribute 'Call' and will exit.

Traceback (most recent call last):
    File "/root/.local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 106, in __next__
        return next(self._wrapped)
    File "/root/.local/lib/python3.7/site-packages/grpc/_channel.py", line 426, in __next__
        return self._next()
    File "/root/.local/lib/python3.7/site-packages/grpc/_channel.py", line 826, in _next
        raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.INTERNAL
    details = "Received RST_STREAM with error code 2"
    debug_error_string = "{"created":"@1637058210.761422301","description":"Error received from peer ipv4:173.194.210.95:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Received RST_STREAM with error code 2","grpc_status":13}"
    
    
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
    File "/root/.local/lib/python3.7/site-packages/google/api_core/bidi.py", line 655, in _thread_main
        response = self._bidi_rpc.recv()
    File "/root/.local/lib/python3.7/site-packages/google/api_core/bidi.py", line 561, in recv
        return self._recoverable(self._recv)
    File "/root/.local/lib/python3.7/site-packages/google/api_core/bidi.py", line 520, in _recoverable
        raise exc
    File "/root/.local/lib/python3.7/site-packages/google/api_core/bidi.py", line 504, in _recoverable
        return method(*args, **kwargs)
    File "/root/.local/lib/python3.7/site-packages/google/api_core/bidi.py", line 558, in _recv
        return next(call)
    File "/root/.local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 109, in __next__
        raise exceptions.from_grpc_error(exc) from exc
    File "/root/.local/lib/python3.7/site-packages/google/api_core/exceptions.py", line 532, in from_grpc_error
        if isinstance(rpc_exc, grpc.Call) or _is_informative_grpc_error(rpc_exc):
AttributeError: 'NoneType' object has no attribute 'Call'"

[parthea]

Hi @orby,

Please could you share sample code to help reproduce the issue?

[orby]

Hi @parthea

Source code to reproduce: https://gist.github.com/orby/30c72d3e05f7022fd02f47f91e0b2acc

The unexpected exception happens after 30 minutes.

[liu-du]

Hi,

Experiencing the same issue recently (was working before), it happens immediately in my case. The stacktrace is:

2022-01-31 00:34:21 INFO google.api_core.bidi Thread-ConsumeBidirectionalStream exiting
2022-01-31 00:34:21 INFO google.cloud.pubsub_v1.subscriber._protocol.leaser Thread-LeaseMaintainer exiting.
2022-01-31 00:34:21 INFO google.cloud.pubsub_v1.subscriber._protocol.heartbeater Thread-Heartbeater exiting.
2022-01-31 00:34:41 INFO google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager RPC termination has signaled streaming pull manager shutdown.
2022-01-31 00:34:41 INFO google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager Observed non-terminating stream error 'NoneType' object has no attribute 'Call'
2022-01-31 00:34:41 INFO google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager Observed non-recoverable stream error 'NoneType' object has no attribute 'Call'
2022-01-31 00:34:41 ERROR google.api_core.bidi Thread-ConsumeBidirectionalStream caught unexpected exception 'NoneType' object has no attribute 'Call' and will exit.
Traceback (most recent call last):
  File "/opt/atlassian/pipelines/agent/build/discovery-client/.tox/py38-bravado11-googleoauth04/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 106, in __next__
    return next(self._wrapped)
  File "/opt/atlassian/pipelines/agent/build/discovery-client/.tox/py38-bravado11-googleoauth04/lib/python3.8/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/opt/atlassian/pipelines/agent/build/discovery-client/.tox/py38-bravado11-googleoauth04/lib/python3.8/site-packages/grpc/_channel.py", line 826, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.CANCELLED
	details = "Locally cancelled by application!"
	debug_error_string = "None"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/atlassian/pipelines/agent/build/discovery-client/.tox/py38-bravado11-googleoauth04/lib/python3.8/site-packages/google/api_core/bidi.py", line 655, in _thread_main
    response = self._bidi_rpc.recv()
  File "/opt/atlassian/pipelines/agent/build/discovery-client/.tox/py38-bravado11-googleoauth04/lib/python3.8/site-packages/google/api_core/bidi.py", line 561, in recv
    return self._recoverable(self._recv)
  File "/opt/atlassian/pipelines/agent/build/discovery-client/.tox/py38-bravado11-googleoauth04/lib/python3.8/site-packages/google/api_core/bidi.py", line 520, in _recoverable
    raise exc
  File "/opt/atlassian/pipelines/agent/build/discovery-client/.tox/py38-bravado11-googleoauth04/lib/python3.8/site-packages/google/api_core/bidi.py", line 504, in _recoverable
    return method(*args, **kwargs)
  File "/opt/atlassian/pipelines/agent/build/discovery-client/.tox/py38-bravado11-googleoauth04/lib/python3.8/site-packages/google/api_core/bidi.py", line 558, in _recv
    return next(call)
  File "/opt/atlassian/pipelines/agent/build/discovery-client/.tox/py38-bravado11-googleoauth04/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 109, in __next__
    raise exceptions.from_grpc_error(exc) from exc
  File "/opt/atlassian/pipelines/agent/build/discovery-client/.tox/py38-bravado11-googleoauth04/lib/python3.8/site-packages/google/api_core/exceptions.py", line 592, in from_grpc_error
    if isinstance(rpc_exc, grpc.Call) or _is_informative_grpc_error(rpc_exc):
AttributeError: 'NoneType' object has no attribute 'Call'

[atulep]

@liu-du Can you post your code and Google dependencies? Similar to what @orby did above.