Packaging issue with GRPC and google-cloud-bigquery + optional packages?

[shollyman]

We're apparently losing some needed dependencies when using the optional installs for google-cloud-bigquery

$ ~/.pyenv/versions/3.8.2/bin/virtualenv repro
$ source repro/bin/activate
$ pip freeze 

(no results)

$ pip install --upgrade google-cloud-bigquery[bqstorage,pandas]
$ pip freeze > before.txt
$ cat before.txt

cachetools==4.1.0
certifi==2020.4.5.1
chardet==3.0.4
google-api-core==1.17.0
google-auth==1.14.1
google-cloud-bigquery==1.24.0
google-cloud-bigquery-storage==0.8.0
google-cloud-core==1.3.0
google-resumable-media==0.5.0
googleapis-common-protos==1.51.0
idna==2.9
numpy==1.18.3
pandas==1.0.3
protobuf==3.11.3
pyarrow==0.17.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
python-dateutil==2.8.1
pytz==2019.3
requests==2.23.0
rsa==4.0
six==1.14.0
urllib3==1.25.9

$ python
>>> from google.cloud import bigquery_storage_v1beta1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/google/home/shollyman/python_envs/repro/lib/python3.8/site-packages/google/cloud/bigquery_storage_v1beta1/__init__.py", line 26, in <module>
    from google.cloud.bigquery_storage_v1beta1 import client
  File "/usr/local/google/home/shollyman/python_envs/repro/lib/python3.8/site-packages/google/cloud/bigquery_storage_v1beta1/client.py", line 24, in <module>
    import google.api_core.gapic_v1.method
  File "/usr/local/google/home/shollyman/python_envs/repro/lib/python3.8/site-packages/google/api_core/gapic_v1/__init__.py", line 16, in <module>
    from google.api_core.gapic_v1 import config
  File "/usr/local/google/home/shollyman/python_envs/repro/lib/python3.8/site-packages/google/api_core/gapic_v1/config.py", line 23, in <module>
    import grpc
ModuleNotFoundError: No module named 'grpc'

$ pip install google-cloud-bigquery-storage
$ pip freeze > after.txt
$ diff before.txt after.txt

grpcio==1.28.1

[plamut]

@shollyman Does the same happen if installing google-cloud-bigquery[all]? Though grpc should still not be missing even when installing a subset of extras.

Let me see if I can reproduce this locally.

[plamut]

I can confirm this.

The reason seems to be that we already specify google-api-core as a dependency (with no extras). When bqstorage extra tries to install BigQuery storage, the latter also specifies google-api-core[grpc] as its depenency, but with the [grpc] extra.

That grpc extra does not get installed, however. It appears that pip skips it, because it detects that google-api-core is already installed, but forgets about the extra. Only a theory, though, because the following actually works:

$ pip install google-api-core
$ pip install google-api-core[grpc]   # <--- installs grpcio

Update:
It seems to be an issue with the pip's dependency resolver. See this comment for the TL; DR version, which is what happens with our google-api-core dependency.