upgrade google-cloud dependency-package in dbt-bigquery

[hui-zheng]

Describe the feature

We need to use dbt with another Python process requiring google-cloud-firestore in the same local environment.
dbt-bigquery is using an old version of google-cloud-core which somehow is incompatible with any version of google-cloud-firestore.

If installing with latest version of google-cloud-firestore

RUN pip install google-cloud-firestore====2.0.1 \
                dbt-bigquery==0.18.1

it errors

ERROR: Cannot install dbt-bigquery==0.18.1 and google-cloud-firestore==2.0.1 because these package versions have conflicting dependencies.
The conflict is caused by:
    dbt-bigquery 0.18.1 depends on google-cloud-core<1.4 and >=1.3.0
    google-cloud-firestore 2.0.1 depends on google-cloud-core<2.0dev and >=1.4.1

If loosing version of google-cloud-firestore and the let pip choose the compatible packages

RUN pip install google-cloud-firestore====2.0.1 \
                dbt-bigquery==0.18.0

Though The build would pass, it errors when running the python process.

Traceback (most recent call last):
  File "flow.local-test.py", line 1, in <module>
    from flow import flow
  File "/app/flow.py", line 18, in <module>
    from google.cloud import firestore
  File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore.py", line 18, in <module>
    from google.cloud.firestore_v1 import __version__
  File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/__init__.py", line 21, in <module>
    from google.cloud.firestore_v1 import types
  File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/types.py", line 29, in <module>
    from google.cloud.firestore_v1.proto import common_pb2
  File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/proto/common_pb2.py", line 24, in <module>
    create_key=_descriptor._internal_create_key,
AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'

Describe alternatives you've considered

Option 1.
Though it's possible to isolate the run-time environment of dbt and the other python process using pyenv, it requires some changes to other open-source packages to properly invoke dbt in its own pyenv environment, which requires more effort for a patch.

Option 2.
I find that I could force an upgrade of protobuf version,

RUN pip install google-cloud-firestore====2.0.1 \
                dbt-bigquery==0.18.0
RUN pip install --upgrade protobuf

Though it errors below, the build will finish.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
dbt-bigquery 0.18.0 requires protobuf<3.12,>=3.6.0, but you have protobuf 3.14.0 which is incompatible.

When I execute the dbt, It seems working but I think it's prone to break in the future.

Additional context

This is specific to dbt-bigquery.

Who will this benefit?

I think this is relevant to anyone who might use dbt with other python processes involving google-cloud in the same run-time environment.

Are you interested in contributing this feature?

yes, I am.

[jtcohen6]

@hui-zheng We just upgraded these in #2877. Here are the dependencies in dev/kiyoshi-kuromiya, which will be shipped with the next minor version of dbt (v0.19.0):
https://github.com/fishtown-analytics/dbt/blob/cd149b68e83a7d3819bc0e8641f08a43ab4318b2/plugins/bigquery/setup.py#L51-L56