Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: transition the library to microgenerator #62

Merged
merged 21 commits into from
Sep 24, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions .kokoro/release/common.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,14 @@ env_vars: {
value: "github/python-bigquery-storage/.kokoro/release.sh"
}

# Fetch PyPI password
before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73713
keyname: "google_cloud_pypi_password"
}
}
# Fetch PyPI password
before_action {
plamut marked this conversation as resolved.
Show resolved Hide resolved
fetch_keystore {
keystore_resource {
keystore_config_id: 73713
keyname: "google_cloud_pypi_password"
}
}
}

# Tokens needed to report release status back to GitHub
Expand Down
19 changes: 0 additions & 19 deletions CONTRIBUTING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -80,25 +80,6 @@ We use `nox <https://nox.readthedocs.io/en/latest/>`__ to instrument our tests.

.. nox: https://pypi.org/project/nox/

Note on Editable Installs / Develop Mode
========================================

- As mentioned previously, using ``setuptools`` in `develop mode`_
or a ``pip`` `editable install`_ is not possible with this
library. This is because this library uses `namespace packages`_.
For context see `Issue #2316`_ and the relevant `PyPA issue`_.

Since ``editable`` / ``develop`` mode can't be used, packages
need to be installed directly. Hence your changes to the source
tree don't get incorporated into the **already installed**
package.

.. _namespace packages: https://www.python.org/dev/peps/pep-0420/
.. _Issue #2316: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2316
.. _PyPA issue: https://github.com/pypa/packaging-problems/issues/12
.. _develop mode: https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode
.. _editable install: https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs

*****************************************
I'm getting weird errors... Can you help?
*****************************************
Expand Down
11 changes: 7 additions & 4 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,14 @@ dependencies.

Supported Python Versions
^^^^^^^^^^^^^^^^^^^^^^^^^
Python >= 3.5
Python >= 3.6

Deprecated Python Versions
^^^^^^^^^^^^^^^^^^^^^^^^^^
Python == 2.7. Python 2.7 support will be removed on January 1, 2020.
Unsupported Python Versions
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Python == 2.7, Python == 3.5.

The last version of this library compatible with Python 2.7 and 3.5 is
``google-cloud-bigquery-storage==1.1.0``.


Mac/Linux
Expand Down
282 changes: 282 additions & 0 deletions UPGRADING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,282 @@
<!--
Copyright 2020 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->


# 2.0.0 Migration Guide

The 2.0 release of the `google-cloud-bigquery-storage` client is a significant
upgrade based on a [next-gen code generator](https://github.com/googleapis/gapic-generator-python),
and includes substantial interface changes. Existing code written for earlier versions
of this library will likely require updates to use this version. This document
describes the changes that have been made, and what you need to do to update your usage.

If you experience issues or have questions, please file an
[issue](https://github.com/googleapis/python-bigquery-storage/issues).


## Supported Python Versions

> **WARNING**: Breaking change

The 2.0.0 release requires Python 3.6+.


## Import Path

The library was moved into `google.cloud.bigquery` namespace. It is recommended
to use this path in order to reduce the chance of future compatibility issues
in case the library is restuctured internally.

**Before:**
```py
from google.cloud.bigquery_storage_v1 import BigQueryReadClient
```

**After:**
```py
from google.cloud.bigquery.storage import BigQueryReadClient
```


## Enum Types

> **WARNING**: Breaking change

Enum types have been moved. Access them through the `types` module.

**Before:**
```py
from google.cloud.bigquery_storage_v1 import enums

data_format = enums.DataFormat.ARROW
```

data_format = BigQueryReadClient.enums.DataFormat.ARROW

**After:**
```py
from google.cloud.bigquery.storage import types

data_format = types.DataFormat.ARROW
```

Additionally, enums cannot be accessed through the client anymore. The following
code wil _not_ work:
```py
data_format = BigQueryReadClient.enums.DataFormat.ARROW
```


## Clients for Beta APIs

> **WARNING**: Breaking change

Clients for beta APIs have been removed. The following import will _not_ work:

```py
from google.cloud.bigquery_storage_v1beta1 import BigQueryStorageClient
from google.cloud.bigquery_storage_v1beta2.gapic.big_query_read_client import BigQueryReadClient
```

The beta APIs are still available on the server side, but you will need to use
the 1.x version of the library to access them.


## Changed Default Value of the `read_rows()` Method's `metadata` Argument

The `client.read_rows()` method does not accept `None` anymore as a valid value
for the optional `metadata` argument. If not given, an empty tuple is used, but
if you want to explicitly pass an "empty" value, you should use an empty tuple, too.

**Before:**
```py
client.read_rows("stream_name", metadata=None)
```

**After:**
```py
client.read_rows("stream_name", metadata=())
```

plamut marked this conversation as resolved.
Show resolved Hide resolved
OR

```py
client.read_rows("stream_name")
```


## Method Calls

> **WARNING**: Breaking change

Most of the client methods that send requests to the backend expect request objects.
We provide a script that will convert most common use cases.

> One exception to this is the `BigQueryReadClient.read_rows()` which is a hand-written
wrapper around the auto-generated `read_rows()` method.

* Install the library

```py
python3 -m pip install google-cloud-bigquery-storage
```

* The script `fixup_storage_v1_keywords.py` is shipped with the library. It expects
an input directory (with the code to convert) and an empty destination directory.

```sh
$ scripts/fixup_storage_v1_keywords.py --input-directory .samples/ --output-directory samples/
```

**Before:**
```py
from google.cloud import bigquery_storage_v1

client = bigquery_storage_v1.BigQueryReadClient()

requested_session = bigquery_storage_v1.types.ReadSession()
requested_session.table = "projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_ID"
requested_session.data_format = bigquery_storage_v1.enums.DataFormat.ARROW

session = client.create_read_session(
"projects/parent_project",
requested_session,
max_stream_count=1,
)
```

**After:**
```py
from google.cloud.bigquery import storage

client = storage.BigQueryReadClient()

requested_session = storage.types.ReadSession(
table="projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_ID",
data_format=storage.types.DataFormat.ARROW,
)
session = client.create_read_session(
request={
"parent": "projects/parent_project",
"read_session": requested_session,
"max_stream_count" 1,
},
)
```

### More Details

In `google-cloud-bigquery-storage<2.0.0`, parameters required by the API were positional
parameters and optional parameters were keyword parameters.

**Before:**
```py
def create_read_session(
self,
parent,
read_session,
max_stream_count=None,
retry=google.api_core.gapic_v1.method.DEFAULT,
timeout=google.api_core.gapic_v1.method.DEFAULT,
metadata=None,
):
```

In the `2.0.0` release, methods that interact with the backend have a single
positional parameter `request`. Method docstrings indicate whether a parameter is
required or optional.

Some methods have additional keyword only parameters. The available parameters depend
on the [`google.api.method_signature` annotation](https://github.com/googleapis/python-bigquery-storage/blob/9e1bf910e6f5010f479cf4592e25c3b3eebb456d/google/cloud/bigquery_storage_v1/proto/storage.proto#L73)
specified by the API producer.


**After:**
```py
def create_read_session(
self,
request: storage.CreateReadSessionRequest = None,
*,
parent: str = None,
read_session: stream.ReadSession = None,
max_stream_count: int = None,
retry: retries.Retry = gapic_v1.method.DEFAULT,
timeout: float = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> stream.ReadSession:
```

> **NOTE:** The `request` parameter and flattened keyword parameters for the API are
> mutually exclusive. Passing both will result in an error.

Both of these calls are valid:

```py
session = client.create_read_session(
request={
"parent": "projects/parent_project",
"read_session": requested_session,
"max_stream_count" 1,
},
)
```

```py
response = client.create_read_session(
parent="projects/parent_project",
read_session=requested_session,
max_stream_count=1,
)
```

This call is _invalid_ because it mixes `request` with a keyword argument
`max_stream_count`. Executing this code will result in an error:

```py
session = client.create_read_session(
request={
"parent": "projects/parent_project",
"read_session": requested_session,
},
max_stream_count=1,
)
```

> **NOTE:** The `request` parameter of some methods can also contain a more rich set of
> options that are otherwise not available as explicit keyword only parameters, thus
> these _must_ be passed through `request`.


## Removed Utility Methods

> **WARNING**: Breaking change

Several utility methods such as `project_path()` and `table_path()` have been removed.
These paths must now be constructed manually:

```py
project_path = f"project/{PROJECT_ID}"
table_path = f"projects/{PROJECT_ID}/datasets/{DATASET_ID}/tables/{TABLE_ID}"
```

The two that remained are `read_session_path()` and `read_stream_path()`.


## Removed `client_config` and `channel` Parameter

The client cannot be constructed with `channel` or `client_config` arguments anymore,
these deprecated parameters have been removed.

If you used `client_config` to customize retry and timeout settings for a particular
method, you now need to do it upon method invocation by passing the custom `timeout` and
`retry` arguments, respectively.
1 change: 1 addition & 0 deletions docs/UPGRADING.md
1 change: 1 addition & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
"sphinx.ext.autosummary",
"sphinx.ext.intersphinx",
"sphinx.ext.coverage",
"sphinx.ext.doctest",
"sphinx.ext.napoleon",
"sphinx.ext.todo",
"sphinx.ext.viewcode",
Expand Down
6 changes: 0 additions & 6 deletions docs/gapic/v1/api.rst

This file was deleted.

5 changes: 0 additions & 5 deletions docs/gapic/v1/types.rst

This file was deleted.

6 changes: 0 additions & 6 deletions docs/gapic/v1beta1/api.rst

This file was deleted.

6 changes: 0 additions & 6 deletions docs/gapic/v1beta1/reader.rst

This file was deleted.

5 changes: 0 additions & 5 deletions docs/gapic/v1beta1/types.rst

This file was deleted.

Loading