Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: transition the library to microgenerator #62

Merged
merged 21 commits into from
Sep 24, 2020
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions .kokoro/release/common.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,14 @@ env_vars: {
value: "github/python-bigquery-storage/.kokoro/release.sh"
}

# Fetch PyPI password
before_action {
fetch_keystore {
keystore_resource {
keystore_config_id: 73713
keyname: "google_cloud_pypi_password"
}
}
# Fetch PyPI password
before_action {
plamut marked this conversation as resolved.
Show resolved Hide resolved
fetch_keystore {
keystore_resource {
keystore_config_id: 73713
keyname: "google_cloud_pypi_password"
}
}
}

# Tokens needed to report release status back to GitHub
Expand Down
19 changes: 0 additions & 19 deletions CONTRIBUTING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -80,25 +80,6 @@ We use `nox <https://nox.readthedocs.io/en/latest/>`__ to instrument our tests.

.. nox: https://pypi.org/project/nox/

Note on Editable Installs / Develop Mode
========================================

- As mentioned previously, using ``setuptools`` in `develop mode`_
or a ``pip`` `editable install`_ is not possible with this
library. This is because this library uses `namespace packages`_.
For context see `Issue #2316`_ and the relevant `PyPA issue`_.

Since ``editable`` / ``develop`` mode can't be used, packages
need to be installed directly. Hence your changes to the source
tree don't get incorporated into the **already installed**
package.

.. _namespace packages: https://www.python.org/dev/peps/pep-0420/
.. _Issue #2316: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2316
.. _PyPA issue: https://github.com/pypa/packaging-problems/issues/12
.. _develop mode: https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode
.. _editable install: https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs

*****************************************
I'm getting weird errors... Can you help?
*****************************************
Expand Down
1 change: 1 addition & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
"sphinx.ext.autosummary",
"sphinx.ext.intersphinx",
"sphinx.ext.coverage",
"sphinx.ext.doctest",
"sphinx.ext.napoleon",
"sphinx.ext.todo",
"sphinx.ext.viewcode",
Expand Down
6 changes: 0 additions & 6 deletions docs/gapic/v1/api.rst

This file was deleted.

5 changes: 0 additions & 5 deletions docs/gapic/v1/types.rst

This file was deleted.

6 changes: 0 additions & 6 deletions docs/gapic/v1beta1/api.rst

This file was deleted.

6 changes: 0 additions & 6 deletions docs/gapic/v1beta1/reader.rst

This file was deleted.

5 changes: 0 additions & 5 deletions docs/gapic/v1beta1/types.rst

This file was deleted.

6 changes: 0 additions & 6 deletions docs/gapic/v1beta2/api.rst

This file was deleted.

5 changes: 0 additions & 5 deletions docs/gapic/v1beta2/types.rst

This file was deleted.

11 changes: 2 additions & 9 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,8 @@ API Reference
.. toctree::
:maxdepth: 2

gapic/v1beta1/api
gapic/v1beta1/reader
gapic/v1beta1/types

gapic/v1beta2/api
gapic/v1beta2/types

gapic/v1/api
gapic/v1/types
storage_v1/services
plamut marked this conversation as resolved.
Show resolved Hide resolved
storage_v1/types

Example Usage
-------------
Expand Down
6 changes: 6 additions & 0 deletions docs/storage_v1/services.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Services for Google Cloud Bigquery Storage v1 API
=================================================

.. automodule:: google.cloud.bigquery.storage_v1.services.big_query_read
:members:
:inherited-members:
5 changes: 5 additions & 0 deletions docs/storage_v1/types.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Types for Google Cloud Bigquery Storage v1 API
==============================================

.. automodule:: google.cloud.bigquery.storage_v1.types
:members:
56 changes: 56 additions & 0 deletions google/cloud/bigquery/storage/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# -*- coding: utf-8 -*-

# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

from google.cloud.bigquery_storage_v1 import BigQueryReadAsyncClient
from google.cloud.bigquery_storage_v1 import BigQueryReadClient
from google.cloud.bigquery_storage_v1 import types
from google.cloud.bigquery_storage_v1 import __version__
from google.cloud.bigquery.storage_v1.types.arrow import ArrowRecordBatch
from google.cloud.bigquery.storage_v1.types.arrow import ArrowSchema
from google.cloud.bigquery.storage_v1.types.avro import AvroRows
from google.cloud.bigquery.storage_v1.types.avro import AvroSchema
from google.cloud.bigquery.storage_v1.types.storage import CreateReadSessionRequest
from google.cloud.bigquery.storage_v1.types.storage import ReadRowsRequest
from google.cloud.bigquery.storage_v1.types.storage import ReadRowsResponse
from google.cloud.bigquery.storage_v1.types.storage import SplitReadStreamRequest
from google.cloud.bigquery.storage_v1.types.storage import SplitReadStreamResponse
from google.cloud.bigquery.storage_v1.types.storage import StreamStats
from google.cloud.bigquery.storage_v1.types.storage import ThrottleState
from google.cloud.bigquery.storage_v1.types.stream import DataFormat
from google.cloud.bigquery.storage_v1.types.stream import ReadSession
from google.cloud.bigquery.storage_v1.types.stream import ReadStream

__all__ = (
"__version__",
"types",
"ArrowRecordBatch",
"ArrowSchema",
"AvroRows",
"AvroSchema",
"BigQueryReadAsyncClient",
"BigQueryReadClient",
"CreateReadSessionRequest",
"DataFormat",
"ReadRowsRequest",
"ReadRowsResponse",
"ReadSession",
"ReadStream",
"SplitReadStreamRequest",
"SplitReadStreamResponse",
"StreamStats",
"ThrottleState",
)
2 changes: 2 additions & 0 deletions google/cloud/bigquery/storage/py.typed
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Marker file for PEP 561.
# The google-cloud-bigquery-storage package uses inline types.
53 changes: 53 additions & 0 deletions google/cloud/bigquery/storage_v1/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# -*- coding: utf-8 -*-

# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

from .services.big_query_read import BigQueryReadAsyncClient
from .services.big_query_read import BigQueryReadClient
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's have this be the manual wrapper.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a generated file and bigquery_storage_v1/client.py imports from it. The latter assumes that the generated client is imported and wraps it, although it could also import directly from storage_v1.services.big_query_read, in principle.

Do you think it's worth maintaining additional synth replacement rules to prevent users from accessing the generated client through google.cloud.bigquery.storage_v1?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it's worth maintaining additional synth replacement rules to prevent users from accessing the generated client through google.cloud.bigquery.storage_v1?

I'm pretty sure that's what I did for the bigquery_storage_v1beta1 package (that is, exclude the equivalent top-level __init__.py from getting copied over.

I'm still flip-flopping about what namespace we should use for the manual clients. It's nice to avoid breaking changes, but if we're breaking anyway, maybe we should be consistent. What do other manual libs do for namespace of manual clients?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PubSub just exposes the generated clients through pubsub_v1.__init__.py import.

I don't know about other libs, but can check (I suspect they left these imports intact, though).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting! I find that it uses the google.pubsub_v1 for generated and google.cloud.pubsub_v1 for handwritten to be pretty confusing, but I guess so long as it's documented well, it'll be okay.

Copy link
Contributor Author

@plamut plamut Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can at least mention this in the UPGRADING guide and just recommend to import from google.cloud.bigquery.storage (the samples use this path, too). This will give us more freedom to restructure the library internally in the future (in theory at least).

Update: FWIW, I found another library with hand-written layer that has already been migrated to microgenerator (AutoML) and it, too, exposes the generated client through google.cloud.automl_v1.

from .types.arrow import ArrowRecordBatch
from .types.arrow import ArrowSchema
from .types.avro import AvroRows
from .types.avro import AvroSchema
from .types.storage import CreateReadSessionRequest
from .types.storage import ReadRowsRequest
from .types.storage import ReadRowsResponse
from .types.storage import SplitReadStreamRequest
from .types.storage import SplitReadStreamResponse
from .types.storage import StreamStats
from .types.storage import ThrottleState
from .types.stream import DataFormat
from .types.stream import ReadSession
from .types.stream import ReadStream


__all__ = (
"ArrowRecordBatch",
"ArrowSchema",
"AvroRows",
"AvroSchema",
"CreateReadSessionRequest",
"DataFormat",
"ReadRowsRequest",
"ReadRowsResponse",
"ReadSession",
"ReadStream",
"SplitReadStreamRequest",
"SplitReadStreamResponse",
"StreamStats",
"ThrottleState",
"BigQueryReadAsyncClient",
"BigQueryReadClient",
)
2 changes: 2 additions & 0 deletions google/cloud/bigquery/storage_v1/py.typed
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Marker file for PEP 561.
# The google-cloud-bigquery-storage package uses inline types.
16 changes: 16 additions & 0 deletions google/cloud/bigquery/storage_v1/services/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# -*- coding: utf-8 -*-

# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
Original file line number Diff line number Diff line change
@@ -1,29 +1,24 @@
# -*- coding: utf-8 -*-
#

# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#


from __future__ import absolute_import

from google.cloud.bigquery_storage_v1 import BigQueryReadClient
from google.cloud.bigquery_storage_v1 import enums
from google.cloud.bigquery_storage_v1 import types

from .client import BigQueryReadClient
from .async_client import BigQueryReadAsyncClient

__all__ = (
"enums",
"types",
"BigQueryReadClient",
"BigQueryReadAsyncClient",
)
Loading