Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use apitools for uploads/downloads #334

Closed
silvolu opened this issue Nov 4, 2014 · 13 comments · Fixed by #407
Closed

Use apitools for uploads/downloads #334

silvolu opened this issue Nov 4, 2014 · 13 comments · Fixed by #407
Assignees
Labels
api: storage Issues related to the Cloud Storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@silvolu
Copy link
Contributor

silvolu commented Nov 4, 2014

uploads_test.py and downloads_test.py should provide good examples for it.

@silvolu silvolu added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. api: storage Issues related to the Cloud Storage API. labels Nov 4, 2014
@silvolu silvolu added this to the M1: core, datastore & storage milestone Nov 4, 2014
@tseaver tseaver self-assigned this Nov 20, 2014
@tseaver
Copy link
Contributor

tseaver commented Nov 20, 2014

I'm working on this, but maybe need some help from @craigcitro . The bits of apitools we need to do the upload / download still have dependencies on protorpc, even outside code generation / generated code:

apitools.base.py.base_api

  • defines two classes derived from protorpc.messages.Message
  • uses protorpc.messages.ValidationError and protorpc.messages.MessageFields classes for error handling / validation.
  • Its _LoadClass uses the protorpc.message_types module for "simple" / scalar types.

apitools.base.py.encoding

  • depends heavily on protorpc.messages.Message (its whole raison-d'etre is marshalling JSON using messages as schema).
  • uses the protorpc.message_types module, but in only one place (and the usage is somewhat odd, excluding "normal" marshalling of protorpc.message_types.DateTimeField)
  • uses the protorpc.protorpc.ProtoJson class to do a "deep copy", and has a helper class derived from it (used from MessageToJson() and JsonToMessage().

apitools.base.py.extra_types

  • depends heavily on protorpc.messages.Message.
  • re-exports protorpc.message_types.DateTimeMessage
  • adds custom marshalling for protorpc.message_types.DateTimeField.
  • uses the protorpc.protorpc.ProtoJson class in its cusoom marshlling of protorpc.message_types.DateTimeField.

For us to use apitools, we really need either a version of google-protorpc that supports our desired Python versions, or else we need to fork the needed bits of both into our codebase. Note that the protorpc modules we use (unlike most of the rest) don't pull in a dependency on GAE, so maybe we should be pulling them out somehow.

@tseaver
Copy link
Contributor

tseaver commented Nov 20, 2014

@dhermes you've got some residual memories of protorpc -- maybe you can suggest something?

@craigcitro
Copy link
Contributor

I'm actively working on getting the protorpc fixes pushed through -- it's going well, i've just got a few too many balls in the air right now.

how far away is protorpc for python3 support? (you'd think i would know, but haven't looked yet.)

@tseaver
Copy link
Contributor

tseaver commented Nov 20, 2014

It is hard to estimate, given that running the tests is an unknown quantity. I've done some work today on adding tox support, but the tests don't all pass in the absence of the not-explicitly-depended-on GAE / protobuf code. I will post a gist showing the modernize results (after cleaning it up a bit).

@dhermes
Copy link
Contributor

dhermes commented Nov 20, 2014

RE: "residual memories" I never got tests running outside of Google's BUILD environment, so I am in the same boat as @tseaver.

As for the usefulness of protorpc, I am not sold on it if we are only using it in generated code.

The value in user-land was that people could create protorpc.Message objects which behaved "exactly" like a typed object (something obviously not supported in Python).

We could write something much simpler given a JSON schema:
https://gist.github.com/dhermes/2cbf4352146239e1aca1

I don't know if @craigcitro has feelings on this, but I'd guess he's thought about it more than I have.

@tseaver
Copy link
Contributor

tseaver commented Nov 20, 2014

As for the usefulness of protorpc, I am not sold on it if we are only using it in generated code.

We use a fairly narrow slice of it in non-generated code (see above).

@tseaver
Copy link
Contributor

tseaver commented Nov 20, 2014

@craigcitro here is the gist I promised (I'd be glad to kick this in as another Hg branch):

https://gist.github.com/tseaver/6d28ef139b2a0dcb2eea

@silvolu
Copy link
Contributor Author

silvolu commented Nov 20, 2014

Couldn't we just use transfer.py for uploads and downloads?
There shouldn't be no need for protorpc in that case.

@tseaver
Copy link
Contributor

tseaver commented Nov 20, 2014

@silvolu looks like we could: that would use http_wrapper, exceptions, and util as well, and none of those import anything from protorpc. We still can't ship with a dependency on protorpc if it doesn't support the versions we want: I guess we could just fork those four modules (they don't have tests) into a "vendored" sub-package for now.

@silvolu
Copy link
Contributor Author

silvolu commented Nov 20, 2014

Yeah, that's the plan B mentioned in the doc I sent along.

@tseaver
Copy link
Contributor

tseaver commented Nov 25, 2014

@silvolu @craigcitro I'm not sure we can make use of apitools.base.py.transfer directly without the support of the stuff in apitools.base.py.base_api. In particular, transfer.Upload.InitializeUpload() and transfer.Download.InitializeDownload() APIs expect to be handed http_request and client arguments which are set up inside base_api.BaseApiService._RunMethod(), which has lots more hair than we should be replicating.

@tseaver
Copy link
Contributor

tseaver commented Nov 25, 2014

@craigcitro maybe I should rephrase that as: "can you point me to an example that uses only apitools.base.py.transfer (and its imports) to do upload / download to gcloud-storage?

@craigcitro
Copy link
Contributor

@tseaver the transfer code was indeed built to be used with base_api. i'll try and take a look at whether/how to disentangle those today.

tseaver added a commit that referenced this issue Dec 2, 2014
@jgeewax jgeewax modified the milestones: M1: core, datastore & storage, Storage Stable Jan 30, 2015
atulep pushed a commit that referenced this issue Apr 3, 2023
fix(deps): require proto-plus>=1.15.0
atulep pushed a commit that referenced this issue Apr 6, 2023
fix(deps): require proto-plus>=1.15.0
atulep pushed a commit that referenced this issue Apr 6, 2023
fix(deps): require proto-plus>=1.15.0
atulep pushed a commit that referenced this issue Apr 18, 2023
fix(deps): require proto-plus>=1.15.0
parthea pushed a commit that referenced this issue Jun 4, 2023
* docs: Add documentation for enums

fix: Add context manager return types

chore: Update gapic-generator-python to v1.8.1
PiperOrigin-RevId: 503210727

Source-Link: googleapis/googleapis@a391fd1

Source-Link: googleapis/googleapis-gen@0080f83
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDA4MGY4MzBkZWMzN2MzMzg0MTU3MDgyYmNlMjc5ZTM3MDc5ZWE1OCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this issue Jun 4, 2023
* chore: Update gapic-generator-python to v1.8.2

PiperOrigin-RevId: 504289125

Source-Link: googleapis/googleapis@38a48a4

Source-Link: googleapis/googleapis-gen@b2dc226
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYjJkYzIyNjYzZGJlNDdhOTcyYzhkOGMyZjhhNGRmMDEzZGFmZGNiYyJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this issue Jun 4, 2023
Follow-up on #305 — send cl/520747639 to address sample includes, so do not merge until snippetbot is happy. Thanks!
parthea pushed a commit that referenced this issue Jun 4, 2023
…334)

Source-Link: googleapis/synthtool@8e55b32
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:c6c965a4bf40c19011b11f87dbc801a66d3a23fbc6704102be064ef31c51f1c3
parthea pushed a commit that referenced this issue Jul 6, 2023
- [ ] Regenerate this pull request now.

PiperOrigin-RevId: 459095142

Source-Link: googleapis/googleapis@4f1be99

Source-Link: googleapis/googleapis-gen@ae686d9
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYWU2ODZkOWNkZTRmYzNlMzZkMGFjMDJlZmI4NjQzYjE1ODkwYzFlZCJ9

feat: add audience parameter
PiperOrigin-RevId: 456827138

Source-Link: googleapis/googleapis@23f1a15

Source-Link: googleapis/googleapis-gen@4075a85
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNDA3NWE4NTE0ZjY3NjY5MWVjMTU2Njg4YTViYmYxODNhYTk4OTNjZSJ9
parthea pushed a commit that referenced this issue Sep 20, 2023
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this issue Sep 20, 2023
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this issue Sep 22, 2023
* chore: Update gapic-generator-python to v1.8.2

PiperOrigin-RevId: 504289125

Source-Link: googleapis/googleapis@38a48a4

Source-Link: googleapis/googleapis-gen@b2dc226
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYjJkYzIyNjYzZGJlNDdhOTcyYzhkOGMyZjhhNGRmMDEzZGFmZGNiYyJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this issue Sep 22, 2023
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
parthea added a commit that referenced this issue Sep 22, 2023
* chore(python): use cov_level in unittest gh action

Source-Link: googleapis/synthtool@e5aaa84
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:d22cd2ddce65fdac6986f115563faf2fc81482b09dfbea83ac2808c92ecfdff0

* set coverage to 100%

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea pushed a commit that referenced this issue Sep 22, 2023
* chore(deps): update all dependencies

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea added a commit that referenced this issue Oct 21, 2023
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea added a commit that referenced this issue Oct 21, 2023
* fix(deps): allow protobuf 3.19.5

* explicitly exclude protobuf 4.21.0
parthea pushed a commit that referenced this issue Oct 21, 2023
Source-Link: googleapis/synthtool@703554a
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:94961fdc5c9ca6d13530a6a414a49d2f607203168215d074cdb0a1df9ec31c0b
parthea pushed a commit that referenced this issue Oct 21, 2023
Source-Link: googleapis/synthtool@7ff4aad
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:462782b0b492346b2d9099aaff52206dd30bc8e031ea97082e6facecc2373244
parthea pushed a commit that referenced this issue Oct 21, 2023
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
parthea pushed a commit that referenced this issue Oct 21, 2023
* chore: Update gapic-generator-python to v1.8.2

PiperOrigin-RevId: 504289125

Source-Link: googleapis/googleapis@38a48a4

Source-Link: googleapis/googleapis-gen@b2dc226
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYjJkYzIyNjYzZGJlNDdhOTcyYzhkOGMyZjhhNGRmMDEzZGFmZGNiYyJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
parthea added a commit that referenced this issue Oct 21, 2023
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea pushed a commit that referenced this issue Oct 21, 2023
Source-Link: googleapis/synthtool@56da63e
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:993a058718e84a82fda04c3177e58f0a43281a996c7c395e0a56ccc4d6d210d7
parthea added a commit that referenced this issue Oct 22, 2023
fix(deps): require proto-plus>=1.15.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants