-
Notifications
You must be signed in to change notification settings - Fork 163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GCS failures with latest version #116
Comments
Probably related to: |
FWIW, same error with 1.2.0 |
Turns out this is likely only GCS (I guess I had confused some of the datastore package names), and the packages that are (were?) having dependency problems are: Hopefully we can resolve these issues without breaking our pinning of the google-cloud-pubsub package to 0.26. |
Stub script to reproduce this: """ from google.cloud import storage file_ = '/tmp/test.txt' c = storage.Client(project='MYPROJECT') |
So, it appears that pinning the pubsub version is what is causing problems (indirectly). When removing the pubsub pinning, the above code runs just fine. Below is the pip freeze diff between two virtualenvs setup with the only difference being the pubsub version being pinned to 0.26. I think the google-cloud-core package is the issue, and is mentioned in the bug referenced above.
|
Ugh, now after pinning google-cloud-core>=0.28.0, something is wrong with psq. Looks like something in the pubsub library? It seems to be the correct version though: """ $ pip list | grep requests $ python -c 'import psq' |
Ok, it seems like this file got moved in this PR: And then the version is updated to 0.27.0, so I guess I need to try to pin 0.26.0. It breaks with 0.25.0 though, so I don't have high hopes. |
Hopefully just needs a minor refactor. Note that PSQ also uses Datastore (transparently) to return task results.
It's broken with:
gapic-google-cloud-datastore-v1 (0.15.3)
google-cloud-datastore (1.4.0)
proto-google-cloud-datastore-v1 (0.90.4)
But works with:
gapic-google-cloud-datastore-v1 (0.15.3)
google-cloud-datastore (1.0.0)
grpc-google-cloud-datastore-v1 (0.14.0)
proto-google-cloud-datastore-v1 (0.90.4)
[INFO] Task failed with exception: [request() got an unexpected keyword argument 'data']
[[INFO] Writing /var/tmp/1511642221-817a08dc1d334e4587df1739da220996-PlasoTask/worker-log.txt to GCS path output/1511642221-817a08dc1d334e4587df1739da220996-PlasoTask/worker-log.txt
DEBUG: traceback.format_exc(): Traceback (most recent call last):
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 312, in run_wrapper
result = self.run(evidence, result)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/plaso.py", line 49, in run
ret, result = self.execute(cmd, result, save_files=[plaso_log], close=True)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 257, in execute
result.close(success=True)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 114, in close
self.save_local_file(logfile)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 177, in save_local_file
new_path = writer.write(file_)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/output_writers.py", line 176, in write
blob.upload_from_filename(file_, client=self.client)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 992, in upload_from_filename
size=total_bytes)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 946, in upload_from_file
client, file_obj, content_type, size, num_retries)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 867, in _do_upload
client, stream, content_type, size, num_retries)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 700, in _do_multipart_upload
transport, data, object_metadata, content_type)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/resumable_media/requests/upload.py", line 97, in transmit
retry_strategy=self._retry_strategy)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/resumable_media/requests/_helpers.py", line 101, in http_request
func, RequestsMixin._get_status_code, retry_strategy)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/resumable_media/_helpers.py", line 146, in wait_and_retry
response = func()
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google_auth_httplib2.py", line 198, in request
uri, method, body=body, headers=request_headers, **kwargs)
TypeError: request() got an unexpected keyword argument 'data'
[INFO] Task failed with exception: [request() got an unexpected keyword argument 'data']
The text was updated successfully, but these errors were encountered: