Encode object name with URI instead of CGI #3081

prapicault · 2019-03-28T14:56:15Z

Hi, this is a fix for issue #3064.
I've updated the code and the tests, and I have also tested the change in my setup.
The only thing that I have not been able to test is whether the bucket name needs to be encoded using CGI or URI.

googlebot · 2019-03-28T14:56:20Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

quartzmo · 2019-03-28T15:39:19Z

Hi @prapicault, thank you for this PR!

The only thing that I have not been able to test is whether the bucket name needs to be encoded using CGI or URI.

I do not believe the bucket name needs to be encoded with either, since it contains only lowercase letters, numbers, dashes (-), underscores (_), and dots (.). (See also Encoding URI path parts.)

@frankyn Does this sound correct to you? Also, will this PR introduce a breaking change for users?

blowmage · 2019-03-28T17:24:08Z

google-cloud-storage/lib/google/cloud/storage/file/signer.rb

@@ -43,9 +43,9 @@ def self.from_bucket bucket, path
          # The external path to the file.
          def ext_path
            escaped_path = String(@path).split("/").map do |node|


This whole method can be replaced with:

Addressable::URI.escape "/#{@bucket}/#{@path}"

blowmage · 2019-03-28T18:05:09Z

Can you also add a direct dependency for the addressable gem to the google-cloud-storage.gemspec file?

prapicault · 2019-03-28T19:33:27Z

Can you also add a direct dependency for the addressable gem to the google-cloud-storage.gemspec file?

Done

prapicault · 2019-03-28T19:45:08Z

Still on working in the CLA :(

frankyn · 2019-03-28T21:14:52Z

LGTM, found a related issue in the google-cloud-python lib: googleapis/google-cloud-python#3809

Percent encode the object name -- except '/' which is done by the split. Please don't remove that.

quartzmo · 2019-03-28T21:25:57Z

google-cloud-storage/lib/google/cloud/storage/file/signer.rb

-              CGI.escape node
-            end.join("/")
-            "/#{CGI.escape @bucket}/#{escaped_path}"
+            Addressable::URI.escape "/#{@bucket}/#{@path}"
          end


@frankyn I think we are relying here on the behavior (seen in the unit tests below) of Addressable::URI.encode to not encode forward slashes at all.

ah, okay, I take back my last comment about not removing the split. Thanks for clarifying @quartzmo.

quartzmo · 2019-03-29T21:29:50Z

Can you please replace require "cgi" with require "addressable/uri" in signer.rb? Thank you!

blowmage · 2019-03-29T21:32:33Z

I believe CGI.encode is still being used in this file, so I would not remove that require.

quartzmo · 2019-03-29T22:35:37Z

True, in url_escape. Should it be?

blowmage · 2019-03-29T23:25:48Z

url_escape should probably be renamed param_escape, and it needs to use CGI. Its different because query strings values are escaped slightly different than paths.

Remove trailing slash from path if no file. Update CGI.escape to Addressable::URI.escape. [refs googleapis#3081]

quartzmo · 2019-04-02T20:15:54Z

@prapicault How is it going? Are you blocked by the CLA? Or anything else?

quartzmo · 2019-04-03T17:55:06Z

@prapicault We are planning a release of google-cloud-storage, it would be great to include this change. Is there anything I can do to help?

quartzmo · 2019-04-04T20:17:21Z

@prapicault We would like to release this fix in the next few working days, are you making any progress with the CLA?

quartzmo · 2019-04-09T15:52:21Z

@prapicault I'm going to re-do this fix today in my own PR, but I will credit you for the find and the idea. I hope this is acceptable. We need to release the fix. Thank you!

blowmage · 2019-04-09T23:25:25Z

This has been superseded by #3109.

prapicault · 2019-04-16T14:28:35Z

Sorry for not being able to follow up. I just got the go to sign the CLA.
@quartzmo Thanks for pushing this forward and for the mention.

Encode object name with URI instead of CGI

0add645

googlebot added the cla: no This human has *not* signed the Contributor License Agreement. label Mar 28, 2019

quartzmo added the api: storage Issues related to the Cloud Storage API. label Mar 28, 2019

quartzmo requested a review from frankyn March 28, 2019 15:31

blowmage reviewed Mar 28, 2019

View reviewed changes

prapicault added 2 commits March 28, 2019 15:31

Simplify implementation of ext_path

a747a4c

Add addressable to cloud-storage gemspec

31cbc13

quartzmo reviewed Mar 28, 2019

View reviewed changes

quartzmo mentioned this pull request Mar 29, 2019

Add support for Storage V4 signed URLs #3057

Merged

7 tasks

quartzmo added a commit to quartzmo/google-cloud-ruby that referenced this pull request Apr 2, 2019

Update signer_v4_test_data.json from google-cloud-dotnet

23e8f93

Remove trailing slash from path if no file. Update CGI.escape to Addressable::URI.escape. [refs googleapis#3081]

blowmage mentioned this pull request Apr 3, 2019

Can not obtain a signed url for a file with a space in its name #3064

Closed

yoshi-automation added the 🚨 This issue needs some love. label Apr 4, 2019

blowmage closed this Apr 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encode object name with URI instead of CGI #3081

Encode object name with URI instead of CGI #3081

prapicault commented Mar 28, 2019

googlebot commented Mar 28, 2019

quartzmo commented Mar 28, 2019

blowmage Mar 28, 2019

prapicault Mar 28, 2019

blowmage commented Mar 28, 2019

prapicault commented Mar 28, 2019 •

edited

Loading

prapicault commented Mar 28, 2019

frankyn commented Mar 28, 2019

quartzmo Mar 28, 2019

frankyn Mar 28, 2019

quartzmo commented Mar 29, 2019

blowmage commented Mar 29, 2019

quartzmo commented Mar 29, 2019

blowmage commented Mar 29, 2019

quartzmo commented Apr 2, 2019

quartzmo commented Apr 3, 2019

quartzmo commented Apr 4, 2019

quartzmo commented Apr 9, 2019

blowmage commented Apr 9, 2019

prapicault commented Apr 16, 2019

Encode object name with URI instead of CGI #3081

Encode object name with URI instead of CGI #3081

Conversation

prapicault commented Mar 28, 2019

googlebot commented Mar 28, 2019

What to do if you already signed the CLA

Individual signers

Corporate signers

quartzmo commented Mar 28, 2019

blowmage Mar 28, 2019

Choose a reason for hiding this comment

prapicault Mar 28, 2019

Choose a reason for hiding this comment

blowmage commented Mar 28, 2019

prapicault commented Mar 28, 2019 • edited Loading

prapicault commented Mar 28, 2019

frankyn commented Mar 28, 2019

quartzmo Mar 28, 2019

Choose a reason for hiding this comment

frankyn Mar 28, 2019

Choose a reason for hiding this comment

quartzmo commented Mar 29, 2019

blowmage commented Mar 29, 2019

quartzmo commented Mar 29, 2019

blowmage commented Mar 29, 2019

quartzmo commented Apr 2, 2019

quartzmo commented Apr 3, 2019

quartzmo commented Apr 4, 2019

quartzmo commented Apr 9, 2019

blowmage commented Apr 9, 2019

prapicault commented Apr 16, 2019

prapicault commented Mar 28, 2019 •

edited

Loading