Fix path parsing to use URL.EscapedPath() instead of Path and unescape individual components

[jessesuen]

This addresses issue #308.

When parsing the request URL, ServeHTTP() currently uses URL.Path instead of URL.EscapedPath(). This makes it impossible for grpc-gateway to support having slashes / in a path even if they are encoded (e.g. http://example.com/some%2Fpath/resource). This change will switch parsing to use URL.EscapedPath(), then unescape the individual components after they have been split by /.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

[jessesuen]

I signed it!

[googlebot]

CLAs look good, thanks!

[achew22]

This looks like a really interesting patch. Can you add some tests that demonstrate the new behavior?

[ivucica]

Possibly more importantly, can you demonstrate old broken behavior and how the new code fixes it?

[jessesuen]

I will most definitely add a test. I just wanted to solicit feedback on this to see if this would be something acceptable upstream before investing more time in this.

[jessesuen]

@achew22, @ivucica -- I added a unit test which can reproduce broken behavior (before my fix) and verifies the desired behavior (after my fix).

However, I am having trouble understanding the remaining node integration test failures:

1) ABitOfEverythingService Create should assign id
1.1) Failed: Object({ url: 'http://localhost:8080/v1/example/a_bit_of_everything/1.5/2.5/4294967296/separator/9223372036854775807/-2147483648/9223372036854775807/4294967295/true/strprefix%2Ffoo/4294967295/2147483647/-4611686018427387904/2147483647/4611686018427387903/camelCase', method: 'POST', headers: Object({ content-type: 'text/plain; charset=utf-8' }), errObj: Error: Not Found, status: 404, statusText: 'Not Found
', data: 'Not Found
' })
1.2) Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
2) ABitOfEverythingService Create should echo the request back
2.1) Failed: Object({ url: 'http://localhost:8080/v1/example/a_bit_of_everything/1.5/2.5/4294967296/separator/9223372036854775807/-2147483648/9223372036854775807/4294967295/true/strprefix%2Ffoo/4294967295/2147483647/-4611686018427387904/2147483647/4611686018427387903/camelCase', method: 'POST', headers: Object({ content-type: 'text/plain; charset=utf-8' }), errObj: Error: Not Found, status: 404, statusText: 'Not Found
', data: 'Not Found

It looks like this ABitOfEverything service has a {string_value=strprefix/*} as part of the path option. However, I don't understand why this wouldn't just be {string_value}.

	rpc Create(ABitOfEverything) returns (ABitOfEverything) {
		// TODO add enum_value
		option (google.api.http) = {
			post: "/v1/example/a_bit_of_everything/{float_value}/{double_value}/{int64_value}/separator/{uint64_value}/{int32_value}/{fixed64_value}/{fixed32_value}/{bool_value}/{string_value=strprefix/*}/{uint32_value}/{sfixed32_value}/{sfixed64_value}/{sint32_value}/{sint64_value}/{nonConventionalNameValue}"
		};
	}

It seems that the test case itself would need to change. Can someone explain the meaning behind {string_value=strprefix/*} ?

[achew22]

It seems that the test case itself would need to change. Can someone explain the meaning behind {string_value=strprefix/*} ?

@yugui, now that I look at that, do you know why that is like that? My guess is that it strips the "strprefix/" from the beginning of the string value, but I don't understand why you wouldn't just put that outside the capture. Do you have any hints?

[ivucica]

It seems that the test case itself would need to change. Can someone explain the meaning behind {string_value=strprefix/*} ?

@yugui, now that I look at that, do you know why that is like that? My guess is that it strips the "strprefix/" from the beginning of the string value, but I don't understand why you wouldn't just put that outside the capture. Do you have any hints?

I first thought this might be a case of https://tools.ietf.org/html/rfc6570 page 8 and page 21. My (untested) understanding was that passing strprefix/a,b,c might result in a list { "a", "b", "c" }.

However, I remembered seeing this syntax before in the wild. Here it is in pubsub v1's pubsub.proto.

I think this might mean that the value strprefix/ is included in the output string string_value, but that it is mandatory. If you would move this outside, strprefix/ would still be mandatory, but no longer included in string_value.

Here's the documentation for the linked RPC.

[ivucica]

I think that interpretation is correct. Cloud Endpoints's ESP:

• Variable = "{" FieldPath [ "=" Segments ] "}" ; and
• Segments = Segment { "/" Segment } ; and
• Segment = "*" | "**" | LITERAL | Variable ;

[achew22]

Reading through, I think I agree with your interpretation of the spec @ivucica.

@jessesuen, I think that should be sufficient information to update the tests to reflect the new behavior. Does that work for you?

[jessesuen]

Sorry, I was OOO the past week. If you mean moving strprefix/ outside the capture, I will be happy to make this change. I'll update the PR.

[jessesuen]

Alternatively, do you believe that strprefix has any value in the example? In other words, could we simplify a_bit_of_everything_service.spec.js from:

post: "..../{bool_value}/strprefix/{string_value}/{uint32_value}..."

into this:

post: "..../{bool_value}/{string_value}/{uint32_value}..."

[ivucica]

On Fri, Jun 15, 2018 at 3:43 PM Jesse Suen ***@***.***> wrote: Sorry, I was OOO the past week. If you mean moving strprefix/ outside the capture, I will be happy to make this change. I'll update the PR.

I don't think moving it out is meant to be equivalent. Leaving it in says "match this pattern if the template field's value starts with strprefix/". When the template field's value is placed into a variable, strprefix/ remains inside. That is: /abc/def/{ghi} is not equivalent to /abc/{ghi=def/*}. Given /abc/def/blah, the former will produce ghi=blah, and the latter will produce ghi=def/blah. I don't know how widely this is used in general, but as you can see, proto for Cloud Pubsub does use it and presumably depends on it. Without looking at the code (no time!), I will assume gRPC-Gateway interprets this pattern equivalent to Google Cloud Endpoints's REST frontend.

[johanbrandhorst]

Bump, what's left to do before we can merge this?

[ivucica]

I believe error from #660 (comment) needs to be fixed?

[johanbrandhorst]

Please rebase on master for new CI functionality.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kindermoumoute]

This MR is still relevant and it is actually fixing a bug.

@jessesuen any update on this?

[johanbrandhorst]

I'd be happy to review any new PRs to fix this.

[v3n]

@johanbrandhorst Would it be possible to get a new version cut that contains this fix?

[johanbrandhorst]

@v3n thanks for the prompting, I can cut v2.7.0 once #2418 has landed or within the next week, whichever comes soonest. Does that sound okay?

[johanbrandhorst]

@v3n I've just released v2.7.0