Add redirect support to SpdyRoundTripper #44451

ncdc · 2017-04-13T15:36:25Z

Add support for following redirects to the SpdyRoundTripper. This is
necessary for clients using it directly (e.g. the apiserver talking
directly to the kubelet) because the CRI streaming server issues a
redirect for streaming requests.

We need this in OpenShift because we have code that executes inside our apiserver that talks directly to the node to perform an attach request, and we need to be able to follow that redirect.

This code was adapted from the upgrade-aware proxy handler.

cc @smarterclayton @sttts @liggitt @timstclair @kubernetes/sig-api-machinery-pr-reviews

k8s-reviewable · 2017-04-13T15:37:36Z

This change is

ncdc · 2017-04-13T15:54:20Z

I seem to have broken TestServePortForward and am looking into it

liggitt · 2017-04-13T16:45:12Z

pkg/kubelet/server/server_test.go

@@ -1552,11 +1552,20 @@ func TestServePortForward(t *testing.T) {
 			url = fmt.Sprintf("%s/portForward/%s/%s", fw.testHTTPServer.URL, podNamespace, podName)
 		}

-		upgradeRoundTripper := spdy.NewRoundTripper(nil)
-		c := &http.Client{Transport: upgradeRoundTripper}
-		// Don't follow redirects, since we want to inspect the redirect response.


this case was odd... it seems like it was explicitly testing a scenario the spdy round tripper wouldn't handle, right?

Yes, but I think it was testing to make sure that the kubelet server could return a redirect. My change here made it start breaking, because the round tripper started following the redirect, so I updated this test case to match what's done in textExecAttach.

liggitt · 2017-04-13T16:59:08Z

staging/src/k8s.io/apimachinery/pkg/util/httpstream/spdy/roundtripper.go

+			method = "GET"
+		}
+
+		r, err := http.NewRequest(method, location.String(), req.Body)


remind me again why reusing the original req.Body doesn't do bad things if we hit this loop more than once?

The request body is never used for spdy requests, but otherwise, we'd probably need to be more careful. This appears to be what the proxy code is doing.

If the request body isn't used, can you just pass nil? That's what the proxy code does for subsequent requests (

kubernetes/staging/src/k8s.io/apiserver/pkg/registry/generic/rest/proxy.go

Line 276 in c3463d7

intermediateConn, err = h.connectBackend("GET", location, initialReq.Header, nil)

)

ncdc · 2017-04-13T17:08:34Z

@k8s-bot verify test this (some verify generated protobuf rsync issue)

ncdc · 2017-04-13T17:43:37Z

@k8s-bot cvm gce e2e test this

timstclair · 2017-04-13T18:09:09Z

staging/src/k8s.io/apimachinery/pkg/util/httpstream/spdy/roundtripper.go

+			method = "GET"
+		}
+
+		r, err := http.NewRequest(method, location.String(), req.Body)


If the request body isn't used, can you just pass nil? That's what the proxy code does for subsequent requests (

kubernetes/staging/src/k8s.io/apiserver/pkg/registry/generic/rest/proxy.go

Line 276 in c3463d7

intermediateConn, err = h.connectBackend("GET", location, initialReq.Header, nil)

)

timstclair · 2017-04-13T18:10:36Z

staging/src/k8s.io/apimachinery/pkg/util/httpstream/spdy/roundtripper.go


+	s.conn = intermediateConn
+	intermediateConn = nil // Don't close the connection when we return it.


fix comment (not returning intermediateConn)

timstclair · 2017-04-13T18:13:28Z

staging/src/k8s.io/apimachinery/pkg/util/httpstream/spdy/roundtripper.go

+	header.Add(httpstream.HeaderConnection, httpstream.HeaderUpgrade)
+	header.Add(httpstream.HeaderUpgrade, HeaderSpdy31)
+
+redirectLoop:


This duplicates a lot of code from the proxy (

kubernetes/staging/src/k8s.io/apiserver/pkg/registry/generic/rest/proxy.go

Line 265 in c3463d7

redirectLoop:

). Is it possible to extract a shared method? Maybe something that takes a connectBackend function?

I will look and see what I can do.

ncdc · 2017-04-17T17:43:29Z

@timstclair @liggitt updated to extract the common logic shared between spdy and the upgrade aware proxy. PTAL. Also, naming is hard, so I'm open to any suggestions for what to call the new interface and method.

sttts · 2017-04-18T08:23:36Z

staging/src/k8s.io/apimachinery/pkg/util/httpstream/spdy/roundtripper.go

 func (s *SpdyRoundTripper) TLSClientConfig() *tls.Config {
 	return s.tlsConfig
 }

+// SendRequest implements k8s.io/apimachinery/pkg/util/net.RequestSender.


better use an anonymous var with type cast instead of a comment.

sttts · 2017-04-18T08:29:11Z

staging/src/k8s.io/apimachinery/pkg/util/net/http.go

+		intermediateConn net.Conn
+		rawResponse      = bytes.NewBuffer(make([]byte, 0, 256))
+		body             = originalBody
+		err              error


err can be local in the loop body.

No, because that would result in intermediateConn being shadowed.

sttts · 2017-04-18T08:31:13Z

Some nits. Otherwise, looks good now that the redirect loop is factored out.

ncdc · 2017-04-18T15:03:26Z

Updated and squashed the code review commits

ncdc · 2017-04-18T15:56:24Z

@k8s-bot gci gce e2e test this (one node didn't come up for some reason).

time="2017-04-18T15:22:47.254030486Z" level=info msg="API listen on /var/run/docker.sock"
time="2017-04-18T15:22:47.280588165Z" level=debug msg="Calling POST /v1.23/images/load?quiet=1"
+ [ ! -s /var/lib/docker/repositories-overlay ]
+ rm -f /var/lib/docker/repositories-overlay
time="2017-04-18T15:23:16.566334781Z" level=info msg="New containerd process, pid: 778\n"
time="2017-04-18T15:23:18.037310875Z" level=error msg="invalid image sha256:0de6cfa909121ffb45151d67e23dd6d769ad481d5f4d7a172725996718524b1e, failed to verify image: sha256:0de6cfa909121ffb45151d67e23dd6d769ad481d5f4d7a172725996718524b1e"
time="2017-04-18T15:23:18.039494774Z" level=fatal msg="Error starting daemon: layer does not exist"

liggitt · 2017-04-18T17:13:18Z

staging/src/k8s.io/apimachinery/pkg/util/net/http.go

+			return nil, nil, fmt.Errorf("too many redirects (%d)", redirects)
+		}
+		if redirects != 0 {
+			// Redirected requests switch to "GET" according to the HTTP spec:


nit: if you want, you can move these down to the Prepare to follow the redirect. section and make unconditional

Good point... Since you've asked for a comment for 'future us' below, I'll make the change.

liggitt · 2017-04-18T17:14:43Z

staging/src/k8s.io/apimachinery/pkg/util/net/http.go

+		if redirectStr == "" {
+			return nil, nil, fmt.Errorf("%d response missing Location header", resp.StatusCode)
+		}
+		location, err = location.Parse(redirectStr)


comment about parsing relative to the last requested location would be helpful here for future us

liggitt · 2017-04-18T17:19:21Z

lgtm

ncdc · 2017-04-18T19:07:02Z

@timstclair @liggitt @sttts ready for final review. Any other comments?

liggitt · 2017-04-18T19:29:07Z

none from me

ncdc · 2017-04-20T18:47:15Z

Just pushed up the new changes. PTAL. I'll say it's a bit weird to have pkg/client/unversioned/remotecommand/remotecommand.go looking at the feature gate...

timstclair · 2017-04-20T18:54:44Z

/lgtm

sttts · 2017-04-21T11:31:18Z

pkg/client/unversioned/remotecommand/remotecommand.go

@@ -78,7 +80,8 @@ func NewExecutor(config *restclient.Config, method string, url *url.URL) (Stream
 		return nil, err
 	}

-	upgradeRoundTripper := spdy.NewRoundTripper(tlsConfig)
+	followRedirects := utilfeature.DefaultFeatureGate.Enabled(genericfeatures.StreamingProxyRedirects)


Having feature gates in client code is awkward. Any reason to have this feature gate at all here? Why don't we enable redirection for remotecommand? If the kubelet still depends on the gate, that's fine and another topic.

This isn't actually used in client code, is it? Maybe I'm confused about how this is used.

I'd be happy to remove it. Do we all agree on that?

FYI here is where the apiserver in OpenShift is creating a spdy roundtripper and executor: https://github.com/openshift/origin/blob/aaf06ed2b62b0e06845ad804932cddeec36b686e/pkg/build/registry/buildconfiginstantiate/rest.go#L237-L238.

ncdc · 2017-04-25T16:56:19Z

Rebased and squashed. PTAL.

/release-note

smarterclayton · 2017-04-25T18:20:34Z

I'm ok without the gate.

…

On Tue, Apr 25, 2017 at 1:00 PM, Andy Goldstein ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In pkg/client/unversioned/remotecommand/remotecommand.go <#44451 (comment)> : > @@ -78,7 +80,8 @@ func NewExecutor(config *restclient.Config, method string, url *url.URL) (Stream return nil, err } - upgradeRoundTripper := spdy.NewRoundTripper(tlsConfig) + followRedirects := utilfeature.DefaultFeatureGate.Enabled(genericfeatures.StreamingProxyRedirects) FYI here is where the apiserver in OpenShift is creating a spdy roundtripper and executor: https://github.com/openshift/origin/blob/ aaf06ed2b62b0e06845ad804932cddeec36b686e/pkg/build/registry/ buildconfiginstantiate/rest.go#L237-L238. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44451 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_pyOs0YyefmsykTzsK_JsFKGdOG8yks5rziaugaJpZM4M83Ao> .

timstclair · 2017-04-25T23:14:47Z

I don't want to keep blocking this - I'm ok with or without the gate.

ncdc · 2017-04-25T23:39:15Z

I'll take it out later tonight when I'm back at a computer.

…

On Tue, Apr 25, 2017 at 4:15 PM Tim St. Clair ***@***.***> wrote: I don't want to keep blocking this - I'm ok with or without the gate. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44451 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAABYq5gzgHhKu4xEhqxBgZx14oFEApRks5rzn6MgaJpZM4M83Ao> .

Add support for following redirects to the SpdyRoundTripper. This is necessary for clients using it directly (e.g. the apiserver talking directly to the kubelet) because the CRI streaming server issues a redirect for streaming requests. Also extract common logic for following redirects.

ncdc · 2017-04-26T13:47:32Z

Ok, removed the flag from the client code. Hopefully this is the last revision :-)

ncdc · 2017-04-26T15:21:53Z

@timstclair @sttts @smarterclayton @liggitt I think this should be good to go, if you're all ok with it

smarterclayton · 2017-04-26T20:51:13Z

/lgtm

k8s-github-robot · 2017-04-26T20:51:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ncdc, smarterclayton, timstclair

Needs approval from an approver in each of these OWNERS Files:

~~pkg/client/OWNERS~~ [smarterclayton]
~~pkg/kubelet/OWNERS~~ [smarterclayton,timstclair]
~~staging/src/k8s.io/apimachinery/pkg/OWNERS~~ [smarterclayton]
~~staging/src/k8s.io/apiserver/OWNERS~~ [smarterclayton]
~~staging/src/k8s.io/client-go/OWNERS~~ [smarterclayton]
~~staging/src/k8s.io/kube-aggregator/OWNERS~~ [smarterclayton]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-04-26T21:47:41Z

Automatic merge from submit-queue

ncdc added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Apr 13, 2017

ncdc assigned sttts, liggitt and timstclair Apr 13, 2017

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 13, 2017

k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note-label-needed labels Apr 13, 2017

ncdc force-pushed the spdy-follow-redirects branch 2 times, most recently from d157bb4 to 23d8754 Compare April 13, 2017 16:35

liggitt reviewed Apr 13, 2017

View reviewed changes

timstclair reviewed Apr 13, 2017

View reviewed changes

ncdc force-pushed the spdy-follow-redirects branch from 23d8754 to 8291d8a Compare April 17, 2017 17:42

sttts reviewed Apr 18, 2017

View reviewed changes

ncdc force-pushed the spdy-follow-redirects branch from 8291d8a to ecf3618 Compare April 18, 2017 15:02

liggitt reviewed Apr 18, 2017

View reviewed changes

ncdc force-pushed the spdy-follow-redirects branch from ecf3618 to 038c7ce Compare April 18, 2017 17:56

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 20, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 20, 2017

sttts reviewed Apr 21, 2017

View reviewed changes

ncdc force-pushed the spdy-follow-redirects branch from fe88f42 to 299e9dd Compare April 25, 2017 16:55

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 25, 2017

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-label-needed labels Apr 25, 2017

ncdc added 2 commits April 26, 2017 09:45

Update bazel

fc2128c

ncdc force-pushed the spdy-follow-redirects branch from 299e9dd to fc2128c Compare April 26, 2017 13:46

ncdc removed the do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. label Apr 26, 2017

k8s-ci-robot assigned smarterclayton Apr 26, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 26, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 26, 2017

k8s-github-robot merged commit 274df99 into kubernetes:master Apr 26, 2017

ncdc deleted the spdy-follow-redirects branch April 26, 2017 23:45

ncdc mentioned this pull request Apr 28, 2017

prevent corrupted spdy stream after hijacking connection #43922

Merged

timstclair mentioned this pull request May 19, 2017

Move StreamingProxyRedirects to GA #46066

Closed

pacoxu mentioned this pull request Mar 16, 2022

Don't follow redirects with spdy #108531

Merged


		s.conn = intermediateConn
		intermediateConn = nil // Don't close the connection when we return it.

Add redirect support to SpdyRoundTripper #44451

Add redirect support to SpdyRoundTripper #44451

Conversation

ncdc commented Apr 13, 2017

k8s-reviewable commented Apr 13, 2017

ncdc commented Apr 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncdc commented Apr 13, 2017

ncdc commented Apr 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncdc commented Apr 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts commented Apr 18, 2017

ncdc commented Apr 18, 2017

ncdc commented Apr 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented Apr 18, 2017

ncdc commented Apr 18, 2017

liggitt commented Apr 18, 2017

ncdc commented Apr 20, 2017

timstclair commented Apr 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncdc commented Apr 25, 2017

smarterclayton commented Apr 25, 2017 via email

timstclair commented Apr 25, 2017

ncdc commented Apr 25, 2017 via email

ncdc commented Apr 26, 2017

ncdc commented Apr 26, 2017

smarterclayton commented Apr 26, 2017

k8s-github-robot commented Apr 26, 2017

k8s-github-robot commented Apr 26, 2017

ncdc commented Apr 18, 2017 •

edited

Loading