java.util.concurrent.ExecutionException: org.eclipse.jetty.websocket.api.UpgradeException: Failed to upgrade to websocket: Unexpected HTTP Response Status Code: 400 Bad Request

[Haimonti]

Hi,
I am trying to setup a websocket server on the Google AppEngine (GAE) Flexible environment (WebSocket support was recently added to GAE). I keep getting the following error:

java.util.concurrent.ExecutionException: org.eclipse.jetty.websocket.api.UpgradeException: Failed to upgrade to websocket: Unexpected HTTP Response Status Code: 400 Bad Request
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
	at interContILP.SendServlet.sendMessageOverWebSocket(SendServlet.java:150)
	at interContILP.SendServlet.doPost(SendServlet.java:90)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
	at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at com.google.apphosting.runtime.jetty9.ParseBlobUploadHandler.handle(ParseBlobUploadHandler.java:119)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1182)
	at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doHandle(AppEngineWebAppContext.java:187)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:293)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:539)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
	at com.google.apphosting.runtime.jetty9.RpcConnection.handle(RpcConnection.java:213)
	at com.google.apphosting.runtime.jetty9.RpcConnector.serviceRequest(RpcConnector.java:81)
	at com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:722)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:685)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:655)
	at com.google.apphosting.runtime.JavaRuntime$NullSandboxRequestRunnable.run(JavaRuntime.java:847)
	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:270)
	at java.lang.Thread.run(Thread.java:748) 
Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Failed to upgrade to websocket: Unexpected HTTP Response Status Code: 400 Bad Request
	at org.eclipse.jetty.websocket.client.WebSocketUpgradeRequest.onComplete(WebSocketUpgradeRequest.java:527)
	at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:196)
	at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:188)
	at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:441)
	at org.eclipse.jetty.client.HttpReceiver.responseSuccess(HttpReceiver.java:387)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.messageComplete(HttpReceiverOverHTTP.java:316)
	at org.eclipse.jetty.http.HttpParser.handleContentMessage(HttpParser.java:599)
	at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1669)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1517)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:172)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:135)
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73)
	at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133)
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:155)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) ... 1 more

I used a Browser WebSocket Client on Chrome and it does not go through either. The Http Header on the request looks like this:

Request header
:authority: steel-earth-236015.appspot.com
:method: POST
:path: /send
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;
q=0.9,image/webp,image/apng,*/*;q=0.8,
application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: max-age=0
content-length: 16
content-type: application/x-www-form-urlencoded
dnt: 1
origin: https://steel-earth-236015.appspot.com
referer: https://steel-earth-236015.appspot.com/index.jsp
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36

Suggestions to debug this are highly appreciated!

[joakime]

Request header
:authority: steel-earth-236015.appspot.com
:method: POST
:path: /send
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;
q=0.9,image/webp,image/apng,/;q=0.8,
application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: max-age=0
content-length: 16
content-type: application/x-www-form-urlencoded
dnt: 1
origin: https://steel-earth-236015.appspot.com
referer: https://steel-earth-236015.appspot.com/index.jsp
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36

A few things come to mind.

1. This looks like an HTTP/2 request. Upgrade to WebSocket over HTTP/2 is not supported. Only (RFC6455) Upgrade to WebSocket over HTTP/1.1 is supported.
2. A WebSocket upgrade is a GET request (not POST) with appropriate headers for WebSocket handshake, and HTTP connection upgrade.
3. Upgrade to WebSocket is not handled by a Servlet. (noting that your interContILP.SendServlet.sendMessageOverWebSocket indicator)

[Haimonti]

Thank you very much for your response. This is very helpful. I am actually trying to execute the code presented here -- https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/appengine-java8/websocket-jetty as an example with small changes (addition of a different package). It seems that the upgrade does indeed occur in a doPost method! I am pretty new to WebSocket Servlet programming and suggestions are very much appreciated!

[joakime]

Wow, that's an overly complicated example of how to use Jetty's WebSocketClient.
I need to submit a PR to fix that.

[Haimonti]

Agreed about the complexity of the code. After reading it several times and building it on the Google App Engine Flex environment, I am able to load the index.jsp but on clicking Send, I get the error message posted above. The SendServlet.java has the WebSocketClient and the following code:
ClientUpgradeRequest request = new ClientUpgradeRequest();
// Attempt connection
Future future = webSocketClient.connect(clientSocket,
new URI(getWebSocketAddress()), request);
// Wait for Connect
Session session = future.get();
// Send a message
session.getRemote().sendString(message);
// Close session
session.close();

And of course the webSocketClient.connect does not go through.
So how should the code be changed so that we have the HTTP/1.1 upgrade request?

[sbordet]

FTR, bootstrapping WebSocket over HTTP/2 will be supported in Jetty 10 via #3537.

[Haimonti]

I am using jetty version 9.4.18.v20190429. This means the request should still be using HTTP/1.1 I guess. But I do not know how to fix the code to reflect the correct protocol. Suggestions please.

[Haimonti]

Wow, that's an overly complicated example of how to use Jetty's WebSocketClient.
I need to submit a PR to fix that.

Joakime, Are you suggesting the code as it stands is unlikely to work? Thanks much for the help with this. It is very much appreciated.

[sbordet]

Jetty's 9.4.x WebSocketClient will only produce HTTP/1.1 upgrades to WebSocket.

Why you end up with a HTTP/2 request? Perhaps GAE converts the HTTP/1.1 request to HTTP/2? If that's the case, can you disable that conversion if it's an upgrade?

[joakime]

I opened GoogleCloudPlatform/java-docs-samples#1526 for updates to the Jetty WebSocketClient usage.

[Haimonti]

I opened GoogleCloudPlatform/java-docs-samples#1526 for updates to the Jetty WebSocketClient usage.

Thank you very much for updating the code. I very much appreciate the help. It appears the error has been resolved and I do not see the exception from before.

But now I am unable to get the client server communication to work i.e. on /send from index.jsp I simply get a blank page back. When testing /js_client.jsp, I do not get any output. I will test more and see what the problem may be. Thanks much!

[Haimonti]

Hi,
I have been able to get the example to work now. I still do have a few questions about the HttpClient. Once the client upgrade request goes through and a session is created, I would like to do two things in my code:
a. To upload a file from the client to the server. I am looking at the documentation (https://www.eclipse.org/jetty/documentation/9.4.x/http-client-api.html) and see that an easy way is to use the following code:

ContentResponse response = httpClient.newRequest("http://domain.com/upload")
        .method(HttpMethod.POST)
        .content(new PathContentProvider(Paths.get("file_to_upload.txt")), "text/plain")
        .send();

How is the "file_to_upload.txt" path specified? There seems to be no discussion about that -- Any suggestions?

b. Once the new session is created, on the Google App Engine Flex Environment, a new process is started for the application especially the client. How does one get to know about this new process that is spawned? I want to see System.out.println messages for the HttpClient -- Is there a way to do this?

[sbordet]

@Haimonti, file_to_upload.txt should be the path of the file to upload on the file system, e.g. /home/ubuntu/file_to_upload.txt.

I don't know much about GAE Flex - @joakime perhaps?

[Haimonti]

@sbordet, Thanks for the clarification. In my code, I am guessing that the client (HttpClient/WebSocketClient) is the laptop (which opens a browser to test the index.jsp on the server) trying to connect to Google App Engine server. The code is however compiled using a Google Cloud Shell. Should the file_to_upload be a path on the laptop or a directory on the Google Cloud Shell? If it is the laptop, is it the case that once the connection is established, any path on the laptop is available to the system for upload/download of files? Or should a jetty server be running on the laptop and a /tmp be made available from its webapp-root folder?

@joakime: It seems with the new code the exception has disappeared. However, I am still not able to see the session.getRemote.sendString(message) actually sent between the client and server. Any suggestions?

Many thanks for all the help!

[joakime]

Closing.