Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Client: Some HTTP/2 requests are never sent #11965

Closed
mperktold opened this issue Jun 26, 2024 · 4 comments · Fixed by #11999
Closed

Client: Some HTTP/2 requests are never sent #11965

mperktold opened this issue Jun 26, 2024 · 4 comments · Fixed by #11999
Assignees
Labels
Bug For general bugs on Jetty side

Comments

@mperktold
Copy link

Jetty version(s)
Jetty 12.0.10

Jetty Environment
core

Java version/vendor (use: java -version)
openjdk version "21.0.3" 2024-04-16 LTS
OpenJDK Runtime Environment Temurin-21.0.3+9 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (build 21.0.3+9-LTS, mixed mode, sharing)

OS type/version
Windows 11

Description
When using the Jetty 12 HttpClient to send a request to certain servers via HTTP/2, the request never completes. In fact, the request is only queued, but sending the request never even begins.

This only happens with some servers, which do support HTTP/2.

How to reproduce?

var connector = new ClientConnector();
connector.setSslContextFactory(new SslContextFactory.Client());
var httpClient = new HttpClient(new HttpClientTransportDynamic(connector,
    HttpClientConnectionFactory.HTTP11,
    new ClientConnectionFactoryOverHTTP2.HTTP2(new HTTP2Client(connector))
));
httpClient.start();
ContentResponse response = httpClient.newRequest("https://microsoft.sharepoint.com")
    .onRequestQueued(r -> System.out.println("request queued"))
    .onRequestBegin(r -> System.out.println("request begin"))
    .onRequestContent((r, c) -> System.out.println("request content"))
    .onRequestSuccess(r -> System.out.println("request success"))
    .onResponseBegin(r -> System.out.println("response begin"))
    .onResponseContent((r, c) -> System.out.println("response content"))
    .onResponseSuccess(r -> System.out.println("response success"))
    .send();
System.out.println(response);
httpClient.stop();

The above snippet creates a HttpClient that supports both HTTP/1.1 and HTTP2, and makes a request to "https://microsoft.sharepoint.com". Several events are logged, so we see that the request is only ever queued, but never sent.

When forcing the client to use HTTP/1.1, either by removing support for HTTP/2, or by using version(HttpVersion.HTTP_1_1), everything works as expected.

Also, the following curl command sends the same request, also using HTTP/2, and works fine:
curl "https://microsoft.sharepoint.com" --http2

@mperktold mperktold added the Bug For general bugs on Jetty side label Jun 26, 2024
@lorban
Copy link
Contributor

lorban commented Jul 2, 2024

I created the following repository to try to reproduce this problem: https://github.com/lorban/h2-sharepoint-11965

and I get the following output which shows everything is working as expected when I run H2SharepointTest which contains your reproducer:

request queued
request begin
request success
response begin
response content
response success
HttpContentResponse[HTTP/2.0 403 null - 13 bytes]

Did I miss something?

@lorban lorban assigned lorban and unassigned sbordet Jul 2, 2024
@mperktold
Copy link
Author

You're right, now it also works for me.
Wow that's weird, I'm pretty sure it didn't work when I posted this. Maybe Microsoft changed something in their server.

Anyway, I found another server that does show the reported behavior: https://cashdesk-hochschwarzwald.stage.peaksolution.com
Can you check with this one?


On a related note, I now also tried to replicate this with the original sharepoint URL by using Conscrypt, changing the first few lines like this:

Security.addProvider(new OpenSSLProvider());
var sslContextFactory = new SslContextFactory.Client();
sslContextFactory.setProvider("Conscrypt");
var connector = new ClientConnector();
connector.setSslContextFactory(sslContextFactory);
...

But then I get this exception:

Exception in thread "main" java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: Unknown authType: GENERIC
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
	at org.eclipse.jetty.client.transport.HttpRequest.send(HttpRequest.java:707)
	at JettyIssue.main(JettyIssue.java:51)
Caused by: javax.net.ssl.SSLHandshakeException: Unknown authType: GENERIC
	at org.conscrypt.SSLUtils.toSSLHandshakeException(SSLUtils.java:361)
	at org.conscrypt.ConscryptEngine.convertException(ConscryptEngine.java:1138)
	at org.conscrypt.ConscryptEngine.readPlaintextData(ConscryptEngine.java:1093)
	at org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:880)
	at org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:751)
	at org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:716)
	at org.conscrypt.Java8EngineWrapper.unwrap(Java8EngineWrapper.java:236)
	at org.eclipse.jetty.io.ssl.SslConnection.unwrap(SslConnection.java:409)
	at org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.fill(SslConnection.java:742)
	at org.eclipse.jetty.io.NegotiatingClientConnection.fill(NegotiatingClientConnection.java:102)
	at org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:84)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.onFillable(SslConnection.java:574)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:390)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:150)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	Suppressed: java.io.IOException: Broken pipe
		at org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.flush(SslConnection.java:1155)
		at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
		at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275)
		at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254)
		at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:368)
		at org.eclipse.jetty.client.transport.internal.HttpSenderOverHTTP$HeadersCallback.process(HttpSenderOverHTTP.java:206)
		at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:250)
		at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:231)
		at org.eclipse.jetty.client.transport.internal.HttpSenderOverHTTP.sendHeaders(HttpSenderOverHTTP.java:78)
		at org.eclipse.jetty.client.transport.HttpSender$ContentSender.process(HttpSender.java:541)
		at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:250)
		at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:231)
		at org.eclipse.jetty.client.transport.HttpSender.send(HttpSender.java:85)
		at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.send(HttpChannelOverHTTP.java:86)
		at org.eclipse.jetty.client.transport.HttpChannel.send(HttpChannel.java:142)
		at org.eclipse.jetty.client.transport.HttpConnection.send(HttpConnection.java:112)
		at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP$Delegate.send(HttpConnectionOverHTTP.java:330)
		at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.send(HttpConnectionOverHTTP.java:159)
		at org.eclipse.jetty.client.transport.HttpDestination.send(HttpDestination.java:444)
		at org.eclipse.jetty.client.transport.HttpDestination.process(HttpDestination.java:420)
		at org.eclipse.jetty.client.transport.HttpDestination.process(HttpDestination.java:375)
		at org.eclipse.jetty.client.transport.HttpDestination.send(HttpDestination.java:358)
		at org.eclipse.jetty.client.transport.HttpDestination.succeeded(HttpDestination.java:291)
		at org.eclipse.jetty.client.AbstractConnectionPool.proceed(AbstractConnectionPool.java:315)
		at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.succeeded(AbstractConnectionPool.java:615)
		at org.eclipse.jetty.client.AbstractConnectionPool$FutureConnection.succeeded(AbstractConnectionPool.java:593)
		at org.eclipse.jetty.util.Promise$Wrapper.succeeded(Promise.java:195)
		at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onOpen(HttpConnectionOverHTTP.java:167)
		at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:435)
		at org.eclipse.jetty.io.NegotiatingClientConnection.replaceConnection(NegotiatingClientConnection.java:117)
		at org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:87)
		... 15 more
Caused by: java.security.cert.CertificateException: Unknown authType: GENERIC
	at java.base/sun.security.validator.EndEntityChecker.checkTLSServer(EndEntityChecker.java:290)
	at java.base/sun.security.validator.EndEntityChecker.check(EndEntityChecker.java:149)
	at java.base/sun.security.validator.Validator.validate(Validator.java:269)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
	at org.conscrypt.Platform.checkServerTrusted(Platform.java:330)
	at org.conscrypt.ConscryptEngine.verifyCertificateChain(ConscryptEngine.java:1643)
	at org.conscrypt.NativeCrypto.ENGINE_SSL_read_direct(Native Method)
	at org.conscrypt.NativeSsl.readDirectByteBuffer(NativeSsl.java:567)
	at org.conscrypt.ConscryptEngine.readPlaintextDataDirect(ConscryptEngine.java:1099)
	at org.conscrypt.ConscryptEngine.readPlaintextData(ConscryptEngine.java:1083)
	... 23 more

The exception appears after "request begin" is logged. After that, nothing happens and the client seems to be waiting forever. Do I need to change some security configuration, or does the server use outdated ones?

@joakime
Copy link
Contributor

joakime commented Jul 2, 2024

The conscrypt folks had a discussion about Unknown authType: GENERIC

google/conscrypt#1033

Interesting to read the other issues that reference that above issue.

Google cloud, palantir, grpc, and more all have encountered this issue.
Several of them report that using the Conscrypt specific TrustStore (not the JDK one) seems to help them.

lorban added a commit that referenced this issue Jul 3, 2024
…eption in SslEndPoint.fill()

Signed-off-by: Ludovic Orban <lorban@bitronix.be>
lorban added a commit that referenced this issue Jul 3, 2024
… fill() is called with an empty buffer

Signed-off-by: Ludovic Orban <lorban@bitronix.be>
@lorban
Copy link
Contributor

lorban commented Jul 3, 2024

I could reproduce the problem with the https://cashdesk-hochschwarzwald.stage.peaksolution.com/ URL, and it is caused by a genuine bug in the Jetty client's TLS negotiating code.

This PR contains a fix, as well as some related improvements: #11999

Thanks for the report!

lorban added a commit that referenced this issue Jul 4, 2024
Signed-off-by: Ludovic Orban <lorban@bitronix.be>
lorban added a commit that referenced this issue Jul 4, 2024
Signed-off-by: Ludovic Orban <lorban@bitronix.be>
lorban added a commit that referenced this issue Jul 10, 2024
#11965 make EMPTY_BUFFER not read-only anymore

Signed-off-by: Ludovic Orban <lorban@bitronix.be>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For general bugs on Jetty side
Projects
No open projects
Status: ✅ Done
Development

Successfully merging a pull request may close this issue.

4 participants