Fixes #6514 - How to warm up SslConnection.

Implemented "priming" of HTTP/1.1 connections using ConnectionPool.preCreateConnections(int) and HttpClientTransportOverHTTP.setInitializeConnections(true).

This sends `OPTIONS * HTTP/1.1` to the server.

I tried to implement this feature by forcing a write of 0 bytes from the layer above `SslConnection`, but it did not work when using TLS because in both WriteFlusher and SslConnection the fact that there are 0 bytes left to write is treated specially.

Other HTTP versions have no problems because they must initialize the connection by e.g. sending a SETTINGS frame, so they would also initialize TLS.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

documentation/jetty/modules/code/examples/src/main/java/org/eclipse/jetty/docs/programming/client/http/HTTPClientDocs.java

@@ -1049,6 +1049,31 @@ public void setConnectionPool() throws Exception
         // end::setConnectionPool[]
     }
 
+    public void preCreateConnections() throws Exception
+    {
+        // tag::preCreateConnections[]
+        HttpClient httpClient = new HttpClient();
+        httpClient.start();
+
+        // For HTTP/1.1, you need to explicitly configure to initialize connections.
+        if (httpClient.getTransport() instanceof HttpClientTransportOverHTTP http1)
+            http1.setInitializeConnections(true);
+
+        // Create a dummy request to the server you want to pre-create connections to.
+        Request request = httpClient.newRequest("https://host/");
+
+        // Resolve the destination for that request.
+        Destination destination = httpClient.resolveDestination(request);
+
+        // Pre-create, for example, half of the connections.
+        int preCreate = httpClient.getMaxConnectionsPerDestination() / 2;
+        CompletableFuture<Void> completable = destination.getConnectionPool().preCreateConnections(preCreate);
+
+        // Wait for the connections to be created.
+        completable.get(5, TimeUnit.SECONDS);
+        // end::preCreateConnections[]
+    }
+
     public void unixDomain() throws Exception
     {
         // tag::unixDomain[]

documentation/jetty/modules/programming-guide/pages/client/http.adoc

@@ -158,7 +158,7 @@ Jetty's client library provides the following `ConnectionPool` implementations:
 * `DuplexConnectionPool`, historically the first implementation, only used by the HTTP/1.1 transport.
 * `MultiplexConnectionPool`, the generic implementation valid for any transport where connections are reused with a most recently used algorithm (that is, the connections most recently returned to the connection pool are the more likely to be used again).
 * `RoundRobinConnectionPool`, similar to `MultiplexConnectionPool` but where connections are reused with a round-robin algorithm.
-* `RandomRobinConnectionPool`, similar to `MultiplexConnectionPool` but where connections are reused with an algorithm that chooses them randomly.
+* `RandomConnectionPool`, similar to `MultiplexConnectionPool` but where connections are reused with an algorithm that chooses them randomly.
 
 The `ConnectionPool` implementation can be customized for each destination in by setting a `ConnectionPool.Factory` on the `HttpClientTransport`:
 
@@ -167,6 +167,34 @@ The `ConnectionPool` implementation can be customized for each destination in by
 include::code:example$src/main/java/org/eclipse/jetty/docs/programming/client/http/HTTPClientDocs.java[tags=setConnectionPool]
 ----
 
+[[connection-pool-precreate-connections]]
+=== Pre-Creating Connections
+
+`ConnectionPool` offers the ability to pre-create connections by calling `ConnectionPool.preCreateConnections(int)`.
+
+Pre-creating the connections saves the time and processing spent to establish the TCP connection, performing the TLS handshake (if necessary) and, for HTTP/2 and HTTP/3, perform the initial protocol setup.
+This is particularly important for HTTP/2 because in the initial protocol setup the server informs the client of the maximum number of concurrent requests per connection (otherwise assumed to be just `1` by the client).
+
+The scenarios where pre-creating connections is useful are, for example:
+
+* Load testing, where you want to prepare the system with connections already created to avoid paying of cost of connection setup.
+* Proxying scenarios, often in conjunction with the use of `RoundRobinConnectionPool` or `RandomConnectionPool`, where the proxy creates early the connections to the backend servers.
+
+This is an example of how to pre-create connections:
+
+[,java,indent=0]
+----
+include::code:example$src/main/java/org/eclipse/jetty/docs/programming/client/http/HTTPClientDocs.java[tags=preCreateConnections]
+----
+
+[NOTE]
+====
+Pre-creating connections for secure HTTP/1.1 requires you to call `HttpClientTransportOverHTTP.setInitializeConnections(true)`, otherwise only the TCP connection is established, but the TLS handshake is not initiated.
+
+To initialize connections for secure HTTP/1.1, the client sends an initial `OPTIONS * HTTP/1.1` request to the server.
+The server must be able to handle this request without closing the connection (in particular it must not add the `Connection: close` header in the response).
+====
+
 [[request-processing]]
 == HttpClient Request Processing
 

jetty-core/jetty-client/src/main/java/org/eclipse/jetty/client/transport/HttpClientConnectionFactory.java

@@ -28,10 +28,23 @@ public class HttpClientConnectionFactory implements ClientConnectionFactory
      */
     public static final Info HTTP11 = new HTTP11(new HttpClientConnectionFactory());
 
+    private boolean initializeConnections;
+
+    public boolean isInitializeConnections()
+    {
+        return initializeConnections;
+    }
+
+    public void setInitializeConnections(boolean initialize)
+    {
+        this.initializeConnections = initialize;
+    }
+
     @Override
     public org.eclipse.jetty.io.Connection newConnection(EndPoint endPoint, Map<String, Object> context)
     {
         HttpConnectionOverHTTP connection = new HttpConnectionOverHTTP(endPoint, context);
+        connection.setInitialize(isInitializeConnections());
         return customize(connection, context);
     }
 

jetty-core/jetty-client/src/main/java/org/eclipse/jetty/client/transport/HttpClientTransportOverHTTP.java

@@ -22,7 +22,6 @@
 import org.eclipse.jetty.client.DuplexConnectionPool;
 import org.eclipse.jetty.client.Origin;
 import org.eclipse.jetty.client.Request;
-import org.eclipse.jetty.io.ClientConnectionFactory;
 import org.eclipse.jetty.io.ClientConnector;
 import org.eclipse.jetty.io.EndPoint;
 import org.eclipse.jetty.util.ProcessorUtils;
@@ -37,7 +36,7 @@ public class HttpClientTransportOverHTTP extends AbstractConnectorHttpClientTran
     public static final Origin.Protocol HTTP11 = new Origin.Protocol(List.of("http/1.1"), false);
     private static final Logger LOG = LoggerFactory.getLogger(HttpClientTransportOverHTTP.class);
 
-    private final ClientConnectionFactory factory = new HttpClientConnectionFactory();
+    private final HttpClientConnectionFactory factory = new HttpClientConnectionFactory();
     private int headerCacheSize = 1024;
     private boolean headerCacheCaseSensitive;
 
@@ -100,4 +99,14 @@ public void setHeaderCacheCaseSensitive(boolean headerCacheCaseSensitive)
     {
         this.headerCacheCaseSensitive = headerCacheCaseSensitive;
     }
+
+    public boolean isInitializeConnections()
+    {
+        return factory.isInitializeConnections();
+    }
+
+    public void setInitializeConnections(boolean initialize)
+    {
+        factory.setInitializeConnections(initialize);
+    }
 }

jetty-core/jetty-client/src/main/java/org/eclipse/jetty/client/transport/internal/HttpConnectionOverHTTP.java

@@ -26,6 +26,7 @@
 import java.util.concurrent.atomic.LongAdder;
 
 import org.eclipse.jetty.client.Connection;
+import org.eclipse.jetty.client.Destination;
 import org.eclipse.jetty.client.HttpClientTransport;
 import org.eclipse.jetty.client.HttpUpgrader;
 import org.eclipse.jetty.client.Request;
@@ -40,6 +41,7 @@
 import org.eclipse.jetty.client.transport.IConnection;
 import org.eclipse.jetty.client.transport.SendFailure;
 import org.eclipse.jetty.http.HttpHeader;
+import org.eclipse.jetty.http.HttpMethod;
 import org.eclipse.jetty.http.HttpVersion;
 import org.eclipse.jetty.io.AbstractConnection;
 import org.eclipse.jetty.io.EndPoint;
@@ -61,6 +63,7 @@ public class HttpConnectionOverHTTP extends AbstractConnection implements IConne
     private final LongAdder bytesIn = new LongAdder();
     private final LongAdder bytesOut = new LongAdder();
     private long idleTimeout;
+    private boolean initialize;
 
     public HttpConnectionOverHTTP(EndPoint endPoint, Map<String, Object> context)
     {
@@ -159,12 +162,40 @@ public SendFailure send(HttpExchange exchange)
         return delegate.send(exchange);
     }
 
+    public boolean isInitialize()
+    {
+        return initialize;
+    }
+
+    public void setInitialize(boolean initialize)
+    {
+        this.initialize = initialize;
+    }
+
     @Override
     public void onOpen()
     {
         super.onOpen();
         fillInterested();
-        promise.succeeded(this);
+        boolean initialize = isInitialize();
+        if (initialize)
+        {
+            Destination destination = getHttpDestination();
+            Request request = destination.getHttpClient().newRequest(destination.getOrigin().asString())
+                .method(HttpMethod.OPTIONS)
+                .path("*");
+            send(request, result ->
+            {
+                if (result.isSucceeded())
+                    promise.succeeded(this);
+                else
+                    promise.failed(result.getFailure());
+            });
+        }
+        else
+        {
+            promise.succeeded(this);
+        }
     }
 
     @Override

jetty-core/jetty-client/src/test/java/org/eclipse/jetty/client/ConnectionPoolTest.java

@@ -703,7 +703,7 @@ public void testCountersSweepToStringThroughLifecycle(ConnectionPoolFactory fact
         assertThat(connectionPool.toString(), not(nullValue()));
     }
 
-    private static class ConnectionPoolFactory
+    public static class ConnectionPoolFactory
     {
         private final String name;
         private final ConnectionPool.Factory factory;

jetty-core/jetty-tests/jetty-test-client-transports/src/test/java/org/eclipse/jetty/test/client/transport/AbstractTest.java

@@ -290,6 +290,12 @@ protected SslContextFactory.Server newSslContextFactoryServer()
     }
 
     protected void startClient(Transport transport) throws Exception
+    {
+        prepareClient(transport);
+        client.start();
+    }
+
+    protected void prepareClient(Transport transport) throws Exception
     {
         QueuedThreadPool clientThreads = new QueuedThreadPool();
         clientThreads.setName("client");
@@ -298,7 +304,6 @@ protected void startClient(Transport transport) throws Exception
         client.setByteBufferPool(clientBufferPool);
         client.setExecutor(clientThreads);
         client.setSocketAddressResolver(new SocketAddressResolver.Sync());
-        client.start();
     }
 
     public AbstractConnector newConnector(Transport transport, Server server)

jetty-core/jetty-tests/jetty-test-client-transports/src/test/java/org/eclipse/jetty/test/client/transport/ConnectionPoolTest.java

@@ -0,0 +1,91 @@
+package org.eclipse.jetty.test.client.transport;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import org.eclipse.jetty.client.Destination;
+import org.eclipse.jetty.client.transport.HttpClientTransportOverHTTP;
+import org.eclipse.jetty.fcgi.server.internal.ServerFCGIConnection;
+import org.eclipse.jetty.http2.server.internal.HTTP2ServerConnection;
+import org.eclipse.jetty.io.Connection;
+import org.eclipse.jetty.io.ssl.SslConnection;
+import org.eclipse.jetty.quic.server.ServerQuicConnection;
+import org.eclipse.jetty.server.internal.HttpConnection;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.empty;
+import static org.hamcrest.Matchers.greaterThan;
+import static org.hamcrest.Matchers.not;
+
+public class ConnectionPoolTest extends AbstractTest
+{
+    @ParameterizedTest
+    @MethodSource("transports")
+    public void testPreCreateConnections(Transport transport) throws Exception
+    {
+        prepareServer(transport, new EmptyServerHandler());
+        ConnectionListener serverConnections = new ConnectionListener();
+        connector.addBean(serverConnections);
+        server.start();
+
+        startClient(transport);
+        client.setMaxConnectionsPerDestination(8);
+        if (transport == Transport.HTTPS)
+            ((HttpClientTransportOverHTTP)client.getTransport()).setInitializeConnections(true);
+
+        var request = client.newRequest(newURI(transport));
+        Destination destination = client.resolveDestination(request);
+        destination.getConnectionPool().preCreateConnections(client.getMaxConnectionsPerDestination())
+            .get(5, TimeUnit.SECONDS);
+
+        // Verify that connections have been created.
+        List<Connection> connections = switch (transport)
+        {
+            case HTTP, HTTPS -> serverConnections.filter(HttpConnection.class);
+            case H2C, H2 -> serverConnections.filter(HTTP2ServerConnection.class);
+            case H3 -> serverConnections.filter(ServerQuicConnection.class);
+            case FCGI -> serverConnections.filter(ServerFCGIConnection.class);
+        };
+        assertThat(connections, not(empty()));
+
+        // Verify that TLS was performed.
+        List<Connection> sslConnections = switch (transport)
+        {
+            case HTTP, H2C, FCGI, H3 -> null;
+            case HTTPS, H2 -> serverConnections.filter(SslConnection.class);
+        };
+        if (sslConnections != null)
+        {
+            assertThat(sslConnections.size(), greaterThan(0));
+            sslConnections.forEach(c -> assertThat(c.getBytesIn(), greaterThan(0L)));
+            sslConnections.forEach(c -> assertThat(c.getBytesOut(), greaterThan(0L)));
+        }
+    }
+
+    private static class ConnectionListener implements Connection.Listener
+    {
+        private final List<Connection> connections = new ArrayList<>();
+
+        @Override
+        public void onOpened(Connection connection)
+        {
+            connections.add(connection);
+        }
+
+        @Override
+        public void onClosed(Connection connection)
+        {
+            connections.remove(connection);
+        }
+
+        private List<Connection> filter(Class<? extends Connection> klass)
+        {
+            return connections.stream()
+                .filter(klass::isInstance)
+                .toList();
+        }
+    }
+}