Is it possible to define max number of virtual threads when VirtualThreadsExecutor is enabled, i.e. max number of http requests being handled in the same time?

[ascho]

12.0.12

Java 21

Question
Hi!
Is it possible to define max number of virtual threads when VirtualThreadsExecutor is enabled, i.e. max number of http requests being handled in the same time?

I tried to dig into source code and as a result assumed that any request submitted to Jetty is dispatched to virtual thread executor via AdaptiveExecutionStrategy as is so it doesn't mater 1k or 1kk requests came.
Thanks!

[joakime]

There is no relationship of connections (or requests) to thread count.
Manipulating threading in an attempt to limit connections (or requests) is not possible.
Drop that as an idea, it's a dead end.

Look into using org.eclipse.jetty.server.ConnectionLimit instead.
https://javadoc.jetty.org/jetty-12/org/eclipse/jetty/server/ConnectionLimit.html

As a different solution, consider QoSHandler
https://jetty.org/docs/jetty/12/programming-guide/server/http.html#handler-use-qos

[joakime]

There is a connectionlimit module available on jetty-base / jetty-home setups that you can enable (and configure via the ${jetty.base}/start.d/connectionlimit.ini) if you choose to use that technique.

[sbordet]

@ascho we are planning to add a limit to VirtualThreadPool that would be configurable.

ConnectionLimit works well for HTTP/1.1, but not so well for HTTP/2, so we really need a limiter at a different level.

[ascho]

@joakime thank you!

I will try to use QoSHandler it seems that it solves the problem of defining the upper bound of handled requests in the moment of time.

As i understand there is an unlimited priority queue to store requests which exceeded max request count and QoSHandler reschedules these suspended requests after MaxSuspend duration only 1 time and next reschedule gives 503?

I am a little bit confused with this logic meaning suspended request is not removed from CyclicTimeouts iterable stuff.

@sbordet thanks 🙏 ! Waiting for it :)

[ascho]

oh i see
expire removes an Entry so next iteration step will not have removed element obviously

[sbordet]

@ascho can you try #12155 and see if it works for you?

[ascho]

@sbordet thank you 🙏
i decided to go with QoSHandler

with all respect to your effort it is too risky for me to use change with concurrency primitives in the prod for now 😭
i
will watch for the changes attentively

[ascho]

@joakime as you adviced i used QoSHandler and noticed interesting issue.

It seems that:

WHEN Jetty is configured to use Virtual Threads
AND QoSHandler is used
AND if http request was suspended
THEN http request is retried on ThreadPool instead of VirtualThreadExecutor :)

i am observing that retrying of the wave of suspended http requests grow ThreadPool to defined maximum of threads.
ViirtualThreadExecutor is used is AdaptiveExecutionStrategy.
And QoSHandler schedules request onThreadPool directly.

Is my assumption correct?
maybe you can advice about how to workaround the issue?

UPD:
if request AttributeNameSet contains 'org.eclipse.jetty.server.handler.QoSHandler.expired' then i am printing current thread info in onResponseBegin method of event handler getting this:

{:xs "VirtualThread[5715]/runnable@ForkJoinPool-1-worker-7"}
{:xs "VirtualThread[5716]/runnable@ForkJoinPool-1-worker-7"}
{:xs "VirtualThread[5717]/runnable@ForkJoinPool-1-worker-2"}
{:xs "VirtualThread[5718]/runnable@ForkJoinPool-1-worker-7"}
{:xs "VirtualThread[5719]/runnable@ForkJoinPool-1-worker-2"}
{:xs "Thread[5687,qtp671989905-5687,5,main]"} <---------------------------------------- not virtual thread
{:xs "Thread[5688,qtp671989905-5688,5,main]"} <---------------------------------------- not virtual thread
{:xs "VirtualThread[5723]/runnable@ForkJoinPool-1-worker-5"}
{:xs "VirtualThread[5724]/runnable@ForkJoinPool-1-worker-5"}
{:xs "VirtualThread[5726]/runnable@ForkJoinPool-1-worker-5"}
{:xs "VirtualThread[5727]/runnable@ForkJoinPool-1-worker-5"}
{:xs "VirtualThread[5728]/runnable@ForkJoinPool-1-worker-5"}
{:xs "VirtualThread[5729]/runnable@ForkJoinPool-1-worker-2"}

thanks 🙏

[sbordet]

@ascho if the request attribute org.eclipse.jetty.server.handler.QoSHandler.expired is present, then the request is failed with a 503, and yes, the response failure may happen on a non-virtual thread.

However, I can see how also resumed requests may end up in a non-virtual thread.
Opened #12171 to track this, please continue conversation there.