Jetty 12.0.5: High CPU/Load after some time in production, no active thread with business code #11326

gbrehmer · 2024-01-26T14:33:12Z

Jetty version(s)
12.0.5

Jetty Environment
ee10? core? (Spring Boot 3.2.2 Jetty embedded)

Java version/vendor (use: java -version)
17.0.9 temurin

OS type/version
Linux

Description
I created a thread dump from spring boot actuator during the high load but probably next time i need a better one which also shows cpu times etc.
All relevant threads are not sitting in parts of our business code. So I assume that there is some infinite loop or something like that in rare cases in jetty. Can you see anything suspicious? Thanks for you support!
I checked also JVM memory stats: Heap was filled by 11% only

One example thread

  {
    "threadName": "qtp198220326-9868",
    "threadId": 9868,
    "blockedTime": -1,
    "blockedCount": 0,
    "waitedTime": -1,
    "waitedCount": 3,
    "lockOwnerId": -1,
    "daemon": false,
    "inNative": false,
    "suspended": false,
    "threadState": "RUNNABLE",
    "priority": 5,
    "stackTrace": [
      {
        "moduleName": "java.base",
        "moduleVersion": "17.0.9",
        "methodName": "wrap",
        "lineNumber": -1,
        "className": "java.util.stream.ReferencePipeline",
        "nativeMethod": false
      },
      {
        "moduleName": "java.base",
        "moduleVersion": "17.0.9",
        "methodName": "spliterator",
        "lineNumber": -1,
        "className": "java.util.stream.AbstractPipeline",
        "nativeMethod": false
      },
      {
        "moduleName": "java.base",
        "moduleVersion": "17.0.9",
        "methodName": "concat",
        "lineNumber": -1,
        "className": "java.util.stream.Stream",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "stream",
        "fileName": "CompoundPool.java",
        "lineNumber": 82,
        "className": "org.eclipse.jetty.io.internal.CompoundPool",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "findOldestEntry",
        "fileName": "ArrayByteBufferPool.java",
        "lineNumber": 461,
        "className": "org.eclipse.jetty.io.ArrayByteBufferPool",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "evict",
        "fileName": "ArrayByteBufferPool.java",
        "lineNumber": 408,
        "className": "org.eclipse.jetty.io.ArrayByteBufferPool",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "releaseExcessMemory",
        "fileName": "ArrayByteBufferPool.java",
        "lineNumber": 383,
        "className": "org.eclipse.jetty.io.ArrayByteBufferPool",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "acquire",
        "fileName": "ArrayByteBufferPool.java",
        "lineNumber": 215,
        "className": "org.eclipse.jetty.io.ArrayByteBufferPool",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "getRequestBuffer",
        "fileName": "HttpConnection.java",
        "lineNumber": 375,
        "className": "org.eclipse.jetty.server.internal.HttpConnection",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "fillRequestBuffer",
        "fileName": "HttpConnection.java",
        "lineNumber": 534,
        "className": "org.eclipse.jetty.server.internal.HttpConnection",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "onFillable",
        "fileName": "HttpConnection.java",
        "lineNumber": 399,
        "className": "org.eclipse.jetty.server.internal.HttpConnection",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "succeeded",
        "fileName": "AbstractConnection.java",
        "lineNumber": 322,
        "className": "org.eclipse.jetty.io.AbstractConnection$ReadCallback",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "fillable",
        "fileName": "FillInterest.java",
        "lineNumber": 99,
        "className": "org.eclipse.jetty.io.FillInterest",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "run",
        "fileName": "SelectableChannelEndPoint.java",
        "lineNumber": 53,
        "className": "org.eclipse.jetty.io.SelectableChannelEndPoint$1",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "runTask",
        "fileName": "AdaptiveExecutionStrategy.java",
        "lineNumber": 478,
        "className": "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "consumeTask",
        "fileName": "AdaptiveExecutionStrategy.java",
        "lineNumber": 441,
        "className": "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "tryProduce",
        "fileName": "AdaptiveExecutionStrategy.java",
        "lineNumber": 293,
        "className": "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "run",
        "fileName": "AdaptiveExecutionStrategy.java",
        "lineNumber": 201,
        "className": "org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "run",
        "fileName": "ReservedThreadExecutor.java",
        "lineNumber": 410,
        "className": "org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "runJob",
        "fileName": "QueuedThreadPool.java",
        "lineNumber": 971,
        "className": "org.eclipse.jetty.util.thread.QueuedThreadPool",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "doRunJob",
        "fileName": "QueuedThreadPool.java",
        "lineNumber": 1201,
        "className": "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner",
        "nativeMethod": false
      },
      {
        "classLoaderName": "app",
        "methodName": "run",
        "fileName": "QueuedThreadPool.java",
        "lineNumber": 1156,
        "className": "org.eclipse.jetty.util.thread.QueuedThreadPool$Runner",
        "nativeMethod": false
      },
      {
        "moduleName": "java.base",
        "moduleVersion": "17.0.9",
        "methodName": "run",
        "lineNumber": -1,
        "className": "java.lang.Thread",
        "nativeMethod": false
      }
    ],
    "lockedMonitors": [],
    "lockedSynchronizers": []
  },

Full threaddump:
jsonformatter.txt

How to reproduce?
sorry currently only occuring like once a week on one production server (pod in k8s, not always same one)

The text was updated successfully, but these errors were encountered:

sbordet · 2024-01-26T15:00:59Z

We are not aware of any CPU spinning loops in Jetty.

The example thread you reported is trying to acquire a buffer as part of the standard processing of a request.
It loops over a snapshot of the buffers, so the iteration is stable and bound to terminate.

Can you report standard, non-JSON, stack traces?
Or, pardon my ignorance, suggest a tool to reconvert the JSON stack traces to standard?
As they are quite impossible to analyze given the JSON noise.

If you can monitor your environment, can you use a tool like TTOP (https://github.com/aragozin/jvm-tools/blob/master/sjk-core/docs/TTOP.md) to check whether the high CPU is caused by 1 thread spinning, or many threads?

Also JMC, if you can connect, will be able to report live whether it's just one thread consuming the CPU, or many threads (and you can get a stack trace from the most CPU consuming thread).

joakime · 2024-01-26T15:37:38Z

You can use the built-in tooling with the JVM to get a reasonable (and useful) thread dump.

jstack -l <pid>

gbrehmer · 2024-01-26T16:27:27Z

Thanks for your fast feedback. Yes jstack was already in my mind, but its currently not out-of-the-box available in our jib created temurin-jre docker containers. of course we could also switch to jdk, but im searching for a more lightweight solution.
so you are searching especially for cpu timing stats right?

joakime · 2024-01-26T17:14:37Z

so you are searching especially for cpu timing stats right?

What the thread dumps say over time is likely going to be more important than cpu timings

gbrehmer · 2024-01-27T21:19:37Z

it will take probably some days to get more details. we had this situation 4 times in the last 3 weeks and sometimes > 7 days between each incident. I will com back when I have the details

in addition i checked also memory stuff and there is all fine, so its not for example GC related

gbrehmer · 2024-02-02T15:27:30Z

Here is a threaddump from jstack:
threaddump_jetty_12_0_5_20240202.txt

Currently its occurring every 2-3 days when we not restart/redploy the nodes manually

We will also try out 12.0.6 now

And if provided data is not enough to get an idea we will check whether we can use the mentioned live monitoring tools

sbordet · 2024-02-02T18:40:52Z

@gbrehmer thanks for the thread dump.

It shows that many Jetty threads are in ArrayByteBufferPool.evict().
We identified problems in that code, filed as #11371.

The question for you is how do you (or Spring) configures the Jetty ArrayByteBufferPool?
We will be interested in the precise settings to see whether they are custom, and if so, if they are a combination that is not good and leads to this additional CPU usage.

Otherwise, we can suggest settings that hopefully would work around the problem and reduce CPU usage.

gbrehmer · 2024-02-02T21:58:06Z

The question for you is how do you (or Spring) configures the Jetty ArrayByteBufferPool?

our setup is from our perspective quite simple. because we just running jetty embedded in a mostly default setup from spring boot 3.2.2 (https://github.com/spring-projects/spring-boot/blob/550651f88fdb69378ea650946c2507fa05cf34fa/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/web/embedded/jetty/JettyServletWebServerFactory.java#L207). So ArrayByteBufferPool is using default values from no-args-constructor. Currently it seems not so easy to change the settings their, because ByteBufferPool must be set in the constructor and spring currently does not support any customization in this location.

The only customizations we made in the spring settings (but not ArrayByteBufferPool related probably):

max-http-request-header-size 64KB
jetty.max-http-form-post-size 2MB

We running this in k8s with 2000 as cpu resource limit and configured the jvm fixed to ActiveProcessorCount = 3

Otherwise, we can suggest settings that hopefully would work around the problem and reduce CPU usage.

This sounds great!

lorban · 2024-02-06T11:40:23Z

@gbrehmer I'm working on an experimental Jetty branch that may help you with your reported problem. Since we don't have too many details, it's hard to say if that branch will solve your issue, but it's certainly worth a shot if you can spare the time.

It contains two improvements:

much improved statistics reported by ArrayByteBufferPool that should help us better understand the state of your pool
a slightly reviewed buffer eviction algorithm that should be more performant

So if you could build this PR's branch, retry your test and collect a few server dumps while you test is running, that data would be invaluable to help us move forward.

Thanks!

gbrehmer · 2024-02-06T13:27:37Z

The problem is that "my test" is our production system. I currently can not reproduce the issue, I only know that after some days the load quickly increases to 100%. From 1-5% as the median load to 100% in 10min, so I assume that the important stats are only available during this timeframe. We will discuss whether we can spent the effort to create a custom build and probably we need a special endpoint to trigger the jetty dump, because we normally have no JMX active.

gbrehmer · 2024-02-06T13:43:25Z

or what about exposing such stats as metrics? We already integrated jetty with micrometer by using InstrumentedQueuedThreadPool
Is there something else which is already available?

lorban · 2024-02-06T15:27:10Z

Unfortunately, we do not have a micrometer integration. But if you could find a way to call dumpStdErr() on your server instance that would be great.

Beware that this is an experimental branch. While it received some testing, if you're willing to try it in production, be prepared to rollback to the stable version if needed!

gbrehmer · 2024-02-07T11:26:28Z

so only a server dump from this branch would be helpful and the current one from 12.0.6 will not give you much more insights? Because added a "dump server" feature is probably easier to accomplish on our side

lorban · 2024-02-07T14:29:21Z

Correct, the information reported by the dump in version 12.0.6 does not contain enough information to get an idea of the state of your pool.

ignusin · 2024-02-08T07:35:26Z

Same issue happened to us, after upgrading to Spring Boot 3.2.1 with Jetty 12.0.5.

So far we're testing our solution with Spring Boot 3.1.8 and Jetty 11. So far seems to work fine.

Steps to reproduce:

Connect 100-500 web-socket clients to Jetty12-based server;
Disconnect all of the clients from step 1;
Repeat step (1) - (2) few times.

Expected behavior: server runs fine.
Actual behavior: server accepts connections, but does not serve them. CPU goes 100% on all cores, Jetty12 jvm-process eats them all.

Here is the stack trace I managed to get from our freezing Jetty container.
Our log was filled with many of the following records (Jetty 12.0.5):

java.lang.NullPointerException:` Cannot invoke "org.eclipse.jetty.io.RetainableByteBuffer.capacity()" because "buffer" is null
	at org.eclipse.jetty.io.ArrayByteBufferPool.evict(ArrayByteBufferPool.java:415) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.io.ArrayByteBufferPool.releaseExcessMemory(ArrayByteBufferPool.java:383) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.io.ArrayByteBufferPool.acquire(ArrayByteBufferPool.java:215) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.server.internal.HttpConnection$SendCallback.process(HttpConnection.java:786) ~[jetty-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:250) ~[jetty-util-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:231) ~[jetty-util-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.server.internal.HttpConnection$HttpStreamOverHTTP1.send(HttpConnection.java:1416) ~[jetty-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.server.HttpStream$Wrapper.send(HttpStream.java:179) ~[jetty-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.server.HttpStream$Wrapper.send(HttpStream.java:179) ~[jetty-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.session.AbstractSessionManager$SessionStreamWrapper.send(AbstractSessionManager.java:1457) ~[jetty-session-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.write(HttpChannelState.java:1215) ~[jetty-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.server.Response$Wrapper.write(Response.java:735) ~[jetty-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56) ~[jetty-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker.upgradeRequest(AbstractHandshaker.java:139) ~[jetty-websocket-core-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.websocket.core.server.internal.HandshakerSelector.upgradeRequest(HandshakerSelector.java:45) ~[jetty-websocket-core-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.websocket.core.server.WebSocketMappings.upgrade(WebSocketMappings.java:296) ~[jetty-websocket-core-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.websocket.server.JettyWebSocketServerContainer.upgrade(JettyWebSocketServerContainer.java:228) ~[jetty-ee10-websocket-jetty-server-12.0.5.jar:12.0.5]
	at org.springframework.web.socket.server.jetty.JettyRequestUpgradeStrategy.upgrade(JettyRequestUpgradeStrategy.java:117) ~[spring-websocket-6.1.2.jar:6.1.2]
	at org.springframework.web.socket.server.support.AbstractHandshakeHandler.doHandshake(AbstractHandshakeHandler.java:259) ~[spring-websocket-6.1.2.jar:6.1.2]
	at org.springframework.web.socket.server.support.WebSocketHttpRequestHandler.handleRequest(WebSocketHttpRequestHandler.java:177) ~[spring-websocket-6.1.2.jar:6.1.2]
	at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:52) ~[spring-webmvc-6.1.2.jar:6.1.2]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) ~[spring-webmvc-6.1.2.jar:6.1.2]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.1.2.jar:6.1.2]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.1.2.jar:6.1.2]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.1.2.jar:6.1.2]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527) ~[jakarta.servlet-api-6.0.0.jar:6.0.0]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.1.2.jar:6.1.2]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614) ~[jakarta.servlet-api-6.0.0.jar:6.0.0]
	at org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:736) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1614) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.websocket.servlet.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:195) ~[jetty-ee10-websocket-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.1.2.jar:6.1.2]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.2.jar:6.1.2]
	at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.1.2.jar:6.1.2]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.2.jar:6.1.2]
	at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.1.2.jar:6.1.2]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.2.jar:6.1.2]
	at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:797) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:428) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:571) ~[jetty-security-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:703) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:761) ~[jetty-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.server.Server.handle(Server.java:179) ~[jetty-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:594) ~[jetty-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:424) ~[jetty-server-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:971) ~[jetty-util-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1201) ~[jetty-util-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1156) ~[jetty-util-12.0.5.jar:12.0.5]
	at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
	Suppressed: java.lang.IllegalStateException: s=HANDLING rs=COMPLETED os=ABORTED is=IDLE awp=false se=false i=false al=0
		at org.eclipse.jetty.ee10.servlet.ServletChannelState.completed(ServletChannelState.java:1052) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.util.Callback$3.failed(Callback.java:168) ~[jetty-util-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.ee10.servlet.HttpOutput.onWriteComplete(HttpOutput.java:252) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.ee10.servlet.HttpOutput$WriteCompleteCB.failed(HttpOutput.java:1760) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.ee10.servlet.HttpOutput$CompleteWriteCompleteCB.failed(HttpOutput.java:1777) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1177) ~[jetty-server-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.server.handler.ContextResponse$1.failed(ContextResponse.java:47) ~[jetty-server-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.lambda$write$2(HttpChannelState.java:1201) ~[jetty-server-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191) ~[jetty-util-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.util.thread.SerializedInvoker.run(SerializedInvoker.java:117) ~[jetty-util-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.write(HttpChannelState.java:1201) ~[jetty-server-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.server.Response$Wrapper.write(Response.java:735) ~[jetty-server-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56) ~[jetty-server-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.ee10.servlet.HttpOutput.channelWrite(HttpOutput.java:204) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.ee10.servlet.HttpOutput.complete(HttpOutput.java:435) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.ee10.servlet.ServletContextResponse.completeOutput(ServletContextResponse.java:209) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
		at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:553) ~[jetty-ee10-servlet-12.0.5.jar:12.0.5]
		... 14 common frames omitted

2024-02-07T10:07:47.158Z  WARN 93 --- [tp461309639-270] o.e.jetty.util.thread.QueuedThreadPool   : Job failed

java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.io.RetainableByteBuffer.capacity()" because "buffer" is null
	at org.eclipse.jetty.io.ArrayByteBufferPool.evict(ArrayByteBufferPool.java:415) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.io.ArrayByteBufferPool.releaseExcessMemory(ArrayByteBufferPool.java:383) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.io.ArrayByteBufferPool.acquire(ArrayByteBufferPool.java:215) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.websocket.core.WebSocketConnection.newNetworkBuffer(WebSocketConnection.java:319) ~[jetty-websocket-core-common-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.websocket.core.WebSocketConnection.acquireNetworkBuffer(WebSocketConnection.java:298) ~[jetty-websocket-core-common-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.websocket.core.WebSocketConnection.fillAndParse(WebSocketConnection.java:441) ~[jetty-websocket-core-common-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.websocket.core.WebSocketConnection.onFillable(WebSocketConnection.java:342) ~[jetty-websocket-core-common-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:971) ~[jetty-util-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1201) ~[jetty-util-12.0.5.jar:12.0.5]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1156) ~[jetty-util-12.0.5.jar:12.0.5]
	at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

lorban · 2024-02-08T09:16:10Z

@ignusin your problem seems to be #11098 which was fixed in Jetty 12.0.6. Could you please try to reconfigure your environment to force Spring Boot to use Jetty 12.0.6 instead of 12.0.5? That should solve that NPE.

@gbrehmer while I think your problem is slightly different, have you tried to force Spring Boot to use Jetty 12.0.6? Did that help?

Thanks!

gbrehmer · 2024-02-10T15:11:49Z

we have 12.0.6 running since 8 days but with interruptions (restarts) caused by new feature deployments and previously it seems that such problems only occur when service is running several days w/o a reboot. So far we had no high cpu usage event anymore, only one unknown k8s oom restart but probably not related. I would say if we have one additional week w/o problems, than 12.0.6 fixes our problems

gbrehmer · 2024-02-22T14:53:17Z

so far no more problems with 12.0.6. Thanks!

gbrehmer added the Bug For general bugs on Jetty side label Jan 26, 2024

lorban mentioned this issue Feb 5, 2024

Add more instrumentation to ArrayByteBufferPool and improve its eviction algorithm #11375

Closed

lorban mentioned this issue Feb 9, 2024

Service degradation at 99.9 percentiles in latest jetty versions #11342

Closed

konstantin-mikheev mentioned this issue Feb 21, 2024

Jetty 12.0.6: NPE in error handling leading to 100% cpu usage by Jetty threads #11431

Closed

gbrehmer closed this as completed Feb 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jetty 12.0.5: High CPU/Load after some time in production, no active thread with business code #11326

Jetty 12.0.5: High CPU/Load after some time in production, no active thread with business code #11326

gbrehmer commented Jan 26, 2024 •

edited

Loading

sbordet commented Jan 26, 2024

joakime commented Jan 26, 2024

gbrehmer commented Jan 26, 2024 •

edited

Loading

joakime commented Jan 26, 2024

gbrehmer commented Jan 27, 2024

gbrehmer commented Feb 2, 2024 •

edited

Loading

sbordet commented Feb 2, 2024

gbrehmer commented Feb 2, 2024 •

edited

Loading

lorban commented Feb 6, 2024

gbrehmer commented Feb 6, 2024 •

edited

Loading

gbrehmer commented Feb 6, 2024

lorban commented Feb 6, 2024

gbrehmer commented Feb 7, 2024

lorban commented Feb 7, 2024

ignusin commented Feb 8, 2024 •

edited

Loading

lorban commented Feb 8, 2024

gbrehmer commented Feb 10, 2024

gbrehmer commented Feb 22, 2024

Jetty 12.0.5: High CPU/Load after some time in production, no active thread with business code #11326

Jetty 12.0.5: High CPU/Load after some time in production, no active thread with business code #11326

Comments

gbrehmer commented Jan 26, 2024 • edited Loading

sbordet commented Jan 26, 2024

joakime commented Jan 26, 2024

gbrehmer commented Jan 26, 2024 • edited Loading

joakime commented Jan 26, 2024

gbrehmer commented Jan 27, 2024

gbrehmer commented Feb 2, 2024 • edited Loading

sbordet commented Feb 2, 2024

gbrehmer commented Feb 2, 2024 • edited Loading

lorban commented Feb 6, 2024

gbrehmer commented Feb 6, 2024 • edited Loading

gbrehmer commented Feb 6, 2024

lorban commented Feb 6, 2024

gbrehmer commented Feb 7, 2024

lorban commented Feb 7, 2024

ignusin commented Feb 8, 2024 • edited Loading

lorban commented Feb 8, 2024

gbrehmer commented Feb 10, 2024

gbrehmer commented Feb 22, 2024

gbrehmer commented Jan 26, 2024 •

edited

Loading

gbrehmer commented Jan 26, 2024 •

edited

Loading

gbrehmer commented Feb 2, 2024 •

edited

Loading

gbrehmer commented Feb 2, 2024 •

edited

Loading

gbrehmer commented Feb 6, 2024 •

edited

Loading

ignusin commented Feb 8, 2024 •

edited

Loading