Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task okhttp3.RealCall$AsyncCall@635f40ba rejected from java.util.concurrent.ThreadPoolExecutor@7899bb30[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] #4448

Closed
umutcann opened this issue Sep 26, 2022 · 11 comments

Comments

@umutcann
Copy link

umutcann commented Sep 26, 2022

Describe the bug

Hi,

I build Kubernetes Client like below

 return new KubernetesClientBuilder().withConfig(new OpenShiftConfigBuilder()
        .withMasterUrl(masterUrl)
        .withOauthToken(authToken)
        .withDisableHostnameVerification(true)
        .withNamespace(null)
        .build()).build();

I experience issues when call the kubernetes apis.

Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@635f40ba rejected from java.util.concurrent.ThreadPoolExecutor@7899bb30[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_202]

I debug the KubernetesClientBuilder.build() method. I think problem is related with clazz.getConstructor(Config.class).newInstance(config); line. It doesn't configure ThreadPoolExecuter properly.

java.util.concurrent.ThreadPoolExecutor@3cfab340[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

By the way I reviewed API/Impl split title in Migration from 5.x to 6.x doc. But sample usage doesn't includes withHttpClientFactory(how to create httpclientfactory)

OpenShiftClient openShiftClient = new new KubernetesClientBuilder().withConfig(new OpenShiftConfigBuilder()./*...*/build()).build().adapt(OpenShiftClient.class);

Do you have any suggestion? Is it bug or am i using builder wrongly?

Fabric8 Kubernetes Client version

6.1.1

Steps to reproduce

  1. build Kubernetes Client like below
 return new KubernetesClientBuilder().withConfig(new ConfigBuilder()
        .withMasterUrl(masterUrl)
        .withOauthToken(authToken)
        .withDisableHostnameVerification(true)
        .withNamespace(null)
        .build()).build();
  1. then call any api

Expected behavior

read configmaps properly

Runtime

OpenShift

Kubernetes API Server version

other (please specify in additional context)

Environment

Windows

Fabric8 Kubernetes Client Logs

at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159) ~[kubernetes-client-api-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:177) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:139) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:88) ~[kubernetes-client-6.1.1.jar:?]
	at com.garanti.ark.batch.scheduler.service.impl.OCPClientServiceImpl.isAppConfigExist(OCPClientServiceImpl.java:126) ~[classes/:?]
	at com.garanti.ark.batch.scheduler.service.impl.BatchServiceImpl.getTaskExecutionCommonArguments(BatchServiceImpl.java:877) ~[classes/:?]
	at com.garanti.ark.batch.scheduler.service.impl.BatchServiceImpl.launchJob(BatchServiceImpl.java:519) ~[classes/:?]
	at com.garanti.ark.batch.scheduler.service.impl.BatchServiceImpl.runService(BatchServiceImpl.java:158) ~[classes/:?]
	at com.garanti.ark.batch.scheduler.rest.contract.impl.JobLauncherControllerImpl.startJob(JobLauncherControllerImpl.java:59) ~[classes/:?]
	at com.garanti.ark.batch.scheduler.rest.contract.impl.JobLauncherControllerImpl$$FastClassBySpringCGLIB$$e5ec9b29.invoke(<generated>) ~[classes/:?]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) ~[spring-aop-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.13.jar:5.3.13]
	at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:123) ~[spring-context-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.13.jar:5.3.13]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698) ~[spring-aop-5.3.13.jar:5.3.13]
	at com.garanti.ark.batch.scheduler.rest.contract.impl.JobLauncherControllerImpl$$EnhancerBySpringCGLIB$$fff5edd1.startJob(<generated>) ~[classes/:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_202]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_202]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_202]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.13.jar:5.3.13]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) [spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) [spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.3.13.jar:5.3.13]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) [spring-webmvc-5.3.13.jar:5.3.13]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) [tomcat-embed-core-9.0.55.jar:4.0.FR]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.3.13.jar:5.3.13]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) [tomcat-embed-core-9.0.55.jar:4.0.FR]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) [spring-web-5.3.13.jar:5.3.13]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.13.jar:5.3.13]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at com.garanti.ark.online.cor.rest.filter.UidFilter.doFilter(UidFilter.java:64) [ark-online-cor-rest-13.1.0-alpha.3.jar:13.1.0-alpha.3]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.springframework.cloud.sleuth.instrument.web.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:50) [spring-cloud-sleuth-core-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at brave.servlet.TracingFilter.doFilter(TracingFilter.java:99) [brave-instrumentation-servlet-5.6.5.jar:?]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at com.garanti.ark.online.cor.rest.filter.LoggingFilter.doFilter(LoggingFilter.java:65) [ark-online-cor-rest-13.1.0-alpha.3.jar:13.1.0-alpha.3]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.13.jar:5.3.13]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.13.jar:5.3.13]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1722) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.55.jar:9.0.55]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: java.io.InterruptedIOException: executor rejected
	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:157) ~[okhttp-3.14.9.jar:?]
	at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:204) ~[okhttp-3.14.9.jar:?]
	at okhttp3.Dispatcher.enqueue(Dispatcher.java:144) ~[okhttp-3.14.9.jar:?]
	at okhttp3.RealCall.enqueue(RealCall.java:93) ~[okhttp-3.14.9.jar:?]
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.sendAsync(OkHttpClientImpl.java:264) ~[kubernetes-httpclient-okhttp-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:598) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:577) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:558) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:469) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:711) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:172) ~[kubernetes-client-6.1.1.jar:?]
	... 75 more
Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@635f40ba rejected from java.util.concurrent.ThreadPoolExecutor@7899bb30[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_202]
	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:154) ~[okhttp-3.14.9.jar:?]
	at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:204) ~[okhttp-3.14.9.jar:?]
	at okhttp3.Dispatcher.enqueue(Dispatcher.java:144) ~[okhttp-3.14.9.jar:?]
	at okhttp3.RealCall.enqueue(RealCall.java:93) ~[okhttp-3.14.9.jar:?]
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.sendAsync(OkHttpClientImpl.java:264) ~[kubernetes-httpclient-okhttp-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:598) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:577) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:558) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:469) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:711) ~[kubernetes-client-6.1.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:172) ~[kubernetes-client-6.1.1.jar:?]
	... 75 more

Additional context

Kubernetes version :v1.21.6+b4b4813

@shawkins
Copy link
Contributor

It's likely that you are attempting to use the client after closing it.

@umutcann
Copy link
Author

umutcann commented Sep 26, 2022

Actually first request is getting error. There is no close action for client in my project.

client.configMaps()
        .inNamespace(projectName).withName(configMapName).get();

Kubernetes Client Builder doesn't configure ThreadPoolExecuter properly.

java.util.concurrent.ThreadPoolExecutor@3cfab340[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

PS: I build two clients in my projects with different masterURL.

@shawkins
Copy link
Contributor

You'll need to post a full example of your code. The only reason for that threadpool to reject is if it has been shutdown.

@umutcann
Copy link
Author

Client creation part:

@OverRide
public OpenShiftClient createOpenshiftClient(String masterUrl, String authToken) {

return new KubernetesClientBuilder()
.withConfig(new OpenShiftConfigBuilder().withMasterUrl(masterUrl)
.withOauthToken(authToken)
.withDisableHostnameVerification(true)
.withNamespace(null).build()).build().adapt(OpenShiftClient.class);

}

Exception thrown line:
io.fabric8.kubernetes.api.model.ConfigMap configMap = client.configMaps()
.inNamespace(projectName).withName(configMapName).get();

@shawkins
Copy link
Contributor

Unfortunately that's not quite complete. You can try the configMap call in your createOpenshiftClient method to confirm that the client is usable at that point or put a breakpoint in OkHttpClientImpl.close to see where it is being closed.

@umutcann
Copy link
Author

I debug it.

There was a try block at point that I call createOpenshiftClient method. At end of the try block, I saw that close method call. After remove try block, I could receive the configmap.

@adietish
Copy link
Contributor

adietish commented Nov 8, 2022

@umutcann can you please elaborate a bit how you fixed your problem? I'd appreciate a lot. I could face something similar, with cient 5.x though.

@umutcann
Copy link
Author

umutcann commented Nov 9, 2022

@adietish before client 5.x version, I was calling my createOpenshiftClient method(I shared above) in try block like below.

try (OpenShiftClient openShiftClient = openshiftClientFactory
.createOpenshiftClient(ocpClientProperties.getFirstOCP(),
ocpClientProperties.getFirstToken())) {
this.firstOpenShiftClient = openShiftClient;
}

I noticed that connection was closing at the end of try block. I removed try block then worked :)

@adietish
Copy link
Contributor

adietish commented Nov 9, 2022

@umutcann: thanks for the details. Your explanation makes a lot of sense: try with resource would close the resource when getting out of the try-block. Gotcha. Unfortunately different from my issue. Will file a new one

@shawkins
Copy link
Contributor

shawkins commented Nov 9, 2022

@adietish no need for a new issue - just determine where you are closing the client.

@wind57
Copy link
Contributor

wind57 commented Dec 7, 2022

It's likely that you are attempting to use the client after closing it.

This was a great hint!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants