Multiple threads blocked in org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970) #15231

skabashnyuk · 2019-11-19T15:22:43Z

Describe the bug

Multiple threads blocked in org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970)


"http-nio-8080-exec-36" #230574 daemon prio=5 os_prio=0 tid=0x00007f73f8079000 nid=0xfd3 in Object.wait() [0x00007f73d8454000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at org.jgroups.protocols.Locking$ClientLock.acquire(Locking.java:1048)
	- locked <0x0000000084af2848> (a org.jgroups.protocols.Locking$ClientLock)
	at org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970)
	at org.jgroups.protocols.Locking.down(Locking.java:155)
	at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:918)
	at org.jgroups.JChannel.down(JChannel.java:627)
	at org.jgroups.blocks.locking.LockService$LockImpl.lock(LockService.java:109)
	at org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.writeLock(JGroupsWorkspaceLockService.java:60)
	at org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.readLock(JGroupsWorkspaceLockService.java:54)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.getStatus(WorkspaceRuntimes.java:363)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.normalizeState(WorkspaceManager.java:491)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.getWorkspaces(WorkspaceManager.java:247)
	at org.eclipse.che.api.workspace.server.WorkspaceService.getWorkspaces(WorkspaceService.java:320)
	at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source)

"Annotated-scheduler-4" #87 prio=5 os_prio=0 tid=0x00007f7431521800 nid=0xec in Object.wait() [0x00007f73ea4f1000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at org.jgroups.protocols.Locking$ClientLock.acquire(Locking.java:1048)
	- locked <0x000000008402d660> (a org.jgroups.protocols.Locking$ClientLock)
	at org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970)
	at org.jgroups.protocols.Locking.down(Locking.java:155)
	at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:918)
	at org.jgroups.JChannel.down(JChannel.java:627)
	at org.jgroups.blocks.locking.LockService$LockImpl.lock(LockService.java:109)
	at org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.writeLock(JGroupsWorkspaceLockService.java:60)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.injectRuntime(WorkspaceRuntimes.java:278)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.normalizeState(WorkspaceManager.java:489)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.getWorkspace(WorkspaceManager.java:203)
	at org.eclipse.che.api.workspace.activity.WorkspaceActivityChecker.stopExpired(WorkspaceActivityChecker.java:120)
	at org.eclipse.che.api.workspace.activity.WorkspaceActivityChecker$$Lambda$711/164664036.accept(Unknown Source)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.eclipse.che.api.workspace.activity.WorkspaceActivityChecker.stopAllExpired(WorkspaceActivityChecker.java:115)
	at org.eclipse.che.api.workspace.activity.WorkspaceActivityChecker.expire(WorkspaceActivityChecker.java:90)
	at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source)

ovember 19th 2019, 09:21:51.746
  failed getting JSON response from Kubernetes Client
  [
    masterUrl=https://172.30.0.1:443/api/v1,
    headers={Authorization=#MASKED:869#},
    connectTimeout=5000,
    readTimeout=30000,
    operationAttempts=3,
    operationSleep=1000,
    streamProvider=
       org.jgroups.protocols.kubernetes.stream.TokenStreamProvider@2dffc50
  ]
  for cluster
    [WorkspaceLocks], namespace [che], labels [app=che,component=che];
  encountered [
    java.lang.Exception: 3 attempt(s) with a 1000ms sleep to execute
    [OpenStream] failed. Last failure was [
      java.io.IOException: Server returned HTTP response code: 403 for URL:
      https://172.30.0.1:443/api/v1/namespaces/che/pods?labelSelector=app%3Dche%2Ccomponent%3Dche
    ]
  ]

This issue caused the suspension of the process of workspace idling.

Che version

7.3.1

Steps to reproduce

Uknown

Expected behavior

no traffic jam

Runtime

kubernetes (include output of kubectl version)
Openshift (include output of oc version)
minikube (include output of minikube version and kubectl version)
minishift (include output of minishift version and oc version)
docker-desktop + K8S (include output of docker version and kubectl version)
other: (please specify)

OpenShift Master:
v3.11.43
Kubernetes Master:
v1.11.0+d4cacc0
OpenShift Web Console:
v3.11.43

Screenshots

Installation method

chectl
che-operator
minishift-addon
I don't know

Environment

Additional context

The text was updated successfully, but these errors were encountered:

mshaposhnik · 2019-11-25T10:08:02Z

So the problem still under invertigation, there was several updates of production with and without view role applied to service account. When role is disabled, it seems there is no problems with workspaces idling and thread locks, but replication didn't work.
We're also set up the same version and granted view role on prod-preview, tried with diagnostic tool and everything seems went fine, replication is working:

h-4.2$ cd /home/user/eclipse-che/tomcat/webapps/api/WEB-INF/lib/
sh-4.2$ java -cp jgroups-4.1.0.Final.jar org.jgroups.tests.Probe
#1 (199 bytes):
local_addr=rhche-5-4n65k-9435
physical_addr=10.131.1.155:7802
view=[rhche-5-4n65k-9435|9] (2) [rhche-5-4n65k-9435, rhche-5-2vrbs-38648]
cluster=WorkspaceStateCache
version=4.1.0.Final (Mont Ventoux)#2 (208 bytes):
local_addr=rhche-5-4n65k-63794
physical_addr=10.131.1.155:7803
view=[rhche-5-4n65k-63794|9] (2) [rhche-5-4n65k-63794, rhche-5-2vrbs-29518]
cluster=EclipseLinkCommandChannel
version=4.1.0.Final (Mont Ventoux)#3 (191 bytes):
local_addr=rhche-5-4n65k-378
physical_addr=10.131.1.155:7801
view=[rhche-5-4n65k-378|9] (2) [rhche-5-4n65k-378, rhche-5-2vrbs-61983]
cluster=WorkspaceLocks
version=4.1.0.Final (Mont Ventoux)#4 (208 bytes):
local_addr=rhche-5-4n65k-49664
physical_addr=10.131.1.155:7800
view=[rhche-5-4n65k-49664|9] (2) [rhche-5-4n65k-49664, rhche-5-2vrbs-15668]
cluster=RemoteSubscriptionChannel
version=4.1.0.Final (Mont Ventoux)4 responses (4 matches, 0 non matches)

Now production is again re-updated with view role and we're monitoring for idling status and thread locks.

JGroups Related PR-s:
#14920 - Initial version and configuration update;

eclipse-che/che-operator#126 - setting KUBERNETES_LABEL env variable to have correct value on all flavors (backport to crw-2.0 eclipse-che/che-operator#127)

#15282 +
eclipse-che/che-operator#128 - align property name (not yet picked into CRW)

mshaposhnik · 2019-11-26T15:09:48Z

Seems working OK on prod ATM. Closing for now.

skabashnyuk · 2019-11-30T15:03:33Z

The same situation happened again

skabashnyuk · 2019-11-30T20:56:34Z

java.lang.NullPointerException: null
	at org.jgroups.util.Owner.compareTo(Owner.java:59)
	at org.jgroups.protocols.Locking$ClientLock.compareTo(Locking.java:1142)
	at org.jgroups.protocols.Locking$ClientLock.compareTo(Locking.java:942)
	at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
	at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:835)
	at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1979)
	at java.util.concurrent.ConcurrentSkipListSet.add(ConcurrentSkipListSet.java:241)
	at org.jgroups.protocols.Locking$ClientLockTable.addToPendingReleaseRequests(Locking.java:1303)
	at org.jgroups.protocols.Locking$ClientLock._unlock(Locking.java:1070)
	at org.jgroups.protocols.Locking$ClientLock.unlock(Locking.java:996)
	at org.jgroups.protocols.Locking.down(Locking.java:176)
	at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:918)
	at org.jgroups.JChannel.down(JChannel.java:627)
	at org.jgroups.blocks.locking.LockService$LockImpl.unlock(LockService.java:167)
	at org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService$UnlockerImpl.unlock(JGroupsWorkspaceLockService.java:73)
	at org.eclipse.che.commons.lang.concurrent.Unlocker.close(Unlocker.java:38)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.getStatus(WorkspaceRuntimes.java:366)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.normalizeState(WorkspaceManager.java:491)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.getWorkspaces(WorkspaceManager.java:247)
	at org.eclipse.che.api.workspace.server.WorkspaceService.getWorkspaces(WorkspaceService.java:320)
	at sun.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
	at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
	at org.everrest.core.impl.RequestDispatcher.invokeResourceMethod(RequestDispatcher.java:268)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:198)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
	at org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
	at org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
	at org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.eclipse.che.core.metrics.ApiResponseMetricFilter.doFilter(ApiResponseMetricFilter.java:46)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:53)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitalizationFilter.doFilter(KeycloakEnvironmentInitalizationFilter.java:151)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitalizationFilter$$EnhancerByGuice$$13fefe1f.CGLIB$doFilter$0(<generated>)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitalizationFilter$$EnhancerByGuice$$13fefe1f$$FastClassByGuice$$893054a1.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.redhat.che.wsmaster.deploy.DisableAuthenticationInterceptor.invoke(DisableAuthenticationInterceptor.java:51)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitalizationFilter$$EnhancerByGuice$$13fefe1f.doFilter(<generated>)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakAuthenticationFilter.doFilter(KeycloakAuthenticationFilter.java:70)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakAuthenticationFilter$$EnhancerByGuice$$d4d76de5.CGLIB$doFilter$0(<generated>)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakAuthenticationFilter$$EnhancerByGuice$$d4d76de5$$FastClassByGuice$$f08095ab.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.redhat.che.wsmaster.deploy.DisableAuthenticationInterceptor.invoke(DisableAuthenticationInterceptor.java:51)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakAuthenticationFilter$$EnhancerByGuice$$d4d76de5.doFilter(<generated>)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:108)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:188)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

skabashnyuk · 2019-11-30T20:59:15Z

Might be related https://issues.jboss.org/browse/JGRP-2299

skabashnyuk · 2019-12-09T12:36:57Z

Still reproducible with 4.1.8Final

Using embedded assembly.
06-Dec-2019 14:12:53.665 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/8.5.46
06-Dec-2019 14:12:53.669 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Sep 16 2019 18:16:19 UTC
06-Dec-2019 14:12:53.669 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 8.5.46.0
06-Dec-2019 14:12:53.669 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
06-Dec-2019 14:12:53.669 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            4.18.0-147.0.3.el8_1.x86_64
06-Dec-2019 14:12:53.669 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
06-Dec-2019 14:12:53.670 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/local/openjdk-8
06-Dec-2019 14:12:53.670 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_232-b09
06-Dec-2019 14:12:53.670 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
06-Dec-2019 14:12:53.670 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /home/user/eclipse-che/tomcat
06-Dec-2019 14:12:53.670 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /home/user/eclipse-che/tomcat
06-Dec-2019 14:12:53.671 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che//tomcat/conf/logging.properties
06-Dec-2019 14:12:53.671 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
06-Dec-2019 14:12:53.671 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxRAMFraction=2
06-Dec-2019 14:12:53.671 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseParallelGC
06-Dec-2019 14:12:53.671 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MinHeapFreeRatio=10
06-Dec-2019 14:12:53.672 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxHeapFreeRatio=20
06-Dec-2019 14:12:53.672 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:GCTimeRatio=4
06-Dec-2019 14:12:53.672 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:AdaptiveSizePolicyWeight=90
06-Dec-2019 14:12:53.672 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UnlockExperimentalVMOptions
06-Dec-2019 14:12:53.672 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseCGroupMemoryLimitForHeap
06-Dec-2019 14:12:53.673 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dsun.zip.disableMemoryMapping=true
06-Dec-2019 14:12:53.673 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms20m
06-Dec-2019 14:12:53.673 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.docker.network=bridge
06-Dec-2019 14:12:53.673 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dport.http=8080
06-Dec-2019 14:12:53.673 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.home=/home/user/eclipse-che/
06-Dec-2019 14:12:53.673 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.dir=/logs/
06-Dec-2019 14:12:53.674 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.level=INFO
06-Dec-2019 14:12:53.674 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che//tomcat/conf/tomcat-logger.xml
06-Dec-2019 14:12:53.674 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
06-Dec-2019 14:12:53.674 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
06-Dec-2019 14:12:53.675 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
06-Dec-2019 14:12:53.677 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.local.conf.dir=/home/user/eclipse-che//tomcat/conf/
06-Dec-2019 14:12:53.710 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
06-Dec-2019 14:12:53.710 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/home/user/eclipse-che//tomcat
06-Dec-2019 14:12:53.710 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/home/user/eclipse-che//tomcat
06-Dec-2019 14:12:53.711 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che//tomcat/temp
06-Dec-2019 14:12:53.832 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
06-Dec-2019 14:12:53.843 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
06-Dec-2019 14:12:53.856 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 595 ms
06-Dec-2019 14:12:53.873 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
06-Dec-2019 14:12:53.873 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.46
06-Dec-2019 14:12:53.909 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war]
06-Dec-2019 14:12:56.554 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [2,643] ms
06-Dec-2019 14:12:56.556 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2019-12-06 14:13:01,297[ost-startStop-1]  [INFO ] [.e.c.c.d.JNDIDataSourceFactory 63]   - This=org.eclipse.che.core.db.postgresql.PostgreSQLJndiDataSourceFactory@6573baa0 obj=ResourceRef[className=javax.sql.DataSource,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.eclipse.che.api.CommonJndiDataSourceFactory}] name=che Context=org.apache.naming.NamingContext@430d5f69 environment={}
2019-12-06 14:13:03,887[ost-startStop-1]  [WARN ] [c.g.i.internal.ProxyFactory 101]     - Method [public void org.eclipse.che.workspace.infrastructure.openshift.provision.RouteTlsProvisioner.provision(org.eclipse.che.workspace.infrastructure.kubernetes.environment.KubernetesEnvironment,org.eclipse.che.api.core.model.workspace.runtime.RuntimeIdentity) throws org.eclipse.che.api.workspace.server.spi.InfrastructureException] is synthetic and is being intercepted by [org.eclipse.che.core.tracing.TracingInterceptor@6d5c98b7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
2019-12-06 14:13:03,928[ost-startStop-1]  [WARN ] [c.g.i.internal.ProxyFactory 101]     - Method [public void org.eclipse.che.workspace.infrastructure.openshift.OpenShiftEnvironmentProvisioner.provision(org.eclipse.che.workspace.infrastructure.kubernetes.environment.KubernetesEnvironment,org.eclipse.che.api.core.model.workspace.runtime.RuntimeIdentity) throws org.eclipse.che.api.workspace.server.spi.InfrastructureException] is synthetic and is being intercepted by [org.eclipse.che.core.tracing.TracingInterceptor@6d5c98b7]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
2019-12-06 14:13:05,917[ost-startStop-1]  [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-76cc57475-ns9kz-48128, cluster=RemoteSubscriptionChannel, physical address=10.128.1.112:7800
-------------------------------------------------------------------
2019-12-06 14:13:09,104[ost-startStop-1]  [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-76cc57475-ns9kz-48128: no members discovered after 3150 ms: creating cluster as coordinator
2019-12-06 14:13:09,159[ost-startStop-1]  [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-76cc57475-ns9kz-34722, cluster=WorkspaceLocks, physical address=10.128.1.112:7801
-------------------------------------------------------------------
2019-12-06 14:13:12,198[ost-startStop-1]  [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-76cc57475-ns9kz-34722: no members discovered after 3032 ms: creating cluster as coordinator
2019-12-06 14:13:12,214[ost-startStop-1]  [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-76cc57475-ns9kz-49695, cluster=WorkspaceStateCache, physical address=10.128.1.112:7802
-------------------------------------------------------------------
2019-12-06 14:13:15,251[ost-startStop-1]  [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-76cc57475-ns9kz-49695: no members discovered after 3028 ms: creating cluster as coordinator
2019-12-06 14:13:15,253[ost-startStop-1]  [INFO ] [o.e.c.m.k.s.KeycloakSettings 91]     - Retrieving OpenId configuration from endpoint: http://keycloak-che.apps-crc.testing/auth/realms/che/.well-known/openid-configuration
2019-12-06 14:13:15,391[ost-startStop-1]  [INFO ] [o.e.c.m.k.s.KeycloakSettings 108]    - openid configuration = {issuer=http://keycloak-che.apps-crc.testing/auth/realms/che, authorization_endpoint=http://keycloak-che.apps-crc.testing/auth/realms/che/protocol/openid-connect/auth, token_endpoint=http://keycloak-che.apps-crc.testing/auth/realms/che/protocol/openid-connect/token, token_introspection_endpoint=http://keycloak-che.apps-crc.testing/auth/realms/che/protocol/openid-connect/token/introspect, userinfo_endpoint=http://keycloak-che.apps-crc.testing/auth/realms/che/protocol/openid-connect/userinfo, end_session_endpoint=http://keycloak-che.apps-crc.testing/auth/realms/che/protocol/openid-connect/logout, jwks_uri=http://keycloak-che.apps-crc.testing/auth/realms/che/protocol/openid-connect/certs, check_session_iframe=http://keycloak-che.apps-crc.testing/auth/realms/che/protocol/openid-connect/login-status-iframe.html, grant_types_supported=[authorization_code, implicit, refresh_token, password, client_credentials], response_types_supported=[code, none, id_token, token, id_token token, code id_token, code token, code id_token token], subject_types_supported=[public, pairwise], id_token_signing_alg_values_supported=[PS384, ES384, RS384, HS256, HS512, ES256, RS256, HS384, ES512, PS256, PS512, RS512], userinfo_signing_alg_values_supported=[PS384, ES384, RS384, HS256, HS512, ES256, RS256, HS384, ES512, PS256, PS512, RS512, none], request_object_signing_alg_values_supported=[PS384, ES384, RS384, ES256, RS256, ES512, PS256, PS512, RS512, none], response_modes_supported=[query, fragment, form_post], registration_endpoint=http://keycloak-che.apps-crc.testing/auth/realms/che/clients-registrations/openid-connect, token_endpoint_auth_methods_supported=[private_key_jwt, client_secret_basic, client_secret_post, client_secret_jwt], token_endpoint_auth_signing_alg_values_supported=[RS256], claims_supported=[aud, sub, iss, auth_time, name, given_name, family_name, preferred_username, email], claim_types_supported=[normal], claims_parameter_supported=false, scopes_supported=[openid, microprofile-jwt, web-origins, roles, phone, address, email, profile, offline_access], request_parameter_supported=true, request_uri_parameter_supported=true, code_challenge_methods_supported=[plain, S256], tls_client_certificate_bound_access_tokens=true, introspection_endpoint=http://keycloak-che.apps-crc.testing/auth/realms/che/protocol/openid-connect/token/introspect}
2019-12-06 14:13:15,705[ost-startStop-1]  [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:postgresql://postgres:5432/dbche (PostgreSQL 9.6)
2019-12-06 14:13:15,760[ost-startStop-1]  [INFO ] [o.f.c.i.util.VersionPrinter 44]      - Flyway 4.2.0 by Boxfuse
2019-12-06 14:13:15,767[ost-startStop-1]  [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:postgresql://postgres:5432/dbche (PostgreSQL 9.6)
2019-12-06 14:13:15,828[ost-startStop-1]  [INFO ] [i.f.CustomSqlMigrationResolver 158]  - Searching for sql scripts in locations [classpath:che-schema]
2019-12-06 14:13:15,922[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbValidate 44]       - Successfully validated 55 migrations (execution time 00:00.103s)
2019-12-06 14:13:15,954[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Current version of schema "public": 7.6.0.1
2019-12-06 14:13:15,956[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Schema "public" is up to date. No migration necessary.
2019-12-06 14:13:17,658[ost-startStop-1]  [INFO ] [o.j.p.kubernetes.KUBE_PING 131]      - namespace che set; clustering enabled

-------------------------------------------------------------------
GMS: address=che-76cc57475-ns9kz-3382, cluster=EclipseLinkCommandChannel, physical address=10.128.1.112:7803
-------------------------------------------------------------------
2019-12-06 14:13:20,718[ost-startStop-1]  [INFO ] [o.jgroups.protocols.pbcast.GMS 125]  - che-76cc57475-ns9kz-3382: no members discovered after 3037 ms: creating cluster as coordinator
2019-12-06 14:13:20,954[ost-startStop-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 173]   - Configured factories for environments: '[openshift, kubernetes, no-environment]'
2019-12-06 14:13:20,955[ost-startStop-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 174]   - Registered infrastructure 'openshift'
2019-12-06 14:13:21,019[ost-startStop-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 648]   - Infrastructure is tracking 0 active runtimes
2019-12-06 14:13:21,077[ost-startStop-1]  [INFO ] [o.e.c.a.c.u.ApiInfoLogInformer 36]   - Eclipse Che Api Core: Build info '7.6.0-SNAPSHOT' scmRevision 'a4a91522d041c0e37ce271ead51fadf3b4959413' implementationVersion '7.6.0-SNAPSHOT'
2019-12-06 14:13:21,182[ost-startStop-1]  [INFO ] [o.e.c.c.metrics.MetricsServer 46]    - Metrics server started at port 8087 successfully 
06-Dec-2019 14:13:21.555 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/api.war] has finished in [24,999] ms
06-Dec-2019 14:13:21.556 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war]
06-Dec-2019 14:13:21.989 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
06-Dec-2019 14:13:21.992 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war] has finished in [436] ms
06-Dec-2019 14:13:21.993 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war]
06-Dec-2019 14:13:23.105 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war] has finished in [1,112] ms
06-Dec-2019 14:13:23.106 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
06-Dec-2019 14:13:23.155 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [49] ms
06-Dec-2019 14:13:23.156 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war]
06-Dec-2019 14:13:23.184 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war] has finished in [28] ms
06-Dec-2019 14:13:23.191 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
06-Dec-2019 14:13:23.200 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 29344 ms
2019-12-06 14:29:56,984[nio-8080-exec-3]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 466]   - Starting workspace 'admin/wksp-7qmo' with id 'workspacekt17ac39c018bq8o' by user 'admin'
2019-12-06 14:29:58,516[nio-8080-exec-6]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 466]   - Starting workspace 'admin/wksp-8pkx' with id 'workspace6d28o0jymdr1kuxf' by user 'admin'
2019-12-06 14:31:04,043[/172.30.0.1/...]  [ERROR] [.w.i.k.n.KubernetesDeployments 500]  - Failed to parse last timestamp of the event. Cause: For input string: "". Event: PodEvent{podName='workspace6d28o0jymdr1kuxf.go-cli-967ccb7f6-l577d', containerName='null', reason='Pulled', message='Successfully pulled image "docker.io/eclipse/che-theia-endpoint-runtime-binary:next"', creationTimestamp='2019-12-06T14:31:04Z', lastTimestamp='2019-12-06T14:31:04Z'}
2019-12-06 14:31:28,413[/172.30.0.1/...]  [ERROR] [.w.i.k.n.KubernetesDeployments 500]  - Failed to parse last timestamp of the event. Cause: For input string: "". Event: PodEvent{podName='workspace6d28o0jymdr1kuxf.go-cli-967ccb7f6-l577d', containerName='go-cli', reason='Started', message='Started container go-cli', creationTimestamp='2019-12-06T14:31:28Z', lastTimestamp='2019-12-06T14:31:28Z'}
2019-12-06 14:32:40,181[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 870]   - Workspace 'admin:wksp-7qmo' with id 'workspacekt17ac39c018bq8o' started by user 'admin'
2019-12-06 14:32:43,392[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 870]   - Workspace 'admin:wksp-8pkx' with id 'workspace6d28o0jymdr1kuxf' started by user 'admin'
2019-12-06 14:34:19,770[nio-8080-exec-8]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 527]   - Workspace 'admin/wksp-8pkx' with id 'workspace6d28o0jymdr1kuxf' is stopping by user 'admin'
2019-12-06 14:34:20,580[nio-8080-exec-9]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 527]   - Workspace 'admin/wksp-7qmo' with id 'workspacekt17ac39c018bq8o' is stopping by user 'admin'
2019-12-06 14:34:24,865[aceSharedPool-2]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 935]   - Workspace 'admin/wksp-8pkx' with id 'workspace6d28o0jymdr1kuxf' is stopped by user 'admin'
2019-12-06 14:34:24,981[aceSharedPool-3]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 935]   - Workspace 'admin/wksp-7qmo' with id 'workspacekt17ac39c018bq8o' is stopped by user 'admin'
2019-12-06 14:34:26,579[nio-8080-exec-1]  [WARN ] [a.c.w.i.BasicWebSocketEndpoint 135]  - Closing unidentified session
2019-12-06 14:38:20,493[nio-8080-exec-3]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 466]   - Starting workspace 'admin/wksp-8pkx' with id 'workspace6d28o0jymdr1kuxf' by user 'admin'
2019-12-06 14:39:22,628[aceSharedPool-4]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 870]   - Workspace 'admin:wksp-8pkx' with id 'workspace6d28o0jymdr1kuxf' started by user 'admin'
2019-12-06 14:54:35,395[nio-8080-exec-3]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 527]   - Workspace 'admin/wksp-8pkx' with id 'workspace6d28o0jymdr1kuxf' is stopping by user 'admin'
2019-12-06 14:54:37,102[aceSharedPool-5]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 935]   - Workspace 'admin/wksp-8pkx' with id 'workspace6d28o0jymdr1kuxf' is stopped by user 'admin'
2019-12-06 14:55:54,595[nio-8080-exec-8]  [ERROR] [o.e.c.servlet.EverrestServlet 71]    - java.lang.NullPointerException
org.everrest.core.UnhandledException: java.lang.NullPointerException
	at org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:73)
	at org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
	at org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.eclipse.che.core.metrics.ApiResponseMetricFilter.doFilter(ApiResponseMetricFilter.java:46)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:125)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:88)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:76)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:189)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
	at org.jgroups.util.Owner.compareTo(Owner.java:59)
	at org.jgroups.protocols.Locking$ClientLock.compareTo(Locking.java:1163)
	at org.jgroups.protocols.Locking$ClientLock.compareTo(Locking.java:942)
	at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
	at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:835)
	at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1979)
	at java.util.concurrent.ConcurrentSkipListSet.add(ConcurrentSkipListSet.java:241)
	at org.jgroups.protocols.Locking$ClientLockTable.addToPendingReleaseRequests(Locking.java:1323)
	at org.jgroups.protocols.Locking$ClientLock._unlock(Locking.java:1070)
	at org.jgroups.protocols.Locking$ClientLock.unlock(Locking.java:996)
	at org.jgroups.protocols.Locking.down(Locking.java:176)
	at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:928)
	at org.jgroups.JChannel.down(JChannel.java:627)
	at org.jgroups.blocks.locking.LockService$LockImpl.unlock(LockService.java:167)
	at org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService$UnlockerImpl.unlock(JGroupsWorkspaceLockService.java:73)
	at org.eclipse.che.commons.lang.concurrent.Unlocker.close(Unlocker.java:38)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.$closeResource(WorkspaceRuntimes.java:314)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.injectRuntime(WorkspaceRuntimes.java:314)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.normalizeState(WorkspaceManager.java:520)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.getWorkspace(WorkspaceManager.java:207)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.WorkspacePermissionsFilter.filter(WorkspacePermissionsFilter.java:153)
	at org.eclipse.che.everrest.CheMethodInvokerFilter.accept(CheMethodInvokerFilter.java:40)
	at org.everrest.core.impl.method.DefaultMethodInvoker.beforeInvokeMethod(DefaultMethodInvoker.java:134)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:59)
	at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
	at org.everrest.core.impl.RequestDispatcher.invokeSubResourceMethod(RequestDispatcher.java:297)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:233)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
	at org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
	... 42 common frames omitted
2019-12-06 14:55:54,646[nio-8080-exec-8]  [ERROR] [o.a.c.c.C.[.[.[/api].[default] 182]  - Servlet.service() for servlet [default] in context with path [/api] threw exception [java.lang.NullPointerException] with root cause
java.lang.NullPointerException: null
	at org.jgroups.util.Owner.compareTo(Owner.java:59)
	at org.jgroups.protocols.Locking$ClientLock.compareTo(Locking.java:1163)
	at org.jgroups.protocols.Locking$ClientLock.compareTo(Locking.java:942)
	at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
	at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:835)
	at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1979)
	at java.util.concurrent.ConcurrentSkipListSet.add(ConcurrentSkipListSet.java:241)
	at org.jgroups.protocols.Locking$ClientLockTable.addToPendingReleaseRequests(Locking.java:1323)
	at org.jgroups.protocols.Locking$ClientLock._unlock(Locking.java:1070)
	at org.jgroups.protocols.Locking$ClientLock.unlock(Locking.java:996)
	at org.jgroups.protocols.Locking.down(Locking.java:176)
	at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:928)
	at org.jgroups.JChannel.down(JChannel.java:627)
	at org.jgroups.blocks.locking.LockService$LockImpl.unlock(LockService.java:167)
	at org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService$UnlockerImpl.unlock(JGroupsWorkspaceLockService.java:73)
	at org.eclipse.che.commons.lang.concurrent.Unlocker.close(Unlocker.java:38)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.$closeResource(WorkspaceRuntimes.java:314)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.injectRuntime(WorkspaceRuntimes.java:314)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.normalizeState(WorkspaceManager.java:520)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.getWorkspace(WorkspaceManager.java:207)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.WorkspacePermissionsFilter.filter(WorkspacePermissionsFilter.java:153)
	at org.eclipse.che.everrest.CheMethodInvokerFilter.accept(CheMethodInvokerFilter.java:40)
	at org.everrest.core.impl.method.DefaultMethodInvoker.beforeInvokeMethod(DefaultMethodInvoker.java:134)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:59)
	at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
	at org.everrest.core.impl.RequestDispatcher.invokeSubResourceMethod(RequestDispatcher.java:297)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:233)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
	at org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
	at org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
	at org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
	at org.eclipse.che.core.metrics.ApiResponseMetricFilter.doFilter(ApiResponseMetricFilter.java:46)
	at org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
	at org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:125)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:88)
	at org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:76)
	at org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
	at io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:189)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

Received SIGTERM
Stopping Che server running on localhost:8080
06-Dec-2019 15:05:24.282 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
06-Dec-2019 15:05:24.288 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]
06-Dec-2019 15:05:24.299 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
2019-12-06 15:05:24,486[ost-startStop-2]  [INFO ] [o.a.c.core.StandardWrapper 180]      - Waiting for [6] instance(s) to be deallocated for Servlet [default]
2019-12-06 15:05:25,489[ost-startStop-2]  [INFO ] [o.a.c.core.StandardWrapper 180]      - Waiting for [6] instance(s) to be deallocated for Servlet [default]
2019-12-06 15:05:26,492[ost-startStop-2]  [INFO ] [o.a.c.core.StandardWrapper 180]      - Waiting for [6] instance(s) to be deallocated for Servlet [default]
06-Dec-2019 15:05:27.287 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [api] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Dec-2019 15:05:27.291 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [jaeger.RemoteReporter-QueueProcessor] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
 io.jaegertracing.internal.reporters.RemoteReporter$QueueProcessor.run(RemoteReporter.java:173)
 java.lang.Thread.run(Thread.java:748)
06-Dec-2019 15:05:27.293 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [jaeger.RemoteReporter-FlushTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.util.TimerThread.mainLoop(Timer.java:552)
 java.util.TimerThread.run(Timer.java:505)
06-Dec-2019 15:05:27.294 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [ServerProbesTimeouts] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:502)
 java.util.TimerThread.mainLoop(Timer.java:526)
 java.util.TimerThread.run(Timer.java:505)
06-Dec-2019 15:05:27.296 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:[
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:502)
 org.jgroups.protocols.Locking$ClientLock.acquire(Locking.java:1048)
 org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970)
 org.jgroups.protocols.Locking.down(Locking.java:155)
 org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:928)
 org.jgroups.JChannel.down(JChannel.java:627)
 org.jgroups.blocks.locking.LockService$LockImpl.lock(LockService.java:109)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.writeLock(JGroupsWorkspaceLockService.java:60)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.readLock(JGroupsWorkspaceLockService.java:54)
 org.eclipse.che.api.workspace.server.WorkspaceRuntimes.getStatus(WorkspaceRuntimes.java:381)
 org.eclipse.che.api.workspace.server.WorkspaceManager.normalizeState(WorkspaceManager.java:522)
 org.eclipse.che.api.workspace.server.WorkspaceManager.getWorkspaces(WorkspaceManager.java:251)
 org.eclipse.che.api.workspace.server.WorkspaceService.getWorkspaces(WorkspaceService.java:342)
 sun.reflect.GeneratedMethodAccessor473.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 java.lang.reflect.Method.invoke(Method.java:498)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
 org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
 org.everrest.core.impl.RequestDispatcher.invokeResourceMethod(RequestDispatcher.java:268)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:198)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
 org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
 org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
 org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
 com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
 com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
 com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
 com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
 org.eclipse.che.core.metrics.ApiResponseMetricFilter.doFilter(ApiResponseMetricFilter.java:46)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:125)
 org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:88)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:76)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:189)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
 com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
 org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
 org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
 org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
 org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
 org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
 org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 java.lang.Thread.run(Thread.java:748)]
06-Dec-2019 15:05:27.298 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:[
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:502)
 org.jgroups.protocols.Locking$ClientLock.acquire(Locking.java:1048)
 org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970)
 org.jgroups.protocols.Locking.down(Locking.java:155)
 org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:928)
 org.jgroups.JChannel.down(JChannel.java:627)
 org.jgroups.blocks.locking.LockService$LockImpl.lock(LockService.java:109)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.writeLock(JGroupsWorkspaceLockService.java:60)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.readLock(JGroupsWorkspaceLockService.java:54)
 org.eclipse.che.api.workspace.server.WorkspaceRuntimes.getStatus(WorkspaceRuntimes.java:381)
 org.eclipse.che.api.workspace.server.WorkspaceManager.normalizeState(WorkspaceManager.java:522)
 org.eclipse.che.api.workspace.server.WorkspaceManager.getWorkspaces(WorkspaceManager.java:251)
 org.eclipse.che.api.workspace.server.WorkspaceService.getWorkspaces(WorkspaceService.java:342)
 sun.reflect.GeneratedMethodAccessor473.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 java.lang.reflect.Method.invoke(Method.java:498)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
 org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
 org.everrest.core.impl.RequestDispatcher.invokeResourceMethod(RequestDispatcher.java:268)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:198)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
 org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
 org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
 org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
 com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
 com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
 com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
 com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
 org.eclipse.che.core.metrics.ApiResponseMetricFilter.doFilter(ApiResponseMetricFilter.java:46)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:125)
 org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:88)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:76)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:189)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
 com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
 org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
 org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
 org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
 org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
 org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
 org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 java.lang.Thread.run(Thread.java:748)]
06-Dec-2019 15:05:27.300 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:[
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:502)
 org.jgroups.protocols.Locking$ClientLock.acquire(Locking.java:1048)
 org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970)
 org.jgroups.protocols.Locking.down(Locking.java:155)
 org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:928)
 org.jgroups.JChannel.down(JChannel.java:627)
 org.jgroups.blocks.locking.LockService$LockImpl.lock(LockService.java:109)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.writeLock(JGroupsWorkspaceLockService.java:60)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.readLock(JGroupsWorkspaceLockService.java:54)
 org.eclipse.che.api.workspace.server.WorkspaceRuntimes.getStatus(WorkspaceRuntimes.java:381)
 org.eclipse.che.api.workspace.server.WorkspaceManager.normalizeState(WorkspaceManager.java:522)
 org.eclipse.che.api.workspace.server.WorkspaceManager.getWorkspaces(WorkspaceManager.java:251)
 org.eclipse.che.api.workspace.server.WorkspaceService.getWorkspaces(WorkspaceService.java:342)
 sun.reflect.GeneratedMethodAccessor473.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 java.lang.reflect.Method.invoke(Method.java:498)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
 org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
 org.everrest.core.impl.RequestDispatcher.invokeResourceMethod(RequestDispatcher.java:268)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:198)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
 org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
 org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
 org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
 com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
 com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
 com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
 com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
 org.eclipse.che.core.metrics.ApiResponseMetricFilter.doFilter(ApiResponseMetricFilter.java:46)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:125)
 org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:88)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:76)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:189)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
 com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
 org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
 org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
 org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
 org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
 org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
 org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 java.lang.Thread.run(Thread.java:748)]
06-Dec-2019 15:05:27.302 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:[
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:502)
 org.jgroups.protocols.Locking$ClientLock.acquire(Locking.java:1048)
 org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970)
 org.jgroups.protocols.Locking.down(Locking.java:155)
 org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:928)
 org.jgroups.JChannel.down(JChannel.java:627)
 org.jgroups.blocks.locking.LockService$LockImpl.lock(LockService.java:109)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.writeLock(JGroupsWorkspaceLockService.java:60)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.readLock(JGroupsWorkspaceLockService.java:54)
 org.eclipse.che.api.workspace.server.WorkspaceRuntimes.getStatus(WorkspaceRuntimes.java:381)
 org.eclipse.che.api.workspace.server.WorkspaceManager.normalizeState(WorkspaceManager.java:522)
 org.eclipse.che.api.workspace.server.WorkspaceManager.getWorkspaces(WorkspaceManager.java:251)
 org.eclipse.che.api.workspace.server.WorkspaceService.getWorkspaces(WorkspaceService.java:342)
 sun.reflect.GeneratedMethodAccessor473.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 java.lang.reflect.Method.invoke(Method.java:498)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
 org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
 org.everrest.core.impl.RequestDispatcher.invokeResourceMethod(RequestDispatcher.java:268)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:198)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
 org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
 org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
 org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
 com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
 com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
 com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
 com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
 org.eclipse.che.core.metrics.ApiResponseMetricFilter.doFilter(ApiResponseMetricFilter.java:46)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:125)
 org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:88)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:76)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:189)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
 com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
 org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
 org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
 org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
 org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
 org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
 org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 java.lang.Thread.run(Thread.java:748)]
06-Dec-2019 15:05:27.304 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:[
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:502)
 org.jgroups.protocols.Locking$ClientLock.acquire(Locking.java:1048)
 org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970)
 org.jgroups.protocols.Locking.down(Locking.java:155)
 org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:928)
 org.jgroups.JChannel.down(JChannel.java:627)
 org.jgroups.blocks.locking.LockService$LockImpl.lock(LockService.java:109)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.writeLock(JGroupsWorkspaceLockService.java:60)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.readLock(JGroupsWorkspaceLockService.java:54)
 org.eclipse.che.api.workspace.server.WorkspaceRuntimes.getStatus(WorkspaceRuntimes.java:381)
 org.eclipse.che.api.workspace.server.WorkspaceManager.normalizeState(WorkspaceManager.java:522)
 org.eclipse.che.api.workspace.server.WorkspaceManager.getWorkspaces(WorkspaceManager.java:251)
 org.eclipse.che.api.workspace.server.WorkspaceService.getWorkspaces(WorkspaceService.java:342)
 sun.reflect.GeneratedMethodAccessor473.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 java.lang.reflect.Method.invoke(Method.java:498)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
 org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
 org.everrest.core.impl.RequestDispatcher.invokeResourceMethod(RequestDispatcher.java:268)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:198)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
 org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
 org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
 org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
 com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
 com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
 com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
 com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
 org.eclipse.che.core.metrics.ApiResponseMetricFilter.doFilter(ApiResponseMetricFilter.java:46)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:125)
 org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:88)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:76)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:189)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
 com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
 org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
 org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
 org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
 org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
 org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
 org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 java.lang.Thread.run(Thread.java:748)]
06-Dec-2019 15:05:27.306 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:[
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:502)
 org.jgroups.protocols.Locking$ClientLock.acquire(Locking.java:1048)
 org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970)
 org.jgroups.protocols.Locking.down(Locking.java:155)
 org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:928)
 org.jgroups.JChannel.down(JChannel.java:627)
 org.jgroups.blocks.locking.LockService$LockImpl.lock(LockService.java:109)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.writeLock(JGroupsWorkspaceLockService.java:60)
 org.eclipse.che.multiuser.api.distributed.lock.JGroupsWorkspaceLockService.readLock(JGroupsWorkspaceLockService.java:54)
 org.eclipse.che.api.workspace.server.WorkspaceRuntimes.getStatus(WorkspaceRuntimes.java:381)
 org.eclipse.che.api.workspace.server.WorkspaceManager.normalizeState(WorkspaceManager.java:522)
 org.eclipse.che.api.workspace.server.WorkspaceManager.getWorkspaces(WorkspaceManager.java:251)
 org.eclipse.che.api.workspace.server.WorkspaceService.getWorkspaces(WorkspaceService.java:342)
 sun.reflect.GeneratedMethodAccessor473.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 java.lang.reflect.Method.invoke(Method.java:498)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
 org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
 org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
 org.everrest.core.impl.RequestDispatcher.invokeResourceMethod(RequestDispatcher.java:268)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:198)
 org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
 org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
 org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
 org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
 com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
 com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
 com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
 com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
 org.eclipse.che.core.metrics.ApiResponseMetricFilter.doFilter(ApiResponseMetricFilter.java:46)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:125)
 org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:88)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:76)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:189)
 com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
 com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
 org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
 org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
 org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
 org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
 org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
 org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 java.lang.Thread.run(Thread.java:748)]
06-Dec-2019 15:05:27.308 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [master-websocket-major-endpoint-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:74)
 org.eclipse.che.commons.observability.CountedThreadFactory$$Lambda$77/951590589.run(Unknown Source)
 java.lang.Thread.run(Thread.java:748)
06-Dec-2019 15:05:27.310 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [master-websocket-major-endpoint-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:74)
 org.eclipse.che.commons.observability.CountedThreadFactory$$Lambda$77/951590589.run(Unknown Source)
 java.lang.Thread.run(Thread.java:748)
06-Dec-2019 15:05:27.312 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [master-websocket-major-endpoint-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:74)
 org.eclipse.che.commons.observability.CountedThreadFactory$$Lambda$77/951590589.run(Unknown Source)
 java.lang.Thread.run(Thread.java:748)
06-Dec-2019 15:05:27.313 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [master-websocket-major-endpoint-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:74)
 org.eclipse.che.commons.observability.CountedThreadFactory$$Lambda$77/951590589.run(Unknown Source)
 java.lang.Thread.run(Thread.java:748)
06-Dec-2019 15:05:27.315 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [master-websocket-major-endpoint-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 org.eclipse.che.commons.observability.CountedThreadFactory.lambda$newThread$0(CountedThreadFactory.java:74)
 org.eclipse.che.commons.observability.CountedThreadFactory$$Lambda$77/951590589.run(Unknown Source)
 java.lang.Thread.run(Thread.java:748)
06-Dec-2019 15:05:27.315 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6c92655b]) and a value of type [com.google.inject.servlet.GuiceFilter.Context] (value [com.google.inject.servlet.GuiceFilter$Context@2b6ad1e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.316 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@62e6eabc]) and a value of type [io.opentracing.util.ThreadLocalScope] (value [io.opentracing.util.ThreadLocalScope@15751d82]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.316 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6a320aca]) and a value of type [org.everrest.core.impl.EnvironmentContext] (value [{interface javax.servlet.http.HttpServletResponse=org.apache.catalina.connector.ResponseFacade@5ddb84cf, interface javax.servlet.ServletContext=org.apache.catalina.core.ApplicationContextFacade@83d36c2, interface javax.servlet.http.HttpServletRequest=com.google.inject.servlet.ServletDefinition$2@3599a34a, class org.everrest.core.tools.ErrorPages=org.everrest.core.tools.ErrorPages@4f079b69, class org.everrest.core.tools.WebApplicationDeclaredRoles=org.everrest.core.tools.WebApplicationDeclaredRoles@4e7b83ba, interface javax.servlet.ServletConfig=com.google.inject.servlet.ServletDefinition$1@2f112791}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.316 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@576a42ca]) and a value of type [org.everrest.core.ApplicationContext] (value [org.everrest.core.ApplicationContext@2445479d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.316 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@10bae214]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.317 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6c92655b]) and a value of type [com.google.inject.servlet.GuiceFilter.Context] (value [com.google.inject.servlet.GuiceFilter$Context@15c38edb]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.317 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@62e6eabc]) and a value of type [io.opentracing.util.ThreadLocalScope] (value [io.opentracing.util.ThreadLocalScope@1b345a51]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.317 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6a320aca]) and a value of type [org.everrest.core.impl.EnvironmentContext] (value [{interface javax.servlet.http.HttpServletResponse=org.apache.catalina.connector.ResponseFacade@260f494d, interface javax.servlet.ServletContext=org.apache.catalina.core.ApplicationContextFacade@83d36c2, interface javax.servlet.http.HttpServletRequest=com.google.inject.servlet.ServletDefinition$2@6162a5da, class org.everrest.core.tools.ErrorPages=org.everrest.core.tools.ErrorPages@4f079b69, class org.everrest.core.tools.WebApplicationDeclaredRoles=org.everrest.core.tools.WebApplicationDeclaredRoles@4e7b83ba, interface javax.servlet.ServletConfig=com.google.inject.servlet.ServletDefinition$1@2f112791}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.317 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@576a42ca]) and a value of type [org.everrest.core.ApplicationContext] (value [org.everrest.core.ApplicationContext@68de5a7b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.317 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@510173e3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.318 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6c92655b]) and a value of type [com.google.inject.servlet.GuiceFilter.Context] (value [com.google.inject.servlet.GuiceFilter$Context@d180daf]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.318 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6a320aca]) and a value of type [org.everrest.core.impl.EnvironmentContext] (value [{interface javax.servlet.http.HttpServletResponse=org.apache.catalina.connector.ResponseFacade@79a025a3, interface javax.servlet.ServletContext=org.apache.catalina.core.ApplicationContextFacade@83d36c2, interface javax.servlet.http.HttpServletRequest=com.google.inject.servlet.ServletDefinition$2@63b7a56e, class org.everrest.core.tools.ErrorPages=org.everrest.core.tools.ErrorPages@4f079b69, class org.everrest.core.tools.WebApplicationDeclaredRoles=org.everrest.core.tools.WebApplicationDeclaredRoles@4e7b83ba, interface javax.servlet.ServletConfig=com.google.inject.servlet.ServletDefinition$1@2f112791}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.318 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@39f35c90]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.318 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@62e6eabc]) and a value of type [io.opentracing.util.ThreadLocalScope] (value [io.opentracing.util.ThreadLocalScope@74cb4b74]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.318 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@576a42ca]) and a value of type [org.everrest.core.ApplicationContext] (value [org.everrest.core.ApplicationContext@c3ee573]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.319 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6c92655b]) and a value of type [com.google.inject.servlet.GuiceFilter.Context] (value [com.google.inject.servlet.GuiceFilter$Context@527cbd4f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.319 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@62e6eabc]) and a value of type [io.opentracing.util.ThreadLocalScope] (value [io.opentracing.util.ThreadLocalScope@71487c4c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.337 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6a320aca]) and a value of type [org.everrest.core.impl.EnvironmentContext] (value [{interface javax.servlet.http.HttpServletResponse=org.apache.catalina.connector.ResponseFacade@5f7609fa, interface javax.servlet.ServletContext=org.apache.catalina.core.ApplicationContextFacade@83d36c2, interface javax.servlet.http.HttpServletRequest=com.google.inject.servlet.ServletDefinition$2@24187a1a, class org.everrest.core.tools.ErrorPages=org.everrest.core.tools.ErrorPages@4f079b69, class org.everrest.core.tools.WebApplicationDeclaredRoles=org.everrest.core.tools.WebApplicationDeclaredRoles@4e7b83ba, interface javax.servlet.ServletConfig=com.google.inject.servlet.ServletDefinition$1@2f112791}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.337 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@576a42ca]) and a value of type [org.everrest.core.ApplicationContext] (value [org.everrest.core.ApplicationContext@13393d13]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.337 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@21a18924]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.338 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6c92655b]) and a value of type [com.google.inject.servlet.GuiceFilter.Context] (value [com.google.inject.servlet.GuiceFilter$Context@4ef76d33]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.338 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@62e6eabc]) and a value of type [io.opentracing.util.ThreadLocalScope] (value [io.opentracing.util.ThreadLocalScope@2958065d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.338 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6a320aca]) and a value of type [org.everrest.core.impl.EnvironmentContext] (value [{interface javax.servlet.http.HttpServletResponse=org.apache.catalina.connector.ResponseFacade@735377c0, interface javax.servlet.ServletContext=org.apache.catalina.core.ApplicationContextFacade@83d36c2, interface javax.servlet.http.HttpServletRequest=com.google.inject.servlet.ServletDefinition$2@5e962827, class org.everrest.core.tools.ErrorPages=org.everrest.core.tools.ErrorPages@4f079b69, class org.everrest.core.tools.WebApplicationDeclaredRoles=org.everrest.core.tools.WebApplicationDeclaredRoles@4e7b83ba, interface javax.servlet.ServletConfig=com.google.inject.servlet.ServletDefinition$1@2f112791}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.339 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@576a42ca]) and a value of type [org.everrest.core.ApplicationContext] (value [org.everrest.core.ApplicationContext@2bfd5833]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.339 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@427ed9ff]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.339 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6c92655b]) and a value of type [com.google.inject.servlet.GuiceFilter.Context] (value [com.google.inject.servlet.GuiceFilter$Context@2afb5b07]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.339 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@62e6eabc]) and a value of type [io.opentracing.util.ThreadLocalScope] (value [io.opentracing.util.ThreadLocalScope@4543896f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.340 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6a320aca]) and a value of type [org.everrest.core.impl.EnvironmentContext] (value [{interface javax.servlet.http.HttpServletResponse=org.apache.catalina.connector.ResponseFacade@4be6d34, interface javax.servlet.ServletContext=org.apache.catalina.core.ApplicationContextFacade@83d36c2, interface javax.servlet.http.HttpServletRequest=com.google.inject.servlet.ServletDefinition$2@303bfd7d, class org.everrest.core.tools.ErrorPages=org.everrest.core.tools.ErrorPages@4f079b69, class org.everrest.core.tools.WebApplicationDeclaredRoles=org.everrest.core.tools.WebApplicationDeclaredRoles@4e7b83ba, interface javax.servlet.ServletConfig=com.google.inject.servlet.ServletDefinition$1@2f112791}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.340 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@576a42ca]) and a value of type [org.everrest.core.ApplicationContext] (value [org.everrest.core.ApplicationContext@4068bb1e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.340 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@19b94a8d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.340 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@6e994343]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.340 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@1a68bf8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.341 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@4136099]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.341 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@57adbfb4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.341 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@738a7d59]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.342 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@46c6c19b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.342 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@59ce6672]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.342 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [api] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@36b166fc]) and a value of type [org.eclipse.che.commons.env.EnvironmentContext] (value [org.eclipse.che.commons.env.EnvironmentContext@632432e0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
06-Dec-2019 15:05:27.373 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
06-Dec-2019 15:05:32.375 WARNING [main] org.apache.tomcat.util.net.AbstractEndpoint.shutdownExecutor The executor associated with thread pool [http-nio-8080] has not fully shutdown. Some application threads may still be running.
06-Dec-2019 15:05:32.376 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]
06-Dec-2019 15:05:32.384 INFO [Thread-5] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [io.jaegertracing.internal.reporters.RemoteReporter$CloseCommand]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
	java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [io.jaegertracing.internal.reporters.RemoteReporter$CloseCommand]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
		at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1380)
		at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1368)
		at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1220)
		at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1182)
		at io.jaegertracing.internal.reporters.RemoteReporter.close(RemoteReporter.java:91)
		at io.jaegertracing.internal.JaegerTracer.close(JaegerTracer.java:225)
		at io.jaegertracing.internal.JaegerTracer$1.run(JaegerTracer.java:135)

skabashnyuk · 2019-12-17T08:31:57Z

Work stopped. Waiting for inputs from jgroups authors https://issues.redhat.com/projects/JGRP/issues/JGRP-2423

nickboldt · 2020-01-06T12:53:15Z

On Dec 23, Bela Ban said:

OK, so I think I may have fixed this... I basically made sure that even null fields don't lead to NPEs.
Try out JGroups-4.1.9-SNAPSHOT and let me know if this works. If not, please feel free to reopen.

-- https://issues.redhat.com/browse/JGRP-2423?focusedCommentId=13939488&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13939488

skabashnyuk added kind/bug Outline of a bug - must adhere to the bug report template. severity/P1 Has a major impact to usage or development of the system. team/platform labels Nov 19, 2019

skabashnyuk added this to the 7.5.0 milestone Nov 19, 2019

skabashnyuk assigned mshaposhnik Nov 19, 2019

skabashnyuk mentioned this issue Nov 20, 2019

Platform-2019-11-26 (Sprint: 175) #15115

Closed

12 tasks

mshaposhnik closed this as completed Nov 26, 2019

skabashnyuk reopened this Nov 30, 2019

skabashnyuk modified the milestones: 7.5.0, 7.6.0 Nov 30, 2019

skabashnyuk self-assigned this Nov 30, 2019

ibuziuk mentioned this issue Dec 2, 2019

Release 7.5.1 #15372

Closed

18 tasks

mshaposhnik mentioned this issue Dec 2, 2019

Update to newest JGroups version due to locking bug eclipse-che/che-parent#141

Merged

ibuziuk mentioned this issue Dec 2, 2019

Release 7.3.3 #15378

Closed

21 tasks

skabashnyuk mentioned this issue Dec 2, 2019

Replace jgroups 4.1.0 with 4.1.8 redhat-developer/rh-che#1720

Merged

skabashnyuk closed this as completed Dec 9, 2019

skabashnyuk reopened this Dec 9, 2019

ibuziuk mentioned this issue Dec 9, 2019

Replace jgroups 4.1.0 with 4.1.8 redhat-developer/rh-che#1728

Merged

ibuziuk mentioned this issue Dec 9, 2019

[DO NOT MERGE] Rolling back to 7.3.2 due to the jgroups issue redhat-developer/rh-che#1729

Merged

skabashnyuk added the status/blocked Issue that can’t be moved forward. Must include a comment on the reason for the blockage. label Dec 17, 2019

skabashnyuk removed this from the 7.6.0 milestone Dec 17, 2019

This was referenced Dec 17, 2019

Release 7.3.4 #15502

Closed

Update Hosted Che to the latest 7.5.1 upstream version #15329

Closed

skabashnyuk mentioned this issue Dec 18, 2019

Platform-2019-12-17 (Sprint: 176) #15341

Closed

9 tasks

skabashnyuk added this to the 7.7.0 milestone Dec 18, 2019

skabashnyuk modified the milestones: 7.7.0, Backlog - Platform Dec 18, 2019

mshaposhnik mentioned this issue Jan 9, 2020

Update JGroups to 4.1.9.Final eclipse-che/che-parent#149

Merged

mshaposhnik closed this as completed Jan 9, 2020

skabashnyuk modified the milestones: Backlog - Platform, 7.7.0 Jan 9, 2020

skabashnyuk removed the status/blocked Issue that can’t be moved forward. Must include a comment on the reason for the blockage. label Jan 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple threads blocked in org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970) #15231

Multiple threads blocked in org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970) #15231

skabashnyuk commented Nov 19, 2019 •

edited

Loading

mshaposhnik commented Nov 25, 2019

mshaposhnik commented Nov 26, 2019 •

edited

Loading

skabashnyuk commented Nov 30, 2019 •

edited

Loading

skabashnyuk commented Nov 30, 2019

skabashnyuk commented Nov 30, 2019

skabashnyuk commented Dec 9, 2019

skabashnyuk commented Dec 17, 2019

nickboldt commented Jan 6, 2020

Multiple threads blocked in org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970) #15231

Multiple threads blocked in org.jgroups.protocols.Locking$ClientLock.lock(Locking.java:970) #15231

Comments

skabashnyuk commented Nov 19, 2019 • edited Loading

Describe the bug

Che version

Steps to reproduce

Expected behavior

Runtime

Screenshots

Installation method

Environment

Additional context

mshaposhnik commented Nov 25, 2019

mshaposhnik commented Nov 26, 2019 • edited Loading

skabashnyuk commented Nov 30, 2019 • edited Loading

skabashnyuk commented Nov 30, 2019

skabashnyuk commented Nov 30, 2019

skabashnyuk commented Dec 9, 2019

skabashnyuk commented Dec 17, 2019

nickboldt commented Jan 6, 2020

skabashnyuk commented Nov 19, 2019 •

edited

Loading

mshaposhnik commented Nov 26, 2019 •

edited

Loading

skabashnyuk commented Nov 30, 2019 •

edited

Loading