Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize PermissionChecker performance #13221

Closed
skabashnyuk opened this issue Apr 24, 2019 · 1 comment
Closed

Optimize PermissionChecker performance #13221

skabashnyuk opened this issue Apr 24, 2019 · 1 comment
Labels
kind/task Internal things, technical debt, and to-do tasks to be performed. severity/P1 Has a major impact to usage or development of the system.
Milestone
7.2.0

Comments

@skabashnyuk
Copy link
Contributor

skabashnyuk commented Apr 24, 2019 •
edited
Loading

Description

At this moment to during permission checking, we are making a couple of requests to the database. That makes some operation extremely ineffective #13131 (comment) . That is happing because L2 Cache for queries is disabled by default https://www.eclipse.org/eclipselink/documentation/2.6/concepts/cache003.htm query hash for Worker.getByUserAndWorkspaceId request.
To enabled L2 cache for this type of operation, we can try to do two things

Che Server thread dump 

"MeteredCheMinorWebSocketEndpointExecutorServiceProvider-43" #282 daemon prio=5 os_prio=0 tid=0x0000562f6a6bd800 nid=0x178 in Object.wait() [0x00007f2c86c8d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:852)
	- locked <0x00000000e1d995d0> (a java.lang.Object)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:416)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:345)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:170)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:313)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1660)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
	- locked <0x00000000e903dea8> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:619)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:564)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2093)
	at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:600)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:270)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:256)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:327)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:722)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2740)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2693)
	at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:563)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1221)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1180)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:464)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1268)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2979)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1892)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1874)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260)
	at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:528)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:402)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.doGet(JpaWorkerDao.java:150)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.CGLIB$doGet$4(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a$$FastClassByGuice$$e63a0040.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.doGet(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.getEntity(JpaWorkerDao.java:134)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:86)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:46)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.get(<generated>)
	at org.eclipse.che.multiuser.api.permission.server.jpa.AbstractJpaPermissionsDao.exists(AbstractJpaPermissionsDao.java:66)
	at org.eclipse.che.multiuser.api.permission.server.PermissionsManager.exists(PermissionsManager.java:197)
	at org.eclipse.che.multiuser.api.permission.server.PermissionCheckerImpl.hasPermission(PermissionCheckerImpl.java:35)
	at org.eclipse.che.multiuser.api.permission.server.AuthorizedSubject.hasPermission(AuthorizedSubject.java:49)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.InstallerServicePermissionFilter.doAccept(InstallerServicePermissionFilter.java:65)
	at org.eclipse.che.multiuser.api.permission.server.jsonrpc.JsonRpcPermissionsFilterAdapter.accept(JsonRpcPermissionsFilterAdapter.java:29)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.filter(RequestHandlerManager.java:293)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.access$2400(RequestHandlerManager.java:48)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.handle(RequestHandlerManager.java:404)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.access$1700(RequestHandlerManager.java:393)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.handle(RequestHandlerManager.java:254)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestDispatcher.dispatch(RequestDispatcher.java:56)
	at org.eclipse.che.api.core.jsonrpc.commons.JsonRpcMessageReceiver$ProcessRequestTask.run(JsonRpcMessageReceiver.java:112)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:160)
	at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:143)
	at io.micrometer.core.instrument.Timer$$Lambda$276/1098018001.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"MeteredCheMinorWebSocketEndpointExecutorServiceProvider-42" #281 daemon prio=5 os_prio=0 tid=0x0000562f6a6bc800 nid=0x177 in Object.wait() [0x00007f2c86d8e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:852)
	- locked <0x00000000e1d995d0> (a java.lang.Object)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:416)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:345)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:170)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:313)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1660)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
	- locked <0x00000000e96c3d18> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:619)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:564)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2093)
	at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:600)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:270)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:256)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:327)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:722)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2740)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2693)
	at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:563)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1221)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1180)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:464)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1268)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2979)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1892)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1874)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260)
	at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:528)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:402)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.doGet(JpaWorkerDao.java:150)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.CGLIB$doGet$4(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a$$FastClassByGuice$$e63a0040.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.doGet(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.getEntity(JpaWorkerDao.java:134)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:86)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:46)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.get(<generated>)
	at org.eclipse.che.multiuser.api.permission.server.jpa.AbstractJpaPermissionsDao.exists(AbstractJpaPermissionsDao.java:66)
	at org.eclipse.che.multiuser.api.permission.server.PermissionsManager.exists(PermissionsManager.java:197)
	at org.eclipse.che.multiuser.api.permission.server.PermissionCheckerImpl.hasPermission(PermissionCheckerImpl.java:35)
	at org.eclipse.che.multiuser.api.permission.server.AuthorizedSubject.hasPermission(AuthorizedSubject.java:49)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.InstallerServicePermissionFilter.doAccept(InstallerServicePermissionFilter.java:65)
	at org.eclipse.che.multiuser.api.permission.server.jsonrpc.JsonRpcPermissionsFilterAdapter.accept(JsonRpcPermissionsFilterAdapter.java:29)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.filter(RequestHandlerManager.java:293)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.access$2400(RequestHandlerManager.java:48)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.handle(RequestHandlerManager.java:404)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.access$1700(RequestHandlerManager.java:393)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.handle(RequestHandlerManager.java:254)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestDispatcher.dispatch(RequestDispatcher.java:56)
	at org.eclipse.che.api.core.jsonrpc.commons.JsonRpcMessageReceiver$ProcessRequestTask.run(JsonRpcMessageReceiver.java:112)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:160)
	at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:143)
	at io.micrometer.core.instrument.Timer$$Lambda$276/1098018001.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"MeteredCheMinorWebSocketEndpointExecutorServiceProvider-90" #280 daemon prio=5 os_prio=0 tid=0x0000562f67ba8800 nid=0x176 in Object.wait() [0x00007f2c86e8f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:852)
	- locked <0x00000000e1d995d0> (a java.lang.Object)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:416)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:345)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:170)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:313)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1660)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
	- locked <0x00000000ea03dc58> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:619)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:564)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2093)
	at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:600)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:270)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:256)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:327)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:722)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2740)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2693)
	at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:563)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1221)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1180)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:464)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1268)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2979)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1892)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1874)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260)
	at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:528)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:402)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.doGet(JpaWorkerDao.java:150)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.CGLIB$doGet$4(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a$$FastClassByGuice$$e63a0040.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.doGet(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.getEntity(JpaWorkerDao.java:134)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:86)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:46)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.get(<generated>)
	at org.eclipse.che.multiuser.api.permission.server.jpa.AbstractJpaPermissionsDao.exists(AbstractJpaPermissionsDao.java:66)
	at org.eclipse.che.multiuser.api.permission.server.PermissionsManager.exists(PermissionsManager.java:197)
	at org.eclipse.che.multiuser.api.permission.server.PermissionCheckerImpl.hasPermission(PermissionCheckerImpl.java:35)
	at org.eclipse.che.multiuser.api.permission.server.AuthorizedSubject.hasPermission(AuthorizedSubject.java:49)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.InstallerServicePermissionFilter.doAccept(InstallerServicePermissionFilter.java:65)
	at org.eclipse.che.multiuser.api.permission.server.jsonrpc.JsonRpcPermissionsFilterAdapter.accept(JsonRpcPermissionsFilterAdapter.java:29)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.filter(RequestHandlerManager.java:293)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.access$2400(RequestHandlerManager.java:48)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.handle(RequestHandlerManager.java:404)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.access$1700(RequestHandlerManager.java:393)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.handle(RequestHandlerManager.java:254)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestDispatcher.dispatch(RequestDispatcher.java:56)
	at org.eclipse.che.api.core.jsonrpc.commons.JsonRpcMessageReceiver$ProcessRequestTask.run(JsonRpcMessageReceiver.java:112)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:160)
	at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:143)
	at io.micrometer.core.instrument.Timer$$Lambda$276/1098018001.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"MeteredCheMinorWebSocketEndpointExecutorServiceProvider-91" #279 daemon prio=5 os_prio=0 tid=0x0000562f67ba8000 nid=0x175 in Object.wait() [0x00007f2c86f90000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:852)
	- locked <0x00000000e1d995d0> (a java.lang.Object)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:416)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:345)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:170)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:313)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1660)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
	- locked <0x00000000e96c4e60> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:619)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:564)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2093)
	at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:600)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:270)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:256)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:327)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:722)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2740)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2693)
	at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:563)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1221)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1180)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:464)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1268)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2979)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1892)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1874)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260)
	at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:528)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:402)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.doGet(JpaWorkerDao.java:150)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.CGLIB$doGet$4(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a$$FastClassByGuice$$e63a0040.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.doGet(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.getEntity(JpaWorkerDao.java:134)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:86)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:46)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.get(<generated>)
	at org.eclipse.che.multiuser.api.permission.server.jpa.AbstractJpaPermissionsDao.exists(AbstractJpaPermissionsDao.java:66)
	at org.eclipse.che.multiuser.api.permission.server.PermissionsManager.exists(PermissionsManager.java:197)
	at org.eclipse.che.multiuser.api.permission.server.PermissionCheckerImpl.hasPermission(PermissionCheckerImpl.java:35)
	at org.eclipse.che.multiuser.api.permission.server.AuthorizedSubject.hasPermission(AuthorizedSubject.java:49)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.InstallerServicePermissionFilter.doAccept(InstallerServicePermissionFilter.java:65)
	at org.eclipse.che.multiuser.api.permission.server.jsonrpc.JsonRpcPermissionsFilterAdapter.accept(JsonRpcPermissionsFilterAdapter.java:29)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.filter(RequestHandlerManager.java:293)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.access$2400(RequestHandlerManager.java:48)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.handle(RequestHandlerManager.java:404)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.access$1700(RequestHandlerManager.java:393)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.handle(RequestHandlerManager.java:254)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestDispatcher.dispatch(RequestDispatcher.java:56)
	at org.eclipse.che.api.core.jsonrpc.commons.JsonRpcMessageReceiver$ProcessRequestTask.run(JsonRpcMessageReceiver.java:112)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:160)
	at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:143)
	at io.micrometer.core.instrument.Timer$$Lambda$276/1098018001.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"MeteredCheMinorWebSocketEndpointExecutorServiceProvider-94" #276 daemon prio=5 os_prio=0 tid=0x0000562f67ba5000 nid=0x174 in Object.wait() [0x00007f2c87091000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:852)
	- locked <0x00000000e1d995d0> (a java.lang.Object)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:416)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:345)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:170)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:313)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1660)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
	- locked <0x00000000e903f990> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:619)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:564)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2093)
	at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:600)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:270)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:256)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:327)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:722)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2740)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2693)
	at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:563)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1221)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1180)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:464)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1268)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2979)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1892)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1874)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260)
	at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:528)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:402)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.doGet(JpaWorkerDao.java:150)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.CGLIB$doGet$4(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a$$FastClassByGuice$$e63a0040.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.doGet(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.getEntity(JpaWorkerDao.java:134)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:86)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:46)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.get(<generated>)
	at org.eclipse.che.multiuser.api.permission.server.jpa.AbstractJpaPermissionsDao.exists(AbstractJpaPermissionsDao.java:66)
	at org.eclipse.che.multiuser.api.permission.server.PermissionsManager.exists(PermissionsManager.java:197)
	at org.eclipse.che.multiuser.api.permission.server.PermissionCheckerImpl.hasPermission(PermissionCheckerImpl.java:35)
	at org.eclipse.che.multiuser.api.permission.server.AuthorizedSubject.hasPermission(AuthorizedSubject.java:49)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.InstallerServicePermissionFilter.doAccept(InstallerServicePermissionFilter.java:65)
	at org.eclipse.che.multiuser.api.permission.server.jsonrpc.JsonRpcPermissionsFilterAdapter.accept(JsonRpcPermissionsFilterAdapter.java:29)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.filter(RequestHandlerManager.java:293)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.access$2400(RequestHandlerManager.java:48)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.handle(RequestHandlerManager.java:404)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.access$1700(RequestHandlerManager.java:393)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.handle(RequestHandlerManager.java:254)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestDispatcher.dispatch(RequestDispatcher.java:56)
	at org.eclipse.che.api.core.jsonrpc.commons.JsonRpcMessageReceiver$ProcessRequestTask.run(JsonRpcMessageReceiver.java:112)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:160)
	at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:143)
	at io.micrometer.core.instrument.Timer$$Lambda$276/1098018001.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"MeteredCheMinorWebSocketEndpointExecutorServiceProvider-95" #275 daemon prio=5 os_prio=0 tid=0x0000562f687fc800 nid=0x173 in Object.wait() [0x00007f2c87192000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:852)
	- locked <0x00000000e1d995d0> (a java.lang.Object)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:416)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:345)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:170)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:313)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1660)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
	- locked <0x00000000e96c60b0> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:619)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:564)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2093)
	at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:600)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:270)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:256)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:327)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:722)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2740)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2693)
	at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:563)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1221)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1180)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:464)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1268)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2979)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1892)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1874)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260)
	at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:528)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:402)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.doGet(JpaWorkerDao.java:150)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.CGLIB$doGet$4(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a$$FastClassByGuice$$e63a0040.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.doGet(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.getEntity(JpaWorkerDao.java:134)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:86)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:46)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.get(<generated>)
	at org.eclipse.che.multiuser.api.permission.server.jpa.AbstractJpaPermissionsDao.exists(AbstractJpaPermissionsDao.java:66)
	at org.eclipse.che.multiuser.api.permission.server.PermissionsManager.exists(PermissionsManager.java:197)
	at org.eclipse.che.multiuser.api.permission.server.PermissionCheckerImpl.hasPermission(PermissionCheckerImpl.java:35)
	at org.eclipse.che.multiuser.api.permission.server.AuthorizedSubject.hasPermission(AuthorizedSubject.java:49)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.InstallerServicePermissionFilter.doAccept(InstallerServicePermissionFilter.java:65)
	at org.eclipse.che.multiuser.api.permission.server.jsonrpc.JsonRpcPermissionsFilterAdapter.accept(JsonRpcPermissionsFilterAdapter.java:29)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.filter(RequestHandlerManager.java:293)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.access$2400(RequestHandlerManager.java:48)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.handle(RequestHandlerManager.java:404)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.access$1700(RequestHandlerManager.java:393)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.handle(RequestHandlerManager.java:254)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestDispatcher.dispatch(RequestDispatcher.java:56)
	at org.eclipse.che.api.core.jsonrpc.commons.JsonRpcMessageReceiver$ProcessRequestTask.run(JsonRpcMessageReceiver.java:112)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:160)
	at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:143)
	at io.micrometer.core.instrument.Timer$$Lambda$276/1098018001.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"MeteredCheMinorWebSocketEndpointExecutorServiceProvider-99" #271 daemon prio=5 os_prio=0 tid=0x0000562f687ff000 nid=0x172 in Object.wait() [0x00007f2c87293000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:852)
	- locked <0x00000000e1d995d0> (a java.lang.Object)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:416)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:345)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:170)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:313)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1660)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
	- locked <0x00000000e96c7290> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:619)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:564)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2093)
	at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:600)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:270)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:256)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:327)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:722)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2740)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2693)
	at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:563)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1221)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1180)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:464)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1268)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2979)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1892)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1874)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260)
	at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:528)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:402)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.doGet(JpaWorkerDao.java:150)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.CGLIB$doGet$4(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a$$FastClassByGuice$$e63a0040.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.doGet(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.getEntity(JpaWorkerDao.java:134)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:86)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:46)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.get(<generated>)
	at org.eclipse.che.multiuser.api.permission.server.jpa.AbstractJpaPermissionsDao.exists(AbstractJpaPermissionsDao.java:66)
	at org.eclipse.che.multiuser.api.permission.server.PermissionsManager.exists(PermissionsManager.java:197)
	at org.eclipse.che.multiuser.api.permission.server.PermissionCheckerImpl.hasPermission(PermissionCheckerImpl.java:35)
	at org.eclipse.che.multiuser.api.permission.server.AuthorizedSubject.hasPermission(AuthorizedSubject.java:49)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.InstallerServicePermissionFilter.doAccept(InstallerServicePermissionFilter.java:65)
	at org.eclipse.che.multiuser.api.permission.server.jsonrpc.JsonRpcPermissionsFilterAdapter.accept(JsonRpcPermissionsFilterAdapter.java:29)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.filter(RequestHandlerManager.java:293)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.access$2400(RequestHandlerManager.java:48)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.handle(RequestHandlerManager.java:404)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.access$1700(RequestHandlerManager.java:393)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.handle(RequestHandlerManager.java:254)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestDispatcher.dispatch(RequestDispatcher.java:56)
	at org.eclipse.che.api.core.jsonrpc.commons.JsonRpcMessageReceiver$ProcessRequestTask.run(JsonRpcMessageReceiver.java:112)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:160)
	at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:143)
	at io.micrometer.core.instrument.Timer$$Lambda$276/1098018001.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"MeteredCheMinorWebSocketEndpointExecutorServiceProvider-97" #273 daemon prio=5 os_prio=0 tid=0x0000562f687fe800 nid=0x171 in Object.wait() [0x00007f2c87394000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:852)
	- locked <0x00000000e1d995d0> (a java.lang.Object)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:416)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:345)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:170)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:313)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1660)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
	- locked <0x00000000e96c8390> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:619)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:564)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2093)
	at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:600)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:270)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:256)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:327)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:722)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2740)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2693)
	at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:563)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1221)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1180)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:464)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1268)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2979)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1892)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1874)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260)
	at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:528)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:402)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.doGet(JpaWorkerDao.java:150)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.CGLIB$doGet$4(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a$$FastClassByGuice$$e63a0040.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.doGet(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.getEntity(JpaWorkerDao.java:134)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:86)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:46)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.get(<generated>)
	at org.eclipse.che.multiuser.api.permission.server.jpa.AbstractJpaPermissionsDao.exists(AbstractJpaPermissionsDao.java:66)
	at org.eclipse.che.multiuser.api.permission.server.PermissionsManager.exists(PermissionsManager.java:197)
	at org.eclipse.che.multiuser.api.permission.server.PermissionCheckerImpl.hasPermission(PermissionCheckerImpl.java:35)
	at org.eclipse.che.multiuser.api.permission.server.AuthorizedSubject.hasPermission(AuthorizedSubject.java:49)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.InstallerServicePermissionFilter.doAccept(InstallerServicePermissionFilter.java:65)
	at org.eclipse.che.multiuser.api.permission.server.jsonrpc.JsonRpcPermissionsFilterAdapter.accept(JsonRpcPermissionsFilterAdapter.java:29)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.filter(RequestHandlerManager.java:293)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.access$2400(RequestHandlerManager.java:48)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.handle(RequestHandlerManager.java:404)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.access$1700(RequestHandlerManager.java:393)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.handle(RequestHandlerManager.java:254)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestDispatcher.dispatch(RequestDispatcher.java:56)
	at org.eclipse.che.api.core.jsonrpc.commons.JsonRpcMessageReceiver$ProcessRequestTask.run(JsonRpcMessageReceiver.java:112)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:160)
	at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:143)
	at io.micrometer.core.instrument.Timer$$Lambda$276/1098018001.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"MeteredCheMinorWebSocketEndpointExecutorServiceProvider-98" #272 daemon prio=5 os_prio=0 tid=0x0000562f687fd800 nid=0x170 in Object.wait() [0x00007f2c87495000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:852)
	- locked <0x00000000e1d995d0> (a java.lang.Object)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:416)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:345)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:170)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:313)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1660)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
	- locked <0x00000000e96c9570> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:619)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:564)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2093)
	at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:600)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:270)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:256)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:327)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:722)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2740)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2693)
	at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:563)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1221)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1180)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:464)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1268)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2979)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1892)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1874)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260)
	at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:528)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:402)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.doGet(JpaWorkerDao.java:150)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.CGLIB$doGet$4(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a$$FastClassByGuice$$e63a0040.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.doGet(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.getEntity(JpaWorkerDao.java:134)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:86)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:46)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.get(<generated>)
	at org.eclipse.che.multiuser.api.permission.server.jpa.AbstractJpaPermissionsDao.exists(AbstractJpaPermissionsDao.java:66)
	at org.eclipse.che.multiuser.api.permission.server.PermissionsManager.exists(PermissionsManager.java:197)
	at org.eclipse.che.multiuser.api.permission.server.PermissionCheckerImpl.hasPermission(PermissionCheckerImpl.java:35)
	at org.eclipse.che.multiuser.api.permission.server.AuthorizedSubject.hasPermission(AuthorizedSubject.java:49)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.InstallerServicePermissionFilter.doAccept(InstallerServicePermissionFilter.java:65)
	at org.eclipse.che.multiuser.api.permission.server.jsonrpc.JsonRpcPermissionsFilterAdapter.accept(JsonRpcPermissionsFilterAdapter.java:29)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.filter(RequestHandlerManager.java:293)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.access$2400(RequestHandlerManager.java:48)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.handle(RequestHandlerManager.java:404)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.access$1700(RequestHandlerManager.java:393)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.handle(RequestHandlerManager.java:254)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestDispatcher.dispatch(RequestDispatcher.java:56)
	at org.eclipse.che.api.core.jsonrpc.commons.JsonRpcMessageReceiver$ProcessRequestTask.run(JsonRpcMessageReceiver.java:112)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:160)
	at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:143)
	at io.micrometer.core.instrument.Timer$$Lambda$276/1098018001.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"MeteredCheMinorWebSocketEndpointExecutorServiceProvider-96" #274 daemon prio=5 os_prio=0 tid=0x0000562f687f8000 nid=0x16f in Object.wait() [0x00007f2c87596000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:852)
	- locked <0x00000000e1d995d0> (a java.lang.Object)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:416)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:345)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:170)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:346)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:313)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:581)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1660)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:321)
	- locked <0x00000000e96ca790> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:619)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:564)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2093)
	at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:600)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:270)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:256)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:327)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:722)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2740)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2693)
	at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:563)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1221)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1180)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:464)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1268)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2979)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1892)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1874)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260)
	at org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:528)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:402)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.doGet(JpaWorkerDao.java:150)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.CGLIB$doGet$4(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a$$FastClassByGuice$$e63a0040.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:64)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.doGet(<generated>)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.getEntity(JpaWorkerDao.java:134)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:86)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao.get(JpaWorkerDao.java:46)
	at org.eclipse.che.multiuser.permission.workspace.server.spi.jpa.JpaWorkerDao$$EnhancerByGuice$$9f9dbe4a.get(<generated>)
	at org.eclipse.che.multiuser.api.permission.server.jpa.AbstractJpaPermissionsDao.exists(AbstractJpaPermissionsDao.java:66)
	at org.eclipse.che.multiuser.api.permission.server.PermissionsManager.exists(PermissionsManager.java:197)
	at org.eclipse.che.multiuser.api.permission.server.PermissionCheckerImpl.hasPermission(PermissionCheckerImpl.java:35)
	at org.eclipse.che.multiuser.api.permission.server.AuthorizedSubject.hasPermission(AuthorizedSubject.java:49)
	at org.eclipse.che.multiuser.permission.workspace.server.filters.InstallerServicePermissionFilter.doAccept(InstallerServicePermissionFilter.java:65)
	at org.eclipse.che.multiuser.api.permission.server.jsonrpc.JsonRpcPermissionsFilterAdapter.accept(JsonRpcPermissionsFilterAdapter.java:29)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.filter(RequestHandlerManager.java:293)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.access$2400(RequestHandlerManager.java:48)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.handle(RequestHandlerManager.java:404)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager$OneToNoneHandler.access$1700(RequestHandlerManager.java:393)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestHandlerManager.handle(RequestHandlerManager.java:254)
	at org.eclipse.che.api.core.jsonrpc.commons.RequestDispatcher.dispatch(RequestDispatcher.java:56)
	at org.eclipse.che.api.core.jsonrpc.commons.JsonRpcMessageReceiver$ProcessRequestTask.run(JsonRpcMessageReceiver.java:112)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:160)
	at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:143)
	at io.micrometer.core.instrument.Timer$$Lambda$276/1098018001.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

@skabashnyuk skabashnyuk added kind/task Internal things, technical debt, and to-do tasks to be performed. team/platform severity/P1 Has a major impact to usage or development of the system. labels Apr 24, 2019
@skabashnyuk
Copy link
Contributor Author

Знімок екрана о 08 09 32
It is noticeable that the maximum time of json/rpc method execution is growing on Che server. I may assume that it is related to permission checking.

@skabashnyuk skabashnyuk modified the milestones: 7.3.0, 7.2.0 Sep 3, 2019
This was referenced Sep 4, 2019
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/task Internal things, technical debt, and to-do tasks to be performed. severity/P1 Has a major impact to usage or development of the system.
Projects
None yet
Milestone
7.2.0
Development

No branches or pull requests
1 participant
@skabashnyuk