Startup failures due to corruption of H2 DB

[sadilchamishka]

Describe the issue:

The following error is logged when server startup when the WSO2CarbonDB related H2 file is corrupted. The workaround for the issue is to replace the corrupted H2 file with a new file and start the server. But this limitation should be fixed or introduce a prevention mechanism without escalating the issue up to server startup failure.

The WSO2CarbonDB keeps the runtime configurations of the server and it should have been required to deployed per node. Also, running H2 in memory database is not recommended for production usages. We have to check if there any alternatives or else any usage of the WSO2CarbonDB for the IS runtime.

TID: [-1234] [] [2021-01-06 11:38:35,598] [] ERROR {org.wso2.carbon.registry.core.jdbc.dao.JDBCPathCache} - Failed to retrieving resource from /. Table "REG_PATH" not found; SQL statement:
SELECT REG_PATH_ID FROM REG_PATH WHERE REG_PATH_VALUE=? AND REG_TENANT_ID=? [42102-199] org.h2.jdbc.JdbcSQLSyntaxErrorException: Table "REG_PATH" not found; SQL statement:
SELECT REG_PATH_ID FROM REG_PATH WHERE REG_PATH_VALUE=? AND REG_TENANT_ID=? [42102-199]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:451)
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:427)
	at org.h2.message.DbException.get(DbException.java:205)
	at org.h2.message.DbException.get(DbException.java:181)
	at org.h2.command.Parser.readTableOrView(Parser.java:7146)
	at org.h2.command.Parser.readTableFilter(Parser.java:1895)
	at org.h2.command.Parser.parseSelectSimpleFromPart(Parser.java:2641)
	at org.h2.command.Parser.parseSelectSimple(Parser.java:2788)
	at org.h2.command.Parser.parseSelectSub(Parser.java:2636)
	at org.h2.command.Parser.parseSelectUnion(Parser.java:2469)
	at org.h2.command.Parser.parseSelect(Parser.java:2440)
	at org.h2.command.Parser.parsePrepared(Parser.java:814)
	at org.h2.command.Parser.parse(Parser.java:788)
	at org.h2.command.Parser.parse(Parser.java:760)
	at org.h2.command.Parser.prepareCommand(Parser.java:683)
	at org.h2.engine.Session.prepareLocal(Session.java:627)
	at org.h2.engine.Session.prepareCommand(Session.java:565)
	at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1292)
	at org.h2.jdbc.JdbcPreparedStatement.<init>(JdbcPreparedStatement.java:77)
	at org.h2.jdbc.JdbcConnection.prepareStatement(JdbcConnection.java:349)
	at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:126)
	at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
	at org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:75)
	at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
	at org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:75)
	at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
	at org.apache.tomcat.jdbc.pool.interceptor.ConnectionState.invoke(ConnectionState.java:152)
	at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
	at org.apache.tomcat.jdbc.pool.TrapException.invoke(TrapException.java:40)
	at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
	at org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:75)
	at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:108)
	at org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:81)
	at com.sun.proxy.$Proxy53.prepareStatement(Unknown Source)
	at org.wso2.carbon.registry.core.jdbc.dataaccess.ConnectionWrapper.prepareStatement(ConnectionWrapper.java:71)
	at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCDatabaseTransaction$ManagedRegistryConnection.prepareStatement(JDBCDatabaseTransaction.java:1475)
	at org.wso2.carbon.registry.core.jdbc.dao.JDBCPathCache.getPathID(JDBCPathCache.java:339)
	at org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getResourceID(JDBCResourceDAO.java:71)
	at org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:155)
	at org.wso2.carbon.registry.core.utils.RegistryUtils.systemResourceShouldBeAdded(RegistryUtils.java:675)
	at org.wso2.carbon.registry.core.session.UserRegistry.addRootCollection(UserRegistry.java:402)
	at org.wso2.carbon.registry.core.session.UserRegistry.init(UserRegistry.java:340)
	at org.wso2.carbon.registry.core.session.UserRegistry.access$100(UserRegistry.java:73)
	at org.wso2.carbon.registry.core.session.UserRegistry$2.run(UserRegistry.java:261)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.wso2.carbon.registry.core.session.UserRegistry.<init>(UserRegistry.java:258)
	at org.wso2.carbon.registry.core.session.UserRegistry.<init>(UserRegistry.java:237)
	at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getUserRegistry(EmbeddedRegistryService.java:427)
	at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getSystemRegistry(EmbeddedRegistryService.java:292)
	at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getSystemRegistry(EmbeddedRegistryService.java:276)
	at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getSystemRegistry(EmbeddedRegistryService.java:262)
	at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.configure(EmbeddedRegistryService.java:219)
	at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.<init>(EmbeddedRegistryService.java:99)
	at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.getEmbeddedRegistryService(RegistryCoreServiceComponent.java:598)
	at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.buildRegistryService(RegistryCoreServiceComponent.java:520)
	at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.activate(RegistryCoreServiceComponent.java:123)

How to reproduce:

The forcefull shutdown can cause this issue.

[mifrazmurthaja]

This also leads to PoolExhaustedException when there is a high load.

TID: [-1234] [carbon.super] [oauth2] [2023-10-05 15:26:12,050] [8d6bbfbe-4646-4566-b049-0447b5b3d089] ERROR {org.wso2.carbon.registry.core.dataaccess.TransactionManager} - Failed to start new registry transaction. org.apache.tomcat.jdbc.pool.PoolExhaustedException: [https-jsse-nio-9443-exec-164] Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none available[size:50; busy:46; idle:0; lastwait:60000].
	at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:738)
	at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:198)
	at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:136)

[ShanChathusanda93]

Closing the issue since the fix is done from [1]

[1] wso2/carbon-kernel#3773