Enhance the JDBC Driver Manager to validate all driver supported properies (JDBC-driver, #1825) #1827

speckyspooky · 2024-08-03T07:53:54Z

Enhance the JDBC Driver Manager to validate all driver supported properties and remove the unsupported from the connection properties, e.g. not all drivers support Bidi-properties according to DuckDB (JDBC-driver, #1825)

...e.birt.report.data.oda.jdbc/src/org/eclipse/birt/report/data/oda/jdbc/JDBCDriverManager.java

merks · 2024-08-03T08:18:15Z

FYI, the build failures might be related to the new version of derby that I just made available in Orbit. I was just about to test a local build.

...e.birt.report.data.oda.jdbc/src/org/eclipse/birt/report/data/oda/jdbc/JDBCDriverManager.java

merks · 2024-08-03T08:41:16Z

FYI, tests are failing with the latest derby:

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.133 s <<< FAILURE! -- in org.eclipse.birt.data.engine.api.ClobAndBlobTest
org.eclipse.birt.data.engine.api.ClobAndBlobTest.testClobAndBlob -- Time elapsed: 0.130 s <<< ERROR!
java.sql.SQLException: Failed to create database 'C:\Users\merks\AppData\Local\Temp\\DTETest', see the next exception for details.
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:115)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:141)
	at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:252)
	at org.apache.derby.impl.jdbc.EmbedConnection.createDatabase(EmbedConnection.java:2656)
	at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:430)
	at org.apache.derby.iapi.jdbc.InternalDriver.getNewEmbedConnection(InternalDriver.java:630)
	at org.apache.derby.iapi.jdbc.InternalDriver.connect(InternalDriver.java:295)
	at org.apache.derby.iapi.jdbc.InternalDriver.connect(InternalDriver.java:922)
	at org.apache.derby.iapi.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:132)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:681)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:190)
	at testutil.JDBCDataSourceUtil.createDerbyConnection(JDBCDataSourceUtil.java:307)
	at testutil.JDBCDataSourceUtil.createDBConnection(JDBCDataSourceUtil.java:283)
	at testutil.JDBCDataSourceUtil.<init>(JDBCDataSourceUtil.java:63)
	at testutil.JDBCDataSource.<init>(JDBCDataSource.java:47)
	at testutil.JDBCDataSource.newInstance(JDBCDataSource.java:40)
	at org.eclipse.birt.data.engine.api.APITestCase.prepareTestTable(APITestCase.java:149)
	at org.eclipse.birt.data.engine.api.APITestCase.prepareDataSet(APITestCase.java:134)
	at org.eclipse.birt.data.engine.api.APITestCase.prepareDataSource(APITestCase.java:93)
	at org.eclipse.birt.data.engine.api.APITestCase.apiSetUp(APITestCase.java:71)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.maven.surefire.api.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:137)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:148)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:88)
	at org.eclipse.tycho.surefire.osgibooter.OsgiSurefireBooter.run(OsgiSurefireBooter.java:140)
	at org.eclipse.tycho.surefire.osgibooter.HeadlessTestApplication.start(HeadlessTestApplication.java:29)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:208)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:143)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:109)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:439)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:271)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:668)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:605)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1481)
	at org.eclipse.equinox.launcher.Main.main(Main.java:1454)
Caused by: ERROR XJ041: Failed to create database 'C:\Users\merks\AppData\Local\Temp\\DTETest', see the next exception for details.
	at org.apache.derby.shared.common.error.StandardException.newException(StandardException.java:299)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory.java:170)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:75)
	... 68 more
Caused by: ERROR XBM0A: The database directory 'C:\Users\merks\AppData\Local\Temp\DTETest' exists. However, it does not contain the expected 'service.properties' file. Perhaps Derby was brought down in the middle of creating this database. You may want to delete this directory and try creating the database again.
	at org.apache.derby.shared.common.error.StandardException.newException(StandardException.java:299)
	at org.apache.derby.shared.common.error.StandardException.newException(StandardException.java:294)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.vetService(StorageFactoryService.java:776)
	at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(StorageFactoryService.java:718)
	at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1741)
	at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(BaseMonitor.java:1011)
	at org.apache.derby.impl.services.monitor.FileMonitor.createPersistentService(FileMonitor.java:45)
	at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Monitor.java:650)
	at org.apache.derby.impl.jdbc.EmbedConnection.createPersistentService(EmbedConnection.java:4004)
	at org.apache.derby.impl.jdbc.EmbedConnection.createDatabase(EmbedConnection.java:2649)
	... 65 more

I'm not an expert in how derby should actually work. 😱

merks · 2024-08-03T09:00:01Z

FYI, the tests are always using the same temp folder name and when I deleted that from my temp, the tests succeeded. So hopefully there are no problems with derby. But I need to verify that before I can spend time on this PR....

merks · 2024-08-03T09:36:43Z

FYI, these tests fail, but when they fail they overwrite the image with the image generated by the test. I can't personally see any visual difference:

speckyspooky · 2024-08-03T10:04:26Z

Confirmed, I cannot see any difference too.

merks · 2024-08-03T11:45:05Z

I think my local failures are not actually related to the derby changes. They're just like the problems I had when I was trying get rid of the base64 encoding stuff. In any case, I have another workspace where I could look at your PR.

I think something like this would do the same thing but is a bit simpler:

	/*
	 * Remove unsupported properties from the connection properties based on the
	 * driver property information set
	 */
	private void removeUnsupportedConnectionProperties(Driver driver, String url, Properties connectionProperties) {
		try {
			DriverPropertyInfo[] supportedProperties = driver.getPropertyInfo(url, connectionProperties);
			if (supportedProperties != null && connectionProperties != null) {
				connectionProperties.keySet().removeIf(property -> {
					String key = property.toString();
					for (DriverPropertyInfo supportedProperty : supportedProperties) {
						if (supportedProperty.name.equalsIgnoreCase(key)) {
							return false;
						}
					}
					return true;
				});
			}
		} catch (SQLException e) {
			// do nothing, standard handling
		}
	}

The keySet() supports remove and we use predicate to specific which keys to remove...

speckyspooky · 2024-08-03T11:57:42Z

Looks very nice, I replaced my version through you suggestion.
My local test was successful with it.

speckyspooky · 2024-08-03T15:42:15Z

The PR is failed again with the message:

2024-08-03T15:35:03.4658092Z Caused by: java.lang.ClassNotFoundException: org.eclipse.jgit.internal.JGitText

merks · 2024-08-03T15:55:34Z

@speckyspooky

These test failures really are test failures caused by the changes in this PR. For example, I see user and password properties being removed. I'm running the test locally and will provide more detail. Special case support for those two reduces it to 4 errors:

merks · 2024-08-03T15:56:45Z

No, the jgit thing is a red herring. Many, many tests fail:

merks · 2024-08-03T16:00:25Z

This works better:

	/*
	 * Remove unsupported properties from the connection properties
     * based on the driver property information set
	 */
	private void removeUnsupportedConnectionProperties(Driver driver, String url, Properties connectionProperties) {
		try {
			DriverPropertyInfo[] supportedProperties = driver.getPropertyInfo(url, connectionProperties);
			if (supportedProperties != null && connectionProperties != null) {
				connectionProperties.keySet().removeIf(property -> {
					String key = property.toString();
					if ("password".equalsIgnoreCase(key) || "user".equalsIgnoreCase(key)) {
						return false;
					}
					for (DriverPropertyInfo supportedProperty : supportedProperties) {
						if (supportedProperty.name.equalsIgnoreCase(key)) {
							return false;
						}
					}
					return true;
				});
			}
		} catch (SQLException e) {
			// do nothing, standard handling
		}
	}

A few of the tests fail randomly when I launch them in the IDE, but hopefully the above consistently passes the tests. Hopefully there are not other properties that might be important that might get removed unexpectedly with this change.

speckyspooky · 2024-08-03T16:06:43Z

Thanks! I will add the special sequence and then we will see with crossed fingures.

properties and remove the unsupported from the connection properties

merks · 2024-08-03T16:08:54Z

Note that the master build succeeded after my derby-related changes:

So if you see failures here, it's highly likely to be a real failure caused by the PR's changes...

speckyspooky · 2024-08-03T16:14:07Z

I fetched your changes to my fork and the project recreation is currently running on my side locally.

speckyspooky · 2024-08-03T16:31:08Z

@merks
Ed, one question, I fetched your PR to my fork and my branch of this PR.
Started the "Perform Setup Task" of the BIRT-modules and started under "Project" the "Clean"-option.
I started eclipse new and repeated the steps 2times.

But the result of my local test of the "ODA Data" is very red with multiple findings in compare to our 4 failures.
Have you an idea here or do you mean this a temporary problem but why... The good thing is the Build was sucessfully.

The core issue message is like originally on your screen where you start with the derby investigation.
COuld it be that I haven't the latest version but I thought with your PR and the refreshs I should get them.

merks · 2024-08-03T16:38:58Z

Yes, that got me too in the beginning and required some debugging.

Delete this from the temp folder. On Windows

C:\Users\merks\AppData\Local\Temp\DTETest

This folder from your test failure:

Better that the tests cleaned it properly. 😢

I'm going out now so help will be delayed until after dinner. At least his PR's test passed. 🥳

speckyspooky · 2024-08-03T17:09:43Z

Yes, the folder clean up help with the listed errors.

merks

A work of art. 🖼️

speckyspooky added the Enhancement Small change to improve the current supported functionality label Aug 3, 2024

speckyspooky added this to the 4.17 milestone Aug 3, 2024

merks reviewed Aug 3, 2024

View reviewed changes

...e.birt.report.data.oda.jdbc/src/org/eclipse/birt/report/data/oda/jdbc/JDBCDriverManager.java Show resolved Hide resolved

...e.birt.report.data.oda.jdbc/src/org/eclipse/birt/report/data/oda/jdbc/JDBCDriverManager.java Outdated Show resolved Hide resolved

speckyspooky force-pushed the enhance_jdbc_driver_manager_1825 branch 2 times, most recently from 4205cac to b344a48 Compare August 3, 2024 08:27

speckyspooky self-assigned this Aug 3, 2024

merks reviewed Aug 3, 2024

View reviewed changes

...e.birt.report.data.oda.jdbc/src/org/eclipse/birt/report/data/oda/jdbc/JDBCDriverManager.java Outdated Show resolved Hide resolved

speckyspooky force-pushed the enhance_jdbc_driver_manager_1825 branch from b344a48 to a40a72e Compare August 3, 2024 10:25

speckyspooky mentioned this pull request Aug 3, 2024

Support for DuckDB JDBC #1825

Closed

speckyspooky force-pushed the enhance_jdbc_driver_manager_1825 branch from a40a72e to 11ccfb4 Compare August 3, 2024 11:56

merks force-pushed the enhance_jdbc_driver_manager_1825 branch from 11ccfb4 to d35f522 Compare August 3, 2024 15:23

Enhance the JDBC Driver Manager to validate all driver supported

0add57f

properties and remove the unsupported from the connection properties

speckyspooky force-pushed the enhance_jdbc_driver_manager_1825 branch from d35f522 to 0add57f Compare August 3, 2024 16:08

Merge branch 'eclipse-birt:master' into enhance_jdbc_driver_manager_1825

b7f9637

speckyspooky requested a review from merks August 3, 2024 17:09

merks approved these changes Aug 3, 2024

View reviewed changes

speckyspooky merged commit 3613adf into eclipse-birt:master Aug 3, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance the JDBC Driver Manager to validate all driver supported properies (JDBC-driver, #1825) #1827

Enhance the JDBC Driver Manager to validate all driver supported properies (JDBC-driver, #1825) #1827

speckyspooky commented Aug 3, 2024

merks commented Aug 3, 2024

merks commented Aug 3, 2024

merks commented Aug 3, 2024

merks commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

merks commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

merks commented Aug 3, 2024

merks commented Aug 3, 2024

merks commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

merks commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

speckyspooky commented Aug 3, 2024 •

edited

Loading

merks commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

merks left a comment

Enhance the JDBC Driver Manager to validate all driver supported properies (JDBC-driver, #1825) #1827

Enhance the JDBC Driver Manager to validate all driver supported properies (JDBC-driver, #1825) #1827

Conversation

speckyspooky commented Aug 3, 2024

merks commented Aug 3, 2024

merks commented Aug 3, 2024

merks commented Aug 3, 2024

merks commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

merks commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

merks commented Aug 3, 2024

merks commented Aug 3, 2024

merks commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

merks commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

speckyspooky commented Aug 3, 2024 • edited Loading

merks commented Aug 3, 2024

speckyspooky commented Aug 3, 2024

merks left a comment

Choose a reason for hiding this comment

speckyspooky commented Aug 3, 2024 •

edited

Loading