Error accessing transitive dependencies via %maven

[andrus]

So I am working on the DFLib Jupyter integration per dflib/dflib#44 . It mostly works well. Found one problem though. I am including all the DFLib modules as "compile" dependencies of "dflib-jupyter" , so I assumed I can only specify %maven com.nhl.dflib:dflib-jupyter:0.6-SNAPSHOT once, and get all of them. Apparently not. Here is a screenshot where Jupyter fails to import a package from one of the dependencies:

Oddly enough, when instead of importing "com.nhl.dflib", I import "com.nhl.dflib.jdbc" package (from another module), the error is NoClassDefFound on the core DFLib package... So it seems to be ok with jdbc import, but still can't find the core library.

[andrus]

Also maybe related... Artifact resolution starts with "central", not "local" per screenshots above. This makes things pretty slow, as it pings central every single time (which is a separate problem, but still worth mentioning).

Anyways, as an experiment, I added %mavenRepo directive to point to DFLib snapshot repo. After that the errors above went away, but only for the first time. When running it again, I am back to the same errors. Strange...

[andrus]

Another pieces of %maven randomness... I have two notebooks with the same imports:

%maven com.nhl.dflib:dflib:0.6-SNAPSHOT
%maven com.nhl.dflib:dflib-jdbc:0.6-SNAPSHOT

So:

1. I start by doing rm -rf ~/.ivy2/ (this often helps with Maven issues).
2. I open Notebook 1 and run the above cell.. All is good...
3. I open Notebook 2 and run the above cell. I end up with the following error

java.lang.RuntimeException: Exception occurred while running line magic 'maven': Error resolving 'com.nhl.dflib:dflib-jdbc:0.6-SNAPSHOT'. [	ERROR: download no: com.nhl.dflib#dflib-jdbc;0.6-SNAPSHOT!dflib-jdbc.jar of jar, 	ERROR: 	from: ArtifactOrigin { isLocal=true, location=file:////Users/andrus/.m2/repository/com/nhl/dflib/dflib-jdbc/0.6-SNAPSHOT/dflib-jdbc-0.6-SNAPSHOT.jar, lastChecked=null, exists=true}, 	ERROR: download failed: com.nhl.dflib#dflib;0.6-SNAPSHOT!dflib.jar(test-jar) of test-jar, 	ERROR: 	Couldn't find artifact., 	ERROR: download no: org.slf4j#slf4j-api;1.7.25!slf4j-api.jar of jar, 	ERROR: 	from: ArtifactOrigin { isLocal=false, location=https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar, lastChecked=null, exists=true}, download failed: com.nhl.dflib#dflib;0.6-SNAPSHOT!dflib.jar(test-jar)]
	at io.github.spencerpark.ijava.runtime.Magics.lineMagic(Magics.java:19)
	at .(#15:1)

After this I can't rerun %maven in the notebook 1 anymore either. The only way to restore a working environment is by doing rm -rf ~/.ivy2/ again and restarting Jupyter.

So it gets stuck when for some reason it tries to download com.nhl.dflib#dflib;0.6-SNAPSHOT!dflib.jar(test-jar)] (which is a test-scope dependency and should really be ignored).

So the culprit is obviously Ivy, but I didn't get any further than that. Any insights are welcome.

[SpencerPark]

Hi @andrus, I really appreciate the detailed error reports here. School has taken priority as of lately and as a result I haven't had much free time to work on IJava and friends other than try and keep up with replying to issues. So please don't take the recent lack of progress as lack of interest in maintaining the project.

There are 2 issues here that are motivating a switch I want to make with the classpath. As the kernel has a dependency on the base kernel and the user space shares the kernel classpath, adding a new basekernel can causes issues with the kernel itself with old classes being replaced. Could you try marking the basekernel with the provided scope?

The other issue is that I would like to have smarter resolution and skip dependencies for which an artifact with the same coordinates is already on the classpath. I'm not sure if we can capture things manually added but at least be smarter about conflict resolution with additional explicit maven dependencies.

We can actually check which snippets of code break when a dependency is added. Maybe it is as simple as running them again after their class gets reloaded by the same one?

[SpencerPark]

I'll also link #22 for reference as if we go the user classpath route we could fix that one at the same time.

[andrus]

School has taken priority as of lately and as a result I haven't had much free time to work on IJava and friends other than try and keep up with replying to issues. So please don't take the recent lack of progress as lack of interest in maintaining the project.

No worries. As a long time open source developer I can relate.

Could you try marking the basekernel with the provided scope?

This solved the second issue - the conflict between 2 notebooks. It no longer appears.

[andrus]

More classpath fun. I reworked DFLibJupyter to use Logack for logging (this allows dynamic log level changes among other things). When I start with a clean Ivy repo (rm -rf ~/.ivy/), it works. But after either restarting the notebook, or Jupyter, I am getting this stack that won't go away until I do rm -rf ~/.ivy/ again.

I don't think this problem falls under your previous description, as it happens even after a restart. Looks like some kind of Ivy issue. Any idea what that might be, or at least where to dig?

Failed to instantiate SLF4J LoggerFactory
Reported exception:
java.lang.NoClassDefFoundError: ch/qos/logback/core/joran/spi/JoranException
	at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
	at com.nhl.dflib.jupyter.DFLibJupyter.getRootLogger(DFLibJupyter.java:92)
	at com.nhl.dflib.jupyter.DFLibJupyter.initLogging(DFLibJupyter.java:76)
	at com.nhl.dflib.jupyter.DFLibJupyter.init(DFLibJupyter.java:43)
	at REPL.$JShell$33.do_it$($JShell$33.java:26)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.github.spencerpark.ijava.execution.IJavaExecutionControl.lambda$execute$1(IJavaExecutionControl.java:95)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassNotFoundException: ch.qos.logback.core.joran.spi.JoranException
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
	at jdk.jshell/jdk.jshell.execution.DefaultLoaderDelegate$RemoteClassLoader.findClass(DefaultLoaderDelegate.java:154)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 16 more

[karussell]

Thanks for providing this nice tool. I also got my small Java test working, but stumble over a very similar problem that is described here when doing the following:

%maven com.graphhopper:graphhopper-core:1.0
%maven com.graphhopper:graphhopper-reader-osm:1.0
%maven com.carrotsearch:hppc:0.8.1
%maven org.slf4j:slf4j-api:1.7.30
%maven org.slf4j:slf4j-log4j12:1.7.30
%maven log4j:log4j:1.2.17

import com.graphhopper.*;
import com.graphhopper.config.*;
import com.graphhopper.util.*;
import com.graphhopper.reader.osm.*;
import com.graphhopper.routing.util.*;

if(!new File("osm.pbf").exists()) {
  System.out.println("download OpenStreetMap data ...");
  new Downloader("GraphHopper Downloader").downloadFile("http://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf", "osm.pbf");
}

System.out.println("start importOrLoad ...");
GraphHopper hopper = new GraphHopperOSM().
    setOSMFile("osm.pbf").
    setProfiles(Collections.singletonList(new Profile("car").setVehicle("car").setWeighting("fastest"))).
    setEncodingManager(EncodingManager.create("car")).
    setGraphHopperLocation("graph-cache").
    importOrLoad();

GHResponse rsp = hopper.route(new GHRequest(52.532932,13.341522, 52.503684,13.404694).setProfile("car"));
if(rsp.hasErrors())
    throw new RuntimeException(rsp.getErrors().get(0));
ResponsePath best = rsp.getBest();
System.out.println("distance " + best.getDistance());

I sometimes get errors that a certain class is missing although it should be included from one of the %maven statements. E.g.:

java.lang.NoClassDefFoundError: org/apache/log4j/Level

or

java.lang.NoClassDefFoundError: com/carrotsearch/hppc/IntLookupContainer

or

java.lang.IllegalStateException: org.slf4j.LoggerFactory in failed state. Original exception was thrown EARLIER. See also http://www.slf4j.org/codes.html#unsuccessfulInit
	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:427)

Sometimes it affects the last %maven statement. Not sure if it helps but this was (sometimes) reproducible at mybinder.org.

And one time I was able to remove such an error when I switched from %%loadFromPOM to %maven but it does not disappear now and is also permanent even if I restart my system. Is there a way to "restart from scratch"?

[andrus]

I've been having this issue consistently, and now I hope myself or my team will be able to dig into the sources and find a solution.

Steps to Reproduce

For the record, here are the maximally simplified steps to reproduce:

1. rm -rf ~/.ivy2
2. Create a notebook with the sections below (all dependencies are publicly-available on Maven central)
3. Run it once (you'll get a DB connection error at the end, but that's fine. It is not the one we are chasing)
4. Close the browser, stop jupyter
5. Running jupyter and going through this notebook again causes NoClassDefFoundError below when instantiating the connector

I don't know if this is a coincidence, but the NoClassDefFoundError is always on the "invisible" dependency (i.e. the classes we don't interact with directly in the code). The failing class is org/slf4j/LoggerFactory, loaded via the following dependency chain: dflib-jupyter -> dflib-jdbc -> slf4j-api, when a class from dflib-jdbc (SimpleDataSource) is invoked explicitly.

Notebook code

%maven org.dflib:dflib-jupyter:1.0.0-M20
%maven org.postgresql:postgresql:42.7.2

import org.dflib.jupyter.*;
import org.dflib.*;
import org.dflib.jdbc.*;

DFLibJupyter.init(getKernelInstance());

var connector = Jdbc.connector("jdbc:postgresql://localhost:15432/")
    .userName("postgres")
    .password("test")
    .build();

connector

Error

ava.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
	at org.dflib.jdbc.datasource.SimpleDataSource.<clinit>(SimpleDataSource.java:19)
	at org.dflib.jdbc.connector.JdbcConnectorBuilder.createDataSource(JdbcConnectorBuilder.java:97)
	at org.dflib.jdbc.connector.JdbcConnectorBuilder.build(JdbcConnectorBuilder.java:76)
	at .do_it$Aux(#24:4)
	at .(#24:1)

[stariy95]

Hi @SpencerPark @andrus
This problem seems could be solved by just updating Ivy. I already opened PR for that: #160.
Current Ivy version just unable to process test-jar dependencies correctly, causing all these troubles.