Jetty 12 fails to start WebApp Bundle with OSGi Boot bundle (or when packaged)

[cmark]

Jetty version(s)
Jetty 12.0.6

Jetty Environment
ee10

Java version/vendor (use: java -version)
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7)
OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode, sharing)

OS type/version
Microsoft Windows 10 22H2 (OS Build 19045.3930)

Description

Thank you for fixing the osgi.boot bundle config errors (#11280) and releasing it as part of 12.0.6.
Unfortunately, we are still unable to our tests and our runtime due to an invalid path exception error.

Local dev env error when starting our test launch configuration from Eclipse:

[2024-02-06T10:55:39.028] WARN  main                                             org.eclipse.jetty.deploy.DeploymentManager                       Unable to reach node goal: started java.nio.file.InvalidPathException: Illegal char <:> at index 4: file:///C:/jettyhome
	at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
	at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
	at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
	at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92)
	at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:232)
	at java.base/java.nio.file.Path.of(Path.java:147)
	at java.base/java.nio.file.Paths.get(Paths.java:69)
	at org.eclipse.jetty.ee10.osgi.boot.EE10Activator$EE10WebAppFactory.createContextHandler(EE10Activator.java:360)

We are configuring the osgi.boot bundle using the jetty.home.bundle jvm argument:

-Djetty.home.bundle=org.eclipse.jetty.ee10.osgi.boot

The error is similar in our build regardless of whether we use the osgi.boot bundle or using custom jetty*.xml configuration:

[2024-02-06T11:17:10.632] WARN  main                                             org.eclipse.jetty.ee10.webapp.WebAppContext                      Failed startup of context oeje10w.WebAppContext@7e461408{/snowowl,/snowowl,b=null,a=STOPPED,h=oeje10s.SessionHandler@7d86a2fd{STOPPED}}{file:/C:/dev/snow-owl-9.x/git/snow-owl/core/com.b2international.snowowl.core.rest/target/com.b2international.snowowl.core.rest-9.0.1-SNAPSHOT.jar} java.nio.file.InvalidPathException: Illegal char <:> at index 4: file:/C:/dev/snow-owl-9.x/git/snow-owl/core/com.b2international.snowowl.core.rest/target/com.b2international.snowowl.core.rest-9.0.1-SNAPSHOT.jar
	at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
	at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
	at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
	at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92)
	at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:232)
	at java.base/java.nio.file.Path.of(Path.java:147)
	at org.eclipse.jetty.ee10.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:242)
	at org.eclipse.jetty.ee10.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:66)
	at org.eclipse.jetty.ee10.webapp.Configurations.preConfigure(Configurations.java:487)
	at org.eclipse.jetty.ee10.webapp.WebAppContext.preConfigure(WebAppContext.java:454)
	at org.eclipse.jetty.ee10.webapp.WebAppContext.doStart(WebAppContext.java:495)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)

Our web app bundle should boot based on a WEB-INF/web.xml file, but it looks like there is a preConfigure step that runs for every recognized configuration class and if one of them fails then the boot process fails.
While this is a warning, there is no other error that can be found in the logs regarding the boot process and everything seems to work if we use explicit jetty.xml configuration files and not embedded ones (or the boot bundle).

Could this be a configuration issue on our end?

How to reproduce?

[cmark]

It looks like our packaged runtime is also failing with a similar error when it tries to boot the webapp bundle. Just guessing, but I think the cause is related to the fact that this is not a war but a jar and the WebInfConfiguration fails to detect that in the preConfigure step for an unknown reason. When running the dev version locally where the webapp bundle is not packaged yet (imported as an Eclipse project) the deployment works without any issue, but packaging it into an OSGi Web App and trying to deploy it fails.

[janbartel]

The jetty code base changed from using the Resource class to resolve locations to using the jvm Path class instead. Resource was very flexible with the way it resolved locations, it could handle URI or just a file location, but Path is more finicky.. Can you try just passing in the location as C:/jettyhome?

[cmark]

I can try but that can only work around the osgi.boot related issue.
The other is about accessing and extracting the contents of the webapp jar file which fails because the URI starts with file:/. It looks like it does some conversion to jar:/ but the code uses the war location path, which is still a file:/ and that does not work with jvm Path.

Correct me if I'm wrong, but shouldn't the code in this block alter the value of the war variable? It seems like that even if jvm Path would resolve the location without errors to a Path object unpack would not happen because the webapp is not a war file.

[cmark]

Can you try just passing in the location as C:/jettyhome?

Setting both the jetty.home.bundle and jetty.home together won't work as the system will ignore the jetty.home.bundle altogether.

[2024-02-07T10:19:00.360] WARN  main    org.eclipse.jetty.osgi.JettyBootstrapActivator                   Both jetty.home and jetty.home.bundle property defined: jetty.home.bundle ignored. 
[2024-02-07T10:19:00.360] WARN  main    org.eclipse.jetty.osgi.JettyBootstrapActivator                   Unable to locate the jetty.home folder C:/jettyhome 

We need the osgi.boot bundle to boot up the jetty context with defaults in OSGi context and also to recognize and boot our webapp bundle.

According to my understanding, the C:/jettyhome path is a fake path generated/set by the osgi.boot bundle during init, at least that was the case with Jetty 9 and 10.

[cmark]

After a bit of further testing, it looks like Jetty 12 and the web app can successfully boot on Unix machines without any Path issues. So this could be only affecting deployments on Windows machines.

[janbartel]

@cmark yeah, I think the change to Path is our issue here, I'll work on a PR to fix it.

[janbartel]

@cmark you might like to build and test the PR #11418 in your environment and see if I've fixed it?

[cmark]

@cmark you might like to build and test the PR #11418 in your environment and see if I've fixed it?

Hey, I can take a look. I'll let you know if the issue is still present or not. Thanks!

[cmark]

@janbartel

Fixes in #11418 do resolve the osgi.boot path issue when running our integration tests on a developer machine with Windows 10. With that we can start using the osgi.boot bundle again in our tests. ☺️

Unfortunately, the other error thrown at org.eclipse.jetty.ee10.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:242) is still present if we build our modules on Windows via Maven.

[janbartel]

Thans @cmark I'll take a look at the WebInfConfiguration issue.

[janbartel]

@joakime as you're looking into windows issues, can you also look at the Exception generated when performing a WebInfConfiguration.unpack on windows with a url like file:/C:/xxx/yyy/zzz.jar (Note, the .jar could be a .war also).

The OP here has the full stack trace: #11377 (comment)

[janbartel]

@cmark I've pushed a speculative fix for WebInfConfiguration.unpack to that branch, any chance you could retest??

[cmark]

@cmark I've pushed a speculative fix for WebInfConfiguration.unpack to that branch, any chance you could retest??

Yeah sure, I will rebuild and retest now.

[cmark]

@janbartel I'm happy to report that the speculative fix resolves the other error. Nice! Thanks!