Jetty start module property values with ${expr} are not being expanded

[stroucki]

Jetty version(s)
12.0.6

Jetty Environment
ee10

Java version/vendor (use: java -version)
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment (build 17.0.10+7-Debian-1deb11u1)
OpenJDK 64-Bit Server VM (build 17.0.10+7-Debian-1deb11u1, mixed mode, sharing)

OS type/version
Devuan Chimaera

Description
${jetty.base} is not expanded in ee10-deploy.ini

How to reproduce?
ee10-deploy.ini contains jetty.deploy.defaultsDescriptorPath=${jetty.base}/etc/webdefault-ee10.xml
java -jar ../jetty-home-12.0.6/start.jar --list-config shows jetty.base is found:

 user.dir = /opt/jetty-base
 jetty.base = /opt/jetty-base
 ${jetty.base} -> /opt/jetty-base
 jetty.base = /opt/jetty-base
 jetty.base.uri = file:///opt/jetty-base

But in a trace Jetty is not opening the correct file:

21523 stat("/opt/jetty-base/resources/${jetty.base}/etc/webdefault-ee10.xml", 0x7f3fa60d6a90) = -1 ENOENT (No such file or directory)
21523 stat("/opt/jetty-base/resources/${jetty.base}/etc/webdefault-ee10.xml", 0x7f3fa60d6b00) = -1 ENOENT (No such file or directory)
21523 stat("${jetty.base}/etc/webdefault-ee10.xml", 0x7f3fa60d6f00) = -1 ENOENT (No such file or directory)
21523 access("/opt/jetty-base/${jetty.base}/etc/webdefault-ee10.xml", F_OK) = -1 ENOENT (No such file or directory)

[joakime]

There's something fishy about your configuration.

Look for anything in your $JETTY_BASE/start.ini or $JETTY_BASE/start.d/*.ini that has an entry that starts with ${jetty.base}=/opt/jetty-base
OR you might have a java system property defined as -D${jetty.base}=/opt/jetty-base

Can you document what steps you used to initialize your jetty.base?
From the fresh creation of the new jetty.base directory, each --add-module=<name> you have done, and then finally the last command where you use --list-config and see what you reported.

[stroucki]

I have no $JETTY_BASE/start.ini. None of my $JETTY_BASE/start.d/*ini files configure a ${jetty.base}. Nothing in my environment contains JETTY_BASE. The only java system property referring to /opt/jetty-base is user.dir:/opt/jetty-base.
I am starting jetty by java -jar ../jetty-home-12.0.6/start.jar

To initialize, I added the modules using add-module, set up ssl and installed a webapp.

[stroucki]

From just starting a jetty-base:

foo:/opt# mkdir jetty-foo
foo:/opt# cd jetty-foo/
foo:/opt/jetty-foo# java -jar ../jetty-home-12.0.6/start.jar --list-config|grep jetty.base
 jetty.base = /opt/jetty-foo
 ${jetty.base} -> /opt/jetty-foo
 jetty.base = /opt/jetty-foo
 jetty.base.uri = file:///opt/jetty-foo

[joakime]

Ah, you filtered the output of --list-config in a way that I misread what you are pointing out.

Let me give this a try.

[joakime]

I am able to replicate.
This is a curious behavior, not limited to ${jetty.base}

Setup fresh jetty.base directory

[bases]$ mkdir issue-11408
[bases]$ cd issue-11408/
[issue-11408]$ java -jar ../../jetty-home-12.0.6/start.jar --add-module=ee10-deploy
INFO  : mkdir ${jetty.base}/start.d
INFO  : server          transitively enabled, ini template available with --add-modules=server
INFO  : sessions        transitively enabled, ini template available with --add-modules=sessions
INFO  : ee10-security   transitively enabled
INFO  : resources       transitively enabled
INFO  : threadpool      transitively enabled, ini template available with --add-modules=threadpool
INFO  : deploy          transitively enabled
INFO  : logging-jetty   transitively enabled
INFO  : security        transitively enabled
INFO  : ee10-deploy     initialized in ${jetty.base}/start.d/ee10-deploy.ini
INFO  : ee10-webapp     transitively enabled, ini template available with --add-modules=ee10-webapp
INFO  : ee10-servlet    transitively enabled
INFO  : logging/slf4j   dynamic dependency of logging-jetty
INFO  : bytebufferpool  transitively enabled, ini template available with --add-modules=bytebufferpool
INFO  : mkdir ${jetty.base}/resources
INFO  : mkdir ${jetty.base}/webapps
INFO  : copy ${jetty.home}/modules/logging/jetty/resources/jetty-logging.properties to ${jetty.base}/resources/jetty-logging.properties
INFO  : Base directory was modified
[issue-11408]$ edit start.d/ee10-deploy.ini
[issue-11408]$ grep defaultsDescriptorPath start.d/ee10-deploy.ini
jetty.deploy.defaultsDescriptorPath=${jetty.base}/etc/webdefault-ee10.xml
[issue-11408]$ mkdir etc
[issue-11408]$ cp ../../jetty-home-12.0.6/etc/webdefault-ee10.xml etc/
[issue-11408]$ ls -la etc
total 28
drwxrwxr-x 2 joakim joakim  4096 Feb 14 14:11 .
drwxrwxr-x 6 joakim joakim  4096 Feb 14 14:11 ..
-r--r--r-- 1 joakim joakim 17219 Feb 14 14:11 webdefault-ee10.xml

Show configuration

[issue-11408]$ java -jar ../../jetty-home-12.0.6/start.jar --list-config

Enabled Modules:
----------------
  0) resources                 transitive provider of resources for logging-jetty
  1) logging/slf4j             dynamic dependency of logging-jetty
  2) logging-jetty             transitive provider of logging for threadpool
  3) bytebufferpool            transitive provider of bytebufferpool for server
                               ini template available with --add-modules=bytebufferpool
  4) threadpool                transitive provider of threadpool for server
                               ini template available with --add-modules=threadpool
  5) server                    transitive provider of server for deploy
                               ini template available with --add-modules=server
  6) deploy                    transitive provider of deploy for ee10-deploy
  7) sessions                  transitive provider of sessions for ee10-servlet
                               ini template available with --add-modules=sessions
  8) ee10-servlet              transitive provider of ee10-servlet for ee10-webapp
  9) security                  transitive provider of security for ee10-security
 10) ee10-security             transitive provider of ee10-security for ee10-webapp
 11) ee10-webapp               transitive provider of ee10-webapp for ee10-deploy
                               ini template available with --add-modules=ee10-webapp
 12) ee10-deploy               ${jetty.base}/start.d/ee10-deploy.ini

JVM Version & Properties:
-------------------------
 java.home = /home/joakim/java/jvm/jdk-17.0.6+10
 java.vm.vendor = Eclipse Adoptium
 java.vm.version = 17.0.6+10
 java.vm.name = OpenJDK 64-Bit Server VM
 java.vm.info = mixed mode, sharing
 java.runtime.name = OpenJDK Runtime Environment
 java.runtime.version = 17.0.6+10
 java.io.tmpdir = /tmp
 user.dir = /home/joakim/code/jetty/distros/bases/issue-11408
 user.language = en
 user.country = US

Jetty Version & Properties:
---------------------------
 jetty.version = 12.0.6
 jetty.tag.version = jetty-12.0.6
 jetty.build = 78ab6e6ba163f89cdd97f2ae0283fbb5e371cfaf
 jetty.home = /home/joakim/code/jetty/distros/jetty-home-12.0.6
 jetty.base = /home/joakim/code/jetty/distros/bases/issue-11408

Config Search Order:
--------------------
 <command-line>
 ${jetty.base} -> /home/joakim/code/jetty/distros/bases/issue-11408
 ${jetty.home} -> /home/joakim/code/jetty/distros/jetty-home-12.0.6

System Properties:
------------------
 (no system properties specified)

Properties: Jetty
-----------------
 java.version = 17.0.6
 java.version.major = 17
 java.version.micro = 6
 java.version.minor = 0
 java.version.platform = 17
 jetty.base = /home/joakim/code/jetty/distros/bases/issue-11408
 jetty.base.uri = file:///home/joakim/code/jetty/distros/bases/issue-11408
 jetty.home = /home/joakim/code/jetty/distros/jetty-home-12.0.6
 jetty.home.uri = file:///home/joakim/code/jetty/distros/jetty-home-12.0.6
 jetty.webapp.addServerClasses = org.eclipse.jetty.logging.,${jetty.home.uri}/lib/logging/,org.slf4j.
 runtime.feature.alpn = true
 slf4j.version = 2.0.9

Classpath: Jetty
----------------
Version Information on 11 entries in the classpath.
Note: order presented here is how they would appear on the classpath.
      changes to the --module=name command line options will be reflected here.
 0:                    (dir) | ${jetty.base}/resources
 1:                    2.0.9 | ${jetty.home}/lib/logging/slf4j-api-2.0.9.jar
 2:                   12.0.6 | ${jetty.home}/lib/logging/jetty-slf4j-impl-12.0.6.jar
 3:                   12.0.6 | ${jetty.home}/lib/jetty-http-12.0.6.jar
 4:                   12.0.6 | ${jetty.home}/lib/jetty-server-12.0.6.jar
 5:                   12.0.6 | ${jetty.home}/lib/jetty-xml-12.0.6.jar
 6:                   12.0.6 | ${jetty.home}/lib/jetty-util-12.0.6.jar
 7:                   12.0.6 | ${jetty.home}/lib/jetty-io-12.0.6.jar
 8:                   12.0.6 | ${jetty.home}/lib/jetty-deploy-12.0.6.jar
 9:                   12.0.6 | ${jetty.home}/lib/jetty-session-12.0.6.jar
10:                   12.0.6 | ${jetty.home}/lib/jetty-security-12.0.6.jar

Active XMLs: Jetty
------------------
 ${jetty.home}/etc/jetty-bytebufferpool.xml
 ${jetty.home}/etc/jetty-threadpool.xml
 ${jetty.home}/etc/jetty.xml
 ${jetty.home}/etc/jetty-deploy.xml
 ${jetty.home}/etc/sessions/id-manager.xml

Properties: ee10
----------------
 contextHandlerClass = org.eclipse.jetty.ee10.webapp.WebAppContext
 jetty.deploy.defaultsDescriptorPath = ${jetty.base}/etc/webdefault-ee10.xml

Classpath: ee10
---------------
Version Information on 3 entries in the classpath.
Note: order presented here is how they would appear on the classpath.
      changes to the --module=name command line options will be reflected here.
 0:                    6.0.0 | ${jetty.home}/lib/jakarta.servlet-api-6.0.0.jar
 1:                   12.0.6 | ${jetty.home}/lib/jetty-ee10-servlet-12.0.6.jar
 2:                   12.0.6 | ${jetty.home}/lib/jetty-ee10-webapp-12.0.6.jar

Active XMLs: ee10
-----------------
 ${jetty.home}/etc/jetty-ee10-webapp.xml
 ${jetty.home}/etc/jetty-ee10-deploy.xml

The property values are not having their ${expr} entries expanded.
We can see this in the lines ...

Properties: Jetty
-----------------
 jetty.webapp.addServerClasses = org.eclipse.jetty.logging.,${jetty.home.uri}/lib/logging/,org.slf4j.

Properties: ee10
----------------
 jetty.deploy.defaultsDescriptorPath = ${jetty.base}/etc/webdefault-ee10.xml

[joakime]

A smaller example ...

[bases]$ mkdir issue-11408
[bases]$ cd issue-11408/
[issue-11408]$ java -jar ../../jetty-home-12.0.6/start.jar --add-module=http
INFO  : mkdir ${jetty.base}/start.d
INFO  : server          transitively enabled, ini template available with --add-modules=server
INFO  : logging-jetty   transitively enabled
INFO  : http            initialized in ${jetty.base}/start.d/http.ini
INFO  : resources       transitively enabled
INFO  : threadpool      transitively enabled, ini template available with --add-modules=threadpool
INFO  : logging/slf4j   dynamic dependency of logging-jetty
INFO  : bytebufferpool  transitively enabled, ini template available with --add-modules=bytebufferpool
INFO  : mkdir ${jetty.base}/resources
INFO  : copy ${jetty.home}/modules/logging/jetty/resources/jetty-logging.properties to ${jetty.base}/resources/jetty-logging.properties
INFO  : Base directory was modified
[issue-11408]$ echo 'key=${jetty.base}/foo' >> start.d/foo.ini
[issue-11408]$ cat start.d/foo.ini 
key=${jetty.base}/foo

[issue-11408]$ java -jar ../../jetty-home-12.0.6/start.jar --list-config | grep key
 key = ${jetty.base}/foo

[joakime]

This would be a new feature, as that kind of expansion isn't being performed in Jetty 9/10/11 either.

[issue-11408]$ java -jar ../../jetty-home-11.0.18/start.jar --list-config | grep key
 key = ${jetty.base}/foo

[issue-11408]$ java -jar ../../jetty-home-9.4.53.v20231009/start.jar --list-config | grep key
 key = ${jetty.base}/foo

So we either implement support for this kind of expansion, or revert the configurations in the *.mod files that rely on this behavior.

In Jetty 12.0.6, the following mod files would need to be fixed ...

$ grep -RHin '${' jetty-home-12.0.6/modules/ | grep =
jetty-home-12.0.6/modules/ee10-deploy.mod:22:# jetty.deploy.defaultsDescriptorPath=${jetty.base}/etc/webdefault-ee10.xml
jetty-home-12.0.6/modules/ee8-deploy.mod:22:# jetty.deploy.defaultsDescriptorPath=${jetty.base}/etc/webdefault-ee8.xml
jetty-home-12.0.6/modules/pid.mod:19:# jetty.pid=${jetty.base}/jetty.pid
jetty-home-12.0.6/modules/test-keystore.mod:28:jetty.webapp.addServerClasses+=,${jetty.base.uri}/lib/bouncycastle/
jetty-home-12.0.6/modules/sessions/infinispan/remote/infinispan-remote-serverclasses.mod:12:jetty.webapp.addServerClasses+=,${jetty.base.uri}/lib/infinispan/
jetty-home-12.0.6/modules/sessions/infinispan/embedded/infinispan-embedded-serverclasses.mod:13:jetty.webapp.addServerClasses+=,${jetty.base.uri}/lib/infinispan/
jetty-home-12.0.6/modules/state.mod:20:# jetty.state=${jetty.base}/jetty.state
jetty-home-12.0.6/modules/logging-jetty.mod:25:jetty.webapp.addServerClasses+=,${jetty.home.uri}/lib/logging/
jetty-home-12.0.6/modules/ee9-deploy.mod:22:# jetty.deploy.defaultsDescriptorPath=${jetty.base}/etc/webdefault-ee9.xml
jetty-home-12.0.6/modules/gcloud.mod:20:jetty.webapp.addServerClasses+=,${jetty.base.uri}/lib/gcloud/
jetty-home-12.0.6/modules/logging-jul-capture.mod:24:-Djava.util.logging.config.file=${jetty.base}/resources/java-util-logging.properties
jetty-home-12.0.6/modules/global-webapp-common.mod:19:# jetty.deploy.webappCommonConfig=${jetty.base}/etc/webapp-common.xml
jetty-home-12.0.6/modules/session-store-file.mod:22:jetty.session.file.storeDir=${jetty.base}/sessions
jetty-home-12.0.6/modules/requestlog.mod:31:# jetty.requestlog.filePath=${jetty.requestlog.dir}/yyyy_mm_dd.request.log
jetty-home-12.0.6/modules/logging-jul.mod:26:-Djava.util.logging.config.file=${jetty.base}/resources/java-util-logging.properties

[stroucki]

My workaround right now is to use use an absolute path for webdefaults-ee10.xml, the only place where this has been an issue for me so far. Could I use something better?

[joakime]

My workaround right now is to use use an absolute path for webdefaults-ee10.xml, the only place where this has been an issue for me so far. Could I use something better?

as long as your ${user.dir} is the same as ${jetty.base}, you could use ...

$ grep defaultsDescriptorPath start.d/ee10-deploy.ini
jetty.deploy.defaultsDescriptorPath=etc/webdefault-ee10.xml

that should work, as that path would then be relative to your ${user.dir} (aka PWD)

[stroucki]

Thanks,. I prefer to start in production by adapting the script in bin/jetty.sh to authbind and change user. Things are working for me now and I'll wait for 12.0.7.

[joakime]

Opened PR #12169 to fix this.
Turns out it was Environment property specific.
Core / Global properties did not have a problem.