Running embedded jetty in my Application through IntelliJ gives no error, but contexts are not running. What could be the reason?

[avin15cs]

Jetty version
9.4.44.v20210927
Java version
java-11-openjdk-11.0.7.10-1.windows.redhat.x86_64
Question
I am using jetty in my application where it is embedded. When I try to access configured contexts while running my project through Services, I am able to get proper response.
But same thing when I run through intelliJ, only http://localhost:9096/ is accessible and for rest URLs I get internal server error 500.

What could have gone wrong here?

[sbordet]

Difficult to say, too little information. Are you using IntelliJ-Jetty integration, or just running the Jetty code?

[avin15cs]

I am running the jetty code, which is written in my application. I wasn't sure what exactly needs to be shared, thats why I didnt share anything. I am using code suggested by Joakime here:
https://stackoverflow.com/questions/69763353/how-to-enable-embedded-jetty-9-in-ssl-mode-through-xml-configuration

[sbordet]

There are multiple snippets of code in that SO question/answer.
If you post what you're doing, we might be able to help.

[avin15cs]

Hi @sbordet
I have used the following code:

Code used:

loadJettyXmlConfigurations();
idMap = configure(xmls);
myServer = (Server) idMap.get("Server");
myServer.start();
    private void loadJettyXmlConfigurations() {
        xmls = new ArrayList<>();
        Path homeXmlPath = Paths.get(myJettyDir+"/home");
        xmls.add(new PathResource(homeXmlPath.resolve("jetty-bytebufferpool.xml")));
        xmls.add(new PathResource(homeXmlPath.resolve("jetty-threadpool.xml")));
        xmls.add(new PathResource(homeXmlPath.resolve("jetty.xml")));
        xmls.add(new PathResource(homeXmlPath.resolve("jetty-http.xml")));
        if (enableHttps)
        {
            xmls.add(new PathResource(homeXmlPath.resolve("jetty-ssl.xml")));
            xmls.add(new PathResource(homeXmlPath.resolve("jetty-ssl-context.xml")));
            xmls.add(new PathResource(homeXmlPath.resolve("jetty-https.xml")));
        }
        xmls.add(new PathResource(homeXmlPath.resolve("jetty-customrequestlog.xml")));

        Path customBasePath = Paths.get(myJettyDir+"/base");
        xmls.add(new PathResource(customBasePath.resolve("context.xml")));
    }

    /**
     * Configure for the list of XML Resources and Properties.
     *
     * @param xmls the xml resources (in order of execution)
     * @return the ID Map of configured objects (key is the id name in the XML, and the value is configured object)
     */
    private Map<String, Object> configure(List<Resource> xmls)
    {
        String basePath = myJettyDir+"/base";
        Path customBasePath = Paths.get(basePath);
        Map<String, String> properties = loadProperties(customBasePath.resolve("config.properties"));
        properties.put("jetty.base", basePath);
        Map<String, Object> idMap = new HashMap<>();

        try {
            for (Resource xmlResource : xmls) {
                XmlConfiguration configuration = new XmlConfiguration(xmlResource);
                configuration.getIdMap().putAll(idMap);
                configuration.getProperties().putAll(properties);
                configuration.configure();
                idMap.putAll(configuration.getIdMap());
            }
        } catch(Exception e) {
            _mpSrvcLogger.info(e.getMessage());
        }

        return idMap;
    }

    private Map<String, String> loadProperties(Path path)
    {
        Properties properties = new Properties();
        try (BufferedReader reader = Files.newBufferedReader(path, StandardCharsets.UTF_8))
        {
            properties.load(reader);
        } catch (IOException e) {
            _mpSrvcLogger.info(e.getMessage());
        }

        return properties.entrySet().stream().collect(
                Collectors.toMap(
                        e -> String.valueOf(e.getKey()),
                        e -> String.valueOf(e.getValue()),
                        (prev, next) -> next, HashMap::new
                ));
    }

I have attached the jetty trace here
jetty_trace.txt

and config.properties.
config.properties.txt

I am getting HTTP 500 server error, on running through intelliJ, but it works fine when I run it using Services.

[joakime]

Add this line and run it again.

myServer = (Server) idMap.get("Server");
myServer.setDumpAfterStart(true); // <-- this line
myServer.start();

The output (to STDERR) will include your server state dump.

[avin15cs]

Hi Joakime,
Please find the jetty_trace_log: jetty_newTrace.txt

[joakime]

You have a started server.

It's listening on port 9096 on every network interface on that machine.
You have a WebApp on context path /cts deployed from directory file:///C:/Documentum/CTS/jetty/base/.
That webapp has 10 servlets deployed against 10 different url-patterns.

Which means you can access your webapp from http://localhost:9096/cts/

Using just http://localhost:9096/ would result in a 404 error, as there is no bound webapp on the root context.

It should be noted as well, that your WebApp doesn't have a normal default servlet, it's configured for [/]=>org.eclipse.jetty.servlet.ServletHandler$Default404Servlet-2a0e2c85
This likely means you have messed up your default descriptor configuration.

[avin15cs]

Hi Joakime,
Yes, you are correct, my webApp runs from http://localhost:9096/cts/

I get proper response somehow, when I run the project through Windows Services. but while I run using intelliJ dev setup I get internal server error 500 which is little weird.

It should be noted as well, that your WebApp doesn't have a normal default servlet, it's configured for [/]=>org.eclipse.jetty.servlet.ServletHandler$Default404Servlet-2a0e2c85
This likely means you have messed up your default descriptor configuration.
What has gone wrong here? How can I correct it?

[joakime]

I get internal server error 500 which is little weird.

What do the server logs say for this 500 error?
Jetty will produce a stacktrace if it's the one creating the 500 error.
If something else is creating the 500 error, then it's up to that something else to produce a reason why.

What has gone wrong here? How can I correct it?

In your example code above you have a line that says ...

xmls.add(new PathResource(customBasePath.resolve("context.xml")));

Is that your /cts context declaration?
If so, can you show us the contents of that XML?

[avin15cs]

Hi Joakime,

500 server error comes when I hit any of the deployed context:
http://localhost:9096/cts/monitor/
http://localhost:9096/cts/getOccupancy/

Pleasze find the stack trace here:
jetty_newTrace.txt and log.txt
ws.request_2022_01_18.log

I dont see any difference with the one shared earlier.

Please find the context.xml here
context.xml.txt

[joakime]

500 server error comes when I hit any of the deployed context:
http://localhost:9096/cts/monitor/
http://localhost:9096/cts/getOccupancy/

Pleasze find the stack trace here:
jetty_newTrace.txt and log.txt
ws.request_2022_01_18.log

The ws.request_2022_01_18.log is your request log, that won't be very useful to troubleshoot this, ignore it for now.

The jetty_newTrace.txt is what I'm looking for, it looks like you have DEBUG enabled for everything Jetty, which is good and should help.
But there's no evidence of any attempted request on those URLs that produced the 500 error for you.
Either the request didn't reach that instance of Jetty, or you copy/pasted/attached the log before you attempted the requests.

If you make the requests that result in a 500 error, the DEBUG logs should at least indicate the incoming request, the parsing of the request, it's path through Jetty to your Servlet context, and the reasons for the 500 error, the generation of the response, the sending of the response, and eventual completion of the response.
I see nothing like that in those logs.

I dont see any difference with the one shared earlier.

Please find the context.xml here
context.xml.txt

Ah, you are directly declaring a ServletContextHandler.
Don't worry about the default descriptor, DefaultServlet, or org.eclipse.jetty.servlet.ServletHandler$Default404Servlet then.
The details in the Server Dump are accurate and appropriate for your configuration.

[joakime]

Seeing as you are running on Windows, make sure you don't have another process listening on port 9096 that is taking your requests. (this would explain the lack of DEBUG logs on the incoming requests)

Windows allows this kind of nonsense setup (more than 1 process listening on the same port), no other OS does.

[avin15cs]

Seeing as you are running on Windows, make sure you don't have another process listening on port 9096 that is taking your requests. (this would explain the lack of DEBUG logs on the incoming requests)

Windows allows this kind of nonsense setup (more than 1 process listening on the same port), no other OS does.

I have checked again and again, no other services are running on port 9096. Do you have any suggestion for me to catch and log record whenever any request is sent to defined URL?

Also, Is it possible to define separate contexthandler for each context? Can you please help me with that using shared context.xml?

[joakime]

Stop your server, and try hitting your URLs again.
Do they respond? If so, then you have another service on that port (or a proxy, or security software, etc) handling the request for you, not your server.

If you get a connection error from the browser then you likely have no service present.
If using chrome, it should say "This site can't be reached" with an "ERR_ADDRESS_UNREACHABLE" result, if you see anything else that means you have some service responding with a formal http response.

Next, ensure your logging is setup to DEBUG for everything in Jetty (the named logger org.eclipse.jetty should be at level DEBUG), your past examples of jetty_newTrace.txt seems to indicate that you have already done this, but just confirm it, as the lack of logs for the requests is a sign something more fundamental is going on. (example, is this a user rights issue, and your server startup is switching users post-start, where the new user is unable to write logs?)
Now run your service, hit it with a browser, and a command line, and any other actions you can come up with, heck try writing a unit test that uses the simple java java.net.http.HttpClient to access anything on the server (even known bad resources).

You need to ensure that the requests reach your server first. (The logs should indicate this, and those DEBUG logs will be crucial to help you)
If they don't reach your server, then there's not much you can do configuration wise on Jetty to fix that.
Start looking at your proxy settings, your security software, etc (according to the logs you have sent so far, something seems to be preventing the request from reaching your server).

Try looking at the complete request/response of the failed 500 attempts, including all headers and body content, there could be something revealing about that information. (I would expect the response headers to indicate that they were produced by Jetty, and the response body content to include the stacktrace explaining the 500 error for example)

[avin15cs]

Hi Joakime,

This seems to be tricky!
When I stopped jetty which was running from intelliJ, I got ERR_CONNECTION_REFUSED error. This is the screenshot:

When project was running through intelliJ, I got following response:

This is the screenshot, when jetty was running through Windows Services:

When I tried with sample project, jetty was running fine using intelliJ. I am not sure what is going wrong when I use it in my project.

Can we log jetty Trace while running it using Services? I use log4j2 in my project and mention appender in this way:

appender.F43.type=RollingFile
appender.F43.name=IMWAppender
appender.F43.fileName=C\:\\Documentum\\CTS\\logs\\IMAGE3_log.txt
appender.F43.filePattern=C\:\\Documentum\\CTS\\logs\\IMAGE3_log.%d{yyyy-MM-dd}-%i
appender.F43.layout.type=PatternLayout
appender.F43.layout.pattern=%d{ABSOLUTE} %5p [%t] %c - %m%n
appender.F43.policies.type=Policies
appender.F43.policies.time.type=TimeBasedTriggeringPolicy
appender.F43.policies.time.interval=1
appender.F43.policies.time.modulate=true
appender.F43.policies.size.type=SizeBasedTriggeringPolicy
appender.F43.policies.size.size=10MB
appender.F43.strategy.type=DefaultRolloverStrategy
appender.F43.strategy.max=5
logger.R43.name=com.documentum.cts.plugin.imw
logger.R43.level=INFO
logger.R43.additivity=false
logger.R43.appenderRef.F43.ref=IMWAppender

How can I create log file for jetty? I will compare the difference between two jetty traces, may be we get some hint.

[avin15cs]

Or, Do I need to pass any VM option or set any property to make it work?
I have set System.setProperty("org.eclipse.jetty.LEVEL", "DEBUG"); for enabling debug logs trace.

[joakime]

In the 500 response, look at the response body content.
It should have the stacktrace we are looking for.

[avin15cs]

Hi @joakime,
Response body content is empty. Can you help me with logging jetty trace while using services so that I cam find out if there is any error.

[joakime]

@avin15cs no response body content means that Jetty is not the one producing the 500 error for you.
Your application code is producing that 500 error.

[avin15cs]

Yes, That might be possibly true, but the jetty stack trace is not able to catch it and idk how through Windows services, this error doesn't come into picture.

[avin15cs]

Hi @joakime ,
Thanks for help!

Your application code is producing that 500 error.

Found out the reason behind this issue, it was not working because I was using older version of servlet-api, removing that worked!

[avin15cs]

@joakime ,
One question: Since, Jetty-all-uber.jar already have javax.servlet, is it required to add servlet api in dependency?

[joakime]

The jetty-all uber jar is not meant to be used by another application as a dependency.

It only exists to help with the tutorial section in the documentation. (it should have been called jetty-tutorial, but that ship has long sailed)
It's meant to be used by the command line only.
It has not, and cannot, contain all of Jetty, that's an impossibility.

See:

• Remove jetty-all from Jetty 10 #5317
• https://www.eclipse.org//lists/jetty-users/msg06030.html

[avin15cs]

Hi @joakime ,

In that case, How can I run embedded jetty without using jetty-all-uber.jar?

[joakime]

Using the normal dependencies and a build tool (like maven, gradle, scala, ant+ivy, grape, leiningen, buildr, bazel, etc) to obtain the top level components you need while letting the build tool pull all of the dependencies for those top level components.

Note, it is not possible to have "all of jetty" in your project, there are many components that require you to choose which implementation you want to use.

Based on your past questions I would start with ...

• https://search.maven.org/artifact/org.eclipse.jetty/jetty-webapp
• https://search.maven.org/artifact/org.eclipse.jetty/jetty-server

Do a build, and then, only after you discover a missing dependency from that build add the extra components you need.

• https://search.maven.org/search?q=org.eclipse.jetty

[avin15cs]

Yes I was able to get the dependencies through Maven tool:

[INFO] org.example:JettyTest:jar:1.0-SNAPSHOT
[INFO] +- commons-lang:commons-lang:jar:2.6:compile
[INFO] +- org.eclipse.jetty:jetty-util:jar:9.4.44.v20210927:compile
[INFO] +- org.eclipse.jetty:jetty-webapp:jar:9.4.44.v20210927:compile
[INFO] | - org.eclipse.jetty:jetty-servlet:jar:9.4.44.v20210927:compile
[INFO] | +- org.eclipse.jetty:jetty-security:jar:9.4.44.v20210927:compile
[INFO] | - org.eclipse.jetty:jetty-util-ajax:jar:9.4.44.v20210927:compile
[INFO] +- org.eclipse.jetty:jetty-server:jar:9.4.44.v20210927:compile
[INFO] | +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] | +- org.eclipse.jetty:jetty-http:jar:9.4.44.v20210927:compile
[INFO] | - org.eclipse.jetty:jetty-io:jar:9.4.44.v20210927:compile
[INFO] - org.eclipse.jetty:jetty-xml:jar:9.4.44.v20210927:compile

Thanks for help, @joakime!