Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproduce JavaSE-17 issues on JDK 11 #951

Closed
wants to merge 1 commit into from

Conversation

aumann
Copy link
Contributor

@aumann aumann commented May 7, 2022

This is only reproducing issue #950 when run locally with JAVA_HOME set to a JDK-11.

If you point me to a similar test I can try and complete the test setup at least (i.e. prepare for this test running, and running on Java 11).

@mickaelistria
Copy link
Contributor

As mentioned in the discussion, this story requires a toolchains.xml to define where to find Java 17. The test would need to include that.

@aumann
Copy link
Contributor Author

aumann commented May 7, 2022

where would I the jdkHome in the toolchains.xml point to? Locally I have this one, but the path will obviously not be valid for CI:

<?xml version="1.0" encoding="UTF8"?>
<toolchains>
  <toolchain>
    <type>jdk</type>
    <provides>
      <version>1.8</version>
      <id>JavaSE-1.8</id>
    </provides>
    <configuration>
      <jdkHome>/usr/lib/jvm/java-8-openjdk-amd64</jdkHome>
      <bootClassPath>
        <includes>
          <include>jre/lib/*</include>
          <include>jre/lib/ext/*</include>
          <include>jre/lib/endorsed/*</include>
        </includes>
      </bootClassPath>
    </configuration>
  </toolchain>
  <toolchain>
    <type>jdk</type>
    <provides>
      <version>17</version>
      <id>JavaSE-17</id>
    </provides>
    <configuration>
      <jdkHome>/home/aleaum/.sdkman/candidates/java/17.0.3-tem</jdkHome>
    </configuration>
  </toolchain>
</toolchains>

I'll try and find one in the other integration tests, if there are obvious other examples I hope I find them...

@aumann
Copy link
Contributor Author

aumann commented May 19, 2022

I forgot to re-ask / mention I did not find a similar toolchains setup. Can someone point me to a similar test (or mention the path I need to put into the toolchains file?)

@github-actions
Copy link

github-actions bot commented Jun 2, 2022

Unit Test Results

0 files  0 suites   0s ⏱️
0 tests 0 ✔️ 0 💤 0

Results for commit d734cf2.

@Bananeweizen
Copy link
Contributor

where would I the jdkHome in the toolchains.xml point to? Locally I have this one, but the path will obviously not be valid for CI:

We use this approach at work:

  • check in a complete maven toolchains.xml with the project, where the paths are just properties
  • inject the actual path of the JDK from the CI system into the file content via some groovy code (in the Jenkinsfile)
  • use --global-toolchains option in the maven call to use the modified file

I'm not sure what would be the best way to inject/overwrite the JDK path here with the Github actions. If all else fails, one could still write such a toolchains file via antcall or similar from within the build.

@laeubi
Copy link
Member

laeubi commented Jun 3, 2022

I'm not sure what would be the best way to inject/overwrite the JDK path here with the Github actions

Toolchains support property expansion, so there should be no need to modify the file itself and one can simply use ENV variables like here:

https://github.com/eclipse/tycho/blob/master/toolchains-gh.xml

@laeubi
Copy link
Member

laeubi commented Jun 15, 2022

@akurtakov as you are dedicated to Java 17 support, can you take a look here what might be the problem?

@laeubi
Copy link
Member

laeubi commented Jun 15, 2022

I forgot to re-ask / mention I did not find a similar toolchains setup. Can someone point me to a similar test (or mention the path I need to put into the toolchains file?)

Tycho GH actions already use the following toolchain: https://github.com/eclipse/tycho/blob/master/toolchains-gh.xml

@akurtakov
Copy link
Member

I honestly don't understand what the issue is here. Toolchain for Java-17 must be setup in order to compile using Java 17 APIs while running on Java 11 . Tycho itself will require Java 17 (#983) soon so I strongly advice switching the JVM driving the build to Java 17 and configure Java 11 in your toolchain. Announced at https://www.eclipse.org/lists/tycho-dev/msg02086.html and https://www.eclipse.org/lists/cross-project-issues-dev/msg19262.html .

@laeubi laeubi marked this pull request as ready for review June 15, 2022 07:57
@laeubi
Copy link
Member

laeubi commented Jun 15, 2022

@akurtakov as far as I understand the issue arise when tycho is running on a lower JVM (11 here) but code should be compiled with a higher one (17 here). Maybe @aumann can give better details.

@laeubi
Copy link
Member

laeubi commented Jun 15, 2022

/rebase

@aumann
Copy link
Contributor Author

aumann commented Jun 15, 2022

@akurtakov as far as I understand the issue arise when tycho is running on a lower JVM (11 here) but code should be compiled with a higher one (17 here). Maybe @aumann can give better details.

Yes, exactly.

I know that this can be worked around - still I expected this to work, and @mickaelistria confirmed my expectations here: #938 (comment) - that's why I've openend the issue.

@akurtakov
Copy link
Member

So you are running Tycho with Java 11, have toolchains configuration for Java17, tycho uses Java 17 and build fails ?

@akurtakov
Copy link
Member

Just to be sure, does it happen with Tycho 3.0.0-SNAPSHOT? I ask that question as if you're using old tycho version it will come with old ecj/jdt.core version which may not support Java 17 constructs yet.

@aumann
Copy link
Contributor Author

aumann commented Jun 15, 2022

So you are running Tycho with Java 11, have toolchains configuration for Java17, tycho uses Java 17 and build fails ?

Yes, that's right.

Just to be sure, does it happen with Tycho 3.0.0-SNAPSHOT? I ask that question as if you're using old tycho version it will come with old ecj/jdt.core version which may not support Java 17 constructs yet.

I can try today in the evening - until now, I've only tested upto 2.7.2

@laeubi
Copy link
Member

laeubi commented Jun 24, 2022

/rebase

@laeubi
Copy link
Member

laeubi commented Jun 24, 2022

@aumann can you please give exact steps to reproduce:

  1. What JDK use for running mvn (e.g. out put from mvn -V)
  2. What JDKs are in your toolchain.xml
  3. in what folder do sou issue what exact mvn command and what is the error you see.
  4. What exact Tycho version is used.

@aumann
Copy link
Contributor Author

aumann commented Jun 24, 2022

@aumann can you please give exact steps to reproduce:

1. What JDK use for running `mvn` (e.g. out put from `mvn -V`)
➜  compiler.jdt.bree.java17 git:(reproduceBree17On11Bug) ✗ mvn -version
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /home/aleaum/.sdkman/candidates/maven/current
Java version: 11.0.15, vendor: Eclipse Adoptium, runtime: /home/aleaum/.sdkman/candidates/java/11.0.15-tem
Default locale: de_DE, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-121-generic", arch: "amd64", family: "unix"

2. What JDKs are in your `toolchain.xml`
➜  compiler.jdt.bree.java17 git:(reproduceBree17On11Bug) ✗ cat ~/.m2/toolchains.xml 
<?xml version="1.0" encoding="UTF8"?>
<toolchains>
  <toolchain>
    <type>jdk</type>
    <provides>
      <version>1.8</version>
      <id>JavaSE-1.8</id>
    </provides>
    <configuration>
      <jdkHome>/usr/lib/jvm/java-8-openjdk-amd64</jdkHome>
      <bootClassPath>
        <includes>
          <include>jre/lib/*</include>
          <include>jre/lib/ext/*</include>
          <include>jre/lib/endorsed/*</include>
        </includes>
      </bootClassPath>
    </configuration>
  </toolchain>
  <toolchain>
    <type>jdk</type>
    <provides>
      <version>17</version>
      <id>JavaSE-17</id>
    </provides>
    <configuration>
      <jdkHome>/home/aleaum/.sdkman/candidates/java/17.0.3-tem</jdkHome>
    </configuration>
  </toolchain>
</toolchains>

3. in what folder do sou issue what exact `mvn` command and what is the error you see.

I'm using the project as committed here, and execute maven from the it-project root:

➜  compiler.jdt.bree.java17 git:(reproduceBree17On11Bug) ✗ pwd
/home/aleaum/Develop/oomph-eclipse/tycho-master/git/org.eclipse.tycho/tycho-its/projects/compiler.jdt.bree.java17

➜  compiler.jdt.bree.java17 git:(reproduceBree17On11Bug) ✗ mvn clean compile
Full output
➜  compiler.jdt.bree.java17 git:(reproduceBree17On11Bug) ✗ mvn clean compile   
[INFO] Scanning for projects...
[WARNING] No explicit target runtime environment configuration. Build is platform dependent.
[INFO] Resolving dependencies of MavenProject: tycho-its-project.compiler.jdt.bree.java17:bundle:1.0.0 @ /home/aleaum/Develop/oomph-eclipse/tycho-master/git/org.eclipse.tycho/tycho-its/projects/compiler.jdt.bree.java17/bundle/pom.xml
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] parent                                                             [pom]
[INFO] bundle                                                  [eclipse-plugin]
[INFO] 
[INFO] ---------< tycho-its-project.compiler.jdt.bree.java17:parent >----------
[INFO] Building parent 0.0.1-SNAPSHOT                                     [1/2]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ parent ---
[INFO] 
[INFO] ---------< tycho-its-project.compiler.jdt.bree.java17:bundle >----------
[INFO] Building bundle 1.0.0                                              [2/2]
[INFO] ---------------------------[ eclipse-plugin ]---------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ bundle ---
[INFO] Deleting /home/aleaum/Develop/oomph-eclipse/tycho-master/git/org.eclipse.tycho/tycho-its/projects/compiler.jdt.bree.java17/bundle/target
[INFO] 
[INFO] --- tycho-packaging-plugin:2.7.2:build-qualifier (default-build-qualifier) @ bundle ---
[INFO] The project's OSGi version is 1.0.0
[INFO] 
[INFO] --- tycho-packaging-plugin:2.7.2:validate-id (default-validate-id) @ bundle ---
[INFO] 
[INFO] --- tycho-packaging-plugin:2.7.2:validate-version (default-validate-version) @ bundle ---
[INFO] 
[INFO] --- target-platform-configuration:2.7.2:target-platform (default-target-platform) @ bundle ---
[INFO] 
[INFO] --- tycho-compiler-plugin:2.7.2:validate-classpath (default-validate-classpath) @ bundle ---
[INFO] Resolving class path of bundle...
[INFO] 
[INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ bundle ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/aleaum/Develop/oomph-eclipse/tycho-master/git/org.eclipse.tycho/tycho-its/projects/compiler.jdt.bree.java17/bundle/src/main/resources
[INFO] 
[INFO] --- tycho-compiler-plugin:2.7.2:compile (default-compile) @ bundle ---
[INFO] Compiling 1 source file to /home/aleaum/Develop/oomph-eclipse/tycho-master/git/org.eclipse.tycho/tycho-its/projects/compiler.jdt.bree.java17/bundle/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] parent 0.0.1-SNAPSHOT .............................. SUCCESS [  0.033 s]
[INFO] bundle 1.0.0 ....................................... FAILURE [  0.983 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.908 s
[INFO] Finished at: 2022-06-24T20:41:14+02:00
[INFO] ------------------------------------------------------------------------


> 4. What exact Tycho version is used.

2.7.2

@akurtakov
Copy link
Member

@laeubi is this one fixed now?

@akurtakov
Copy link
Member

@aumann Would you please sign https://www.eclipse.org/legal/ECA.php ? Without it we can not accept patch from you.

@laeubi
Copy link
Member

laeubi commented Jul 19, 2022

I don't think so @aumann has provided steps to reproduce but I can't tell if this is an issue in Tycho or EJC ...

@aumann
Copy link
Contributor Author

aumann commented Jul 21, 2022

@aumann Would you please sign https://www.eclipse.org/legal/ECA.php ? Without it we can not accept patch from you.

I've re-signed with the correct mail address.

@github-actions
Copy link

github-actions bot commented Aug 29, 2022

Test Results

324 files  324 suites   2h 20m 41s ⏱️
290 tests 286 ✔️ 4 💤 0
580 runs  571 ✔️ 9 💤 0

Results for commit 2fe1a59.

♻️ This comment has been updated with latest results.

@laeubi
Copy link
Member

laeubi commented Sep 15, 2022

@akurtakov any chance we can finish this for 3.0? Is this an issue with Tycho or with EJC?

Change-Id: Ib9180671e66d5c004b3b1083d0436cfcbae4d625
@akurtakov
Copy link
Member

Well, it's probably time to move Tycho to require Java 17 actually. The jdt bug that forced this revert is supposed to be fixed now.

@laeubi
Copy link
Member

laeubi commented Sep 15, 2022

Well, it's probably time to move Tycho to require Java 17 actually. The jdt bug that forced this revert is supposed to be fixed now.

I think we should first fix all Java 17 related issue to be sure people can still compile for older versions and never versions...

@akurtakov
Copy link
Member

This one happens when running on Java 11 but targetting Java 17 which would no longer be relevant if Tycho requires Java 17 as runtime JVM.

@laeubi
Copy link
Member

laeubi commented Sep 15, 2022

This one happens when running on Java 11 but targetting Java 17 which would no longer be relevant if Tycho requires Java 17 as runtime JVM.

But Java 19 and so on will be, so this should be fixed first before we move on to a never JVM that possible hides an issue with compiling with higher JVMs thath those the Tycho build is run with.

@akurtakov
Copy link
Member

akurtakov commented Sep 15, 2022

I highly doubt that is true (esp. without a reproducer). My look into jdt.core points to isJRE12Plus as the most likely cause so if requiring Java 17 I think this bug will not be hidden but the combination it happens to be no longer supported.

@laeubi
Copy link
Member

laeubi commented Sep 15, 2022

So if its a JDT Bug we should report it there.

@laeubi
Copy link
Member

laeubi commented Sep 15, 2022

@iloveeclipse can you help investigate if this should be supported by EJC or Tycho si doing something wrong?

@laeubi
Copy link
Member

laeubi commented Sep 15, 2022

I highly doubt that is true (esp. without a reproducer). My look into jdt.core points to isJRE12Plus as the most likely cause so if requiring Java 17 I think this bug will not be hidden but the combination it happens to be no longer supported.

What I don't get is that the code should actually be compiled with the Java 17 JVM so why should it matter what is the "Tycho jvm"? Because I can happily run my eclipse with java 11 and use j17 jvm to compile ...

@laeubi
Copy link
Member

laeubi commented Sep 15, 2022

@aumann can you rebase this on top of current master branch?

@laeubi
Copy link
Member

laeubi commented Sep 15, 2022

@aumann can you rebase this on top of current master branch?

Forget about that... I have already rebased this :-)

But it seems the test now succeeds, so if jenkins also succeeds I think we can merge this as fixed?

@iloveeclipse
Copy link

can you help investigate if this should be supported by EJC or Tycho si doing something wrong?

I'm not a maven expert and have no idea how translate maven config to the ECJ command line.

Can you please update bug description and explain what is the problem and in particular, what part of the problem is related to ECJ and how? Ideally with a simple example input arguments / ecj command line / observed output / expected output etc.

@akurtakov
Copy link
Member

@aumann can you rebase this on top of current master branch?

Forget about that... I have already rebased this :-)

But it seems the test now succeeds, so if jenkins also succeeds I think we can merge this as fixed?

There is no test run. This PR is just a project to use while testing locally.

@akurtakov
Copy link
Member

akurtakov commented Sep 15, 2022

IMHO compiling Java 17 while running Tycho with Java 11 shouldn't even be smth that we consider supporting. If one needs newer JVM features, one should run Tycho with that or newer. I at least don't plan to spend time on it.

@laeubi
Copy link
Member

laeubi commented Sep 15, 2022

IMHO compiling Java 17 while running Tycho with Java 11 shouldn't even be smth that we consider supporting.

Why not? Maven even support compiling Java 17 when running with Java 8, so I don't see any relation here...

And as @mickaelistria also indicated it "should" work, there must be something wrong in the setup of the project (I don't see any?) or a bug in Tycho, or maybe something wrong in EJC ...

@laeubi
Copy link
Member

laeubi commented Sep 15, 2022

@aumann can you integrate the example into the integration test suite?

@akurtakov
Copy link
Member

IMHO compiling Java 17 while running Tycho with Java 11 shouldn't even be smth that we consider supporting.

Why not? Maven even support compiling Java 17 when running with Java 8, so I don't see any relation here...

It just doesn't climb enough on the important bug/feature scale. If your code requires newer JVM features it should be fine to run your tool with that version too.

@mickaelistria
Copy link
Contributor

It looks like the JDT compiler doesn't support this case anyway:

mistria@localhost:~/git/org.eclipse.tycho/tycho-its/projects/compiler.jdt.bree.java17$ /usr/lib/jvm/java-11/bin/java -jar ~/apps/ecj-4.25.jar /home/mistria/git/org.eclipse.tycho/tycho-its/projects/compiler.jdt.bree.java17/bundle/src/org/eclipse/tycho/test/compiler/Main.java -target 17 -source 17
----------
1. ERROR in /home/mistria/git/org.eclipse.tycho/tycho-its/projects/compiler.jdt.bree.java17/bundle/src/org/eclipse/tycho/test/compiler/Main.java (at line 1)
	package org.eclipse.tycho.test.compiler;
	^
java.lang.Record cannot be resolved to a type
----------
2. ERROR in /home/mistria/git/org.eclipse.tycho/tycho-its/projects/compiler.jdt.bree.java17/bundle/src/org/eclipse/tycho/test/compiler/Main.java (at line 8)
	System.out.println(List.of("a", "b").stream().toList());
	                                              ^^^^^^
The method toList() is undefined for the type Stream<String>
----------
2 problems (2 errors)

@iloveeclipse is this intended? I personally don't mind if it's a rule that to compile for X, you need the compiler to run a Y>=X, it makes a lot of sense.
If it is so, then we just need to clarify in Tycho doc, and this issue becomes invalid.

@iloveeclipse
Copy link

I honestly don't know, but I would expect that ecj should be able to compile with Java 17 target if running on 11 if it would know where Java 17 SDK files are.
Inside IDE that is given by the workspace config I believe, so we could find Java 17 classes, but not sure how to specify that on command line.

Other way around it works because modern Java 9+ SDK bring stubs of all older releases in ct.sym file, so compiler knows where to find "Record" etc.

@mickaelistria
Copy link
Contributor

Thanks for the hint @iloveeclipse , I found that passing --system works, eg $ /usr/lib/jvm/java-11/bin/java -jar ~/apps/ecj-4.25.jar ~/sandbox/Main.java -target 17 -source 17 --system /usr/lib/jvm/java-17-openjdk-17.0.4.0.8-1.fc36.x86_64/ is successful.
It's surprising we see this issue only now, but it seems relatively easy to fix!
`

@laeubi
Copy link
Member

laeubi commented Sep 16, 2022

I honestly don't know, but I would expect that ecj should be able to compile with Java 17 target if running on 11 if it would know where Java 17 SDK files are.

At least as mentioned I'm running my IDE with Java 11 and can compile for Java 17 and Java 19 without a problem

Inside IDE that is given by the workspace config I believe, so we could find Java 17 classes, but not sure how to specify that on command line.

That's what the mentioned toolchains.xml of maven are meant for.

@laeubi
Copy link
Member

laeubi commented Sep 16, 2022

By the way, Tycho has even two modes:

  1. compile embedded
  2. compile in extra process

so even if ecj would not work for JVMs < running JVM Tycho could still fall back to "ompile in extra process" with the given (higher) JVM from toolchains and I think that is how maven-compiler works and why (maven) people complain that it is slow to use toolchains :-)

S the ability of JDT to "cross-compile" using different higher/lower jvms is really a great benefit!

@mickaelistria
Copy link
Contributor

Unfortunately, the compile in extra process mode (fork=true on tycho-compiler-plugin) does not work. However, a custom JAVA_HOME is supposed to be provided for that case, and should result into something similar. So I tried something

$ JAVA_HOME=/usr/lib/jvm/java-17-openjdk-17.0.4.0.8-1.fc36.x86_64 /usr/lib/jvm/java-11/bin/java -jar ~/apps/ecj-4.25.jar ~/sandbox/Main.java -target 17 -source 17 --system /usr/lib/jvm/java-17-openjdk-17.0.4.0.8-1.fc36.x86_64/
# success with --system explicitly set


$ JAVA_HOME=/usr/lib/jvm/java-17-openjdk-17.0.4.0.8-1.fc36.x86_64 /usr/lib/jvm/java-11/bin/java -jar ~/apps/ecj-4.25.jar ~/sandbox/Main.java -target 17 -source 17 ----------
----------
1. ERROR in /home/mistria/sandbox/Main.java (at line 1)
	import java.util.List;
	^
java.lang.Record cannot be resolved to a type
----------
2. ERROR in /home/mistria/sandbox/Main.java (at line 6)
	System.out.println(List.of("a", "b").stream().toList());
	                                              ^^^^^^
The method toList() is undefined for the type Stream<String>
----------
2 problems (2 errors)

So this demonstrates that -unlike javac- the JAVA_HOME setting is not sufficient for ECJ to use a different --system (@iloveeclipse do you think it's a bug that should be reported to JDT?).
Mitigations in Tycho can be (not exclusively)

  • Implement the fork properly and use it when toolchain != system JRE
  • pass the --system (but still system JRE is actually used for the compilation)

@laeubi
Copy link
Member

laeubi commented Sep 16, 2022

  • pass the --system (but still system JRE is actually used for the compilation)

Yes I think that's missing and we had some strange behavior in the past, so I think this actually is a good show-case to improve Tycho then!

@mickaelistria
Copy link
Contributor

unfortunately, this test cannot be easily made to execute properly on CI, so I'm closing it as not actionable.
#1351 was merged and should contain a fix. Please give it a try within a couple of hours to verify it fixes your issue (spoiler: I tried it ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants