Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add jbang jdk install <version> <path> #1252

Merged

Conversation

nandorholozsnyak
Copy link
Contributor

  • Ability to link pre-installed JDK directory to JBang
  • Documentation update

- Ability to link pre-installed JDK directory to JBang
- Documentation update
@codecov
Copy link

codecov bot commented Feb 23, 2022

Codecov Report

Merging #1252 (2b0d8fd) into main (8e77cef) will increase coverage by 0.73%.
The diff coverage is 83.87%.

Impacted file tree graph

@@             Coverage Diff              @@
##               main    #1252      +/-   ##
============================================
+ Coverage     58.28%   59.01%   +0.73%     
- Complexity     1077     1139      +62     
============================================
  Files            86       96      +10     
  Lines          5818     5995     +177     
  Branches        996      996              
============================================
+ Hits           3391     3538     +147     
- Misses         1925     1950      +25     
- Partials        502      507       +5     
Flag Coverage Δ
Linux 57.78% <83.87%> (+0.71%) ⬆️
Windows 57.83% <83.87%> (+0.76%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/main/java/dev/jbang/util/JavaUtil.java 51.19% <0.00%> (-0.62%) ⬇️
src/main/java/dev/jbang/cli/Jdk.java 25.67% <33.33%> (+2.06%) ⬆️
src/main/java/dev/jbang/net/JdkManager.java 64.44% <96.15%> (+6.84%) ⬆️
src/main/java/dev/jbang/cli/Build.java 75.00% <0.00%> (-14.48%) ⬇️
src/main/java/dev/jbang/source/ResourceRef.java 45.28% <0.00%> (-5.67%) ⬇️
src/main/java/dev/jbang/cli/Run.java 67.56% <0.00%> (-2.97%) ⬇️
src/main/java/dev/jbang/cli/Edit.java 46.95% <0.00%> (-1.05%) ⬇️
...c/main/java/dev/jbang/cli/DependencyInfoMixin.java 100.00% <0.00%> (ø)
src/main/java/dev/jbang/source/ScriptSource.java
... and 27 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e77cef...2b0d8fd. Read the comment docs.

Copy link
Contributor

@quintesse quintesse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good in general! Good job!

@@ -56,6 +56,9 @@ Installs a JDK.
_version_::
The version to install

_[existingJdkPath]_::
Already existing JDK's path on the filesystem. If path is given, its version will be checked against the incoming one, if not the same, the process will not finish successfully.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A minor thing but I think it's best to rewrite this using a positive view. Eg. ".. and if the same, the given JDK will be added successfully". (The use of "added" is somewhat vague and ambiguous, perhaps there is a better word?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added could be linked maybe?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"linked" is good 👍

* @param path path to the pre-installed JDK.
* @param version requested version to link.
*/
public static void linkToExistingDirectory(String path, int version) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: I'd prefer "linkToExistingJdk"

if (Files.exists(jdkPath)) {
Util.verboseMsg("JBang managed JDK already exists, must be deleted to make sure linking works");
try {
FileUtils.deleteDirectory(jdkPath.toFile());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't use this, use our Util.deletePath(), that already handles any errors (or you can tell it to be quiet).

@@ -203,6 +207,40 @@ public static void uninstallJdk(int version) {
}
}

/**
* Links JBang JDK folder to an already existing JDK path with a soft link. It
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't specify "soft", because the linking code will actually create a hard link if it can't create a soft link for some reason.

src/main/java/dev/jbang/cli/Jdk.java Outdated Show resolved Hide resolved
Util.infoMsg("JDK " + version + " has been linked to: " + linkedJdkPath);
} else {
throw new ExitException(EXIT_INVALID_INPUT, "Java version in given path: " + path
+ " is not having the requested version " + version + ", please provide a proper path");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

say " is X, which does not match the requested version Y". Remove the "please provide..." part.

Copy link
Contributor Author

@nandorholozsnyak nandorholozsnyak Feb 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may not be able to infer the X value, cause this path could point to anywhere, it can have no Java inside it, I can do an evaluation and if it does not work then fallback to the original message if it is okay.

@@ -81,6 +81,7 @@ private static int determineJavaVersion(Path javaCmd) {
String output = Util.runCommand(javaCmd.toString(), "-version");
int version = parseJavaOutput(output);
if (version == 0) {
Util.verboseMsg("Version is 0, reading it from java.version property");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

say something like "Version could not be determined from: '$javaCmd -version', trying 'java.version' property"

@@ -104,7 +105,7 @@ public static Path getJdkHome() {
public static int parseJavaOutput(String output) {
if (output != null) {
Matcher m = javaVersionPattern.matcher(output);
if (m.find() && m.groupCount() == 2) {
if (m.find() && m.groupCount() == 1) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That does indeed seem to have been wrong all this time, good catch!

@quintesse
Copy link
Contributor

Oh and add "Fixes #1248" to the end of the commit message, then GitHub will automatically close the issue when your PR gets merged (you probably know that and just forgot 😄)

@nandorholozsnyak
Copy link
Contributor Author

Mentioned things are fixed, I'm coming soon with the tests.

@nandorholozsnyak
Copy link
Contributor Author

Coverage should be a bit better now.

@nandorholozsnyak
Copy link
Contributor Author

Any cool idea how to fix this error happening with the Windows build?

TestJdk > testJdkInstallWithLinkingToExistingJdkPathWhenJBangManagedVersionDoesNotExist(File) FAILED
    java.io.IOException at TempDirectory.java:262

TestJdk > testJdkInstallWithLinkingToExistingJdkPathWhenJBangManagedVersionExistsAndInstallIsForced(File) FAILED
    java.io.IOException at TempDirectory.java:262
^"C:\Users\RUNNER~1\AppData\Local\Temp\junit5745621985287379323\jbang\cache\jdks\11\bin\java.exe^" -Dfoo=bar ^"-Dbar=aap^ noot^ mies^" -classpath ^"D:\a\jbang\jbang\build\classes\java\test\itests\hellojar.jar^" helloworld
java -Dfoo=bar ^"-Dbar=aap^ noot^ mies^" -classpath ^"C:\Users\RUNNER~1\AppData\Local\Temp\junit8100379350[68](https://github.com/jbangdev/jbang/runs/5324909962?check_suite_focus=true#step:6:68)2[69](https://github.com/jbangdev/jbang/runs/5324909962?check_suite_focus=true#step:6:69)4745\jbang\cache\jars\helloworld.java.6c29e59df0efda6a7a9783b1de6b823dc03[71](https://github.com/jbangdev/jbang/runs/5324909962?check_suite_focus=true#step:6:71)000e3c8d32b2b143a430c20900b.jar^" ^"-XX:ArchiveClassesAtExit=C:\Users\RUNNER~1\AppData\Local\Temp\junit8100379350682694[74](https://github.com/jbangdev/jbang/runs/5324909962?check_suite_focus=true#step:6:74)5\jbang\cache\jars\helloworld.java.6c29e59df0efda6a7a9783b1de6b823dc0371000e3c8d32b2b143a430c20900b.jar.jsa^" helloworld

TestJdkManager > testResolveJavaVersionFromPathWhenJavaVersionHasNewVersioning(File) FAILED
    java.io.IOException at TempDirectory.java:262

TestJdkManager > testResolveJavaVersionFromPathWhenJavaVersionHasOldVersioning(File) FAILED
    java.io.IOException at TempDirectory.java:262

@quintesse
Copy link
Contributor

Any cool idea ...

Is that error for real? Because that path looks very wrong! There are URLs and what not inside of it.

@nandorholozsnyak
Copy link
Contributor Author

Hm, on a Windows based laptop the build is fine. Any guesses to the TempDir?
Which path is not looking good @quintesse ?

@quintesse
Copy link
Contributor

This one :-)

"C:\Users\RUNNER~1\AppData\Local\Temp\junit8100379350[68](https://github.com/jbangdev/jbang/runs/5324909962?check_suite_focus=true#step:6:68)2[69](https://github.com/jbangdev/jbang/runs/5324909962?check_suite_focus=true#step:6:69)4745\jbang\cache\jars\helloworld.java.6c29e59df0efda6a7a9783b1de6b823dc03[71](https://github.com/jbangdev/jbang/runs/5324909962?check_suite_focus=true#step:6:71)000e3c8d32b2b143a430c20900b.jar^" ^"-XX:ArchiveClassesAtExit=C:\Users\RUNNER~1\AppData\Local\Temp\junit8100379350682694[74](https://github.com/jbangdev/jbang/runs/5324909962?check_suite_focus=true#step:6:74)5\jbang\cache\jars\helloworld.java.6c29e59df0efda6a7a9783b1de6b823dc0371000e3c8d32b2b143a430c20900b.jar.jsa^"

@nandorholozsnyak
Copy link
Contributor Author

Sorry I still had no time to finalize the wrong tests, I ran the tests on Linux and on Windows too. I think there were some reasons why this part was not having tests :D @quintesse do you have any tips to resolve the problem? What could be causing the problem on the CI?

@quintesse
Copy link
Contributor

quintesse commented Mar 3, 2022

Well actually I think the weird path must have been a copy&paste issue on your part, because if I look at the logs it looks normal:

TestJdk > testJdkInstallWithLinkingToExistingJdkPathWhenJBangManagedVersionDoesNotExist(File) FAILED
    java.io.IOException at TempDirectory.java:262

TestJdk > testJdkInstallWithLinkingToExistingJdkPathWithDifferentVersion(File) FAILED
    java.io.IOException at TempDirectory.java:262

TestJdk > testJdkInstallWithLinkingToExistingJdkPathWhenJBangManagedVersionExistsAndInstallIsForced(File) FAILED
    java.io.IOException at TempDirectory.java:262
^"C:\Users\RUNNER~1\AppData\Local\Temp\junit4891103843310719245\jbang\cache\jdks\11\bin\java.exe^" -Dfoo=bar ^"-Dbar=aap^ noot^ mies^" -classpath ^"D:\a\jbang\jbang\build\classes\java\test\itests\hellojar.jar^" helloworld
java -Dfoo=bar ^"-Dbar=aap^ noot^ mies^" -classpath ^"C:\Users\RUNNER~1\AppData\Local\Temp\junit8179139871602089251\jbang\cache\jars\helloworld.java.6c29e59df0efda6a7a9783b1de6b823dc0371000e3c8d32b2b143a430c20900b.jar^" ^"-XX:ArchiveClassesAtExit=C:\Users\RUNNER~1\AppData\Local\Temp\junit8179139871602089251\jbang\cache\jars\helloworld.java.6c29e59df0efda6a7a9783b1de6b823dc0371000e3c8d32b2b143a430c20900b.jar.jsa^" helloworld

TestJdkManager > testResolveJavaVersionFromPathWhenReleaseFileHasNoProperJavaVersion(File) FAILED
    java.io.IOException at TempDirectory.java:262

TestJdkManager > testResolveJavaVersionFromPathWhenJavaVersionHasNewVersioning(File) FAILED
    java.io.IOException at TempDirectory.java:262

TestJdkManager > testResolveJavaVersionFromPathWhenJavaVersionHasOldVersioning(File) FAILED
    java.io.IOException at TempDirectory.java:262

Edit: actually it's an issue with copy&pasting from the logs. Somewhere GitHub decided to insert links for things it thinks are commit ids. ANyway, I cleaned it up and the above logs are displayed correctly now.

Edit2: Haven't had time to look at why it's failing though.

- Disabling tests on Windows that are using symlinks
@nandorholozsnyak
Copy link
Contributor Author

On windows the symlinks are a bit clunky but I think you are aware of that, so I used @DisabledOnOs(OS.WINDOWS) on those tests that are creating symlinks.
This solves all other failings, that are really suprising.
@quintesse do I still need something to fix on it?

@maxandersen
Copy link
Collaborator

I thought we do symbolic linking for jdk install testing elsewhere so what is special about this usecase?

@nandorholozsnyak
Copy link
Contributor Author

I thought we do symbolic linking for jdk install testing elsewhere so what is special about this usecase?

I do not really see any tests that are installing the JDKs (of course there are but not in the TestJdk class). What I could basically debug is that the creation of hard links on Windows works only if you link one file, not a whole directory to another (like in the TestEdit test cases).
On my local Windows environment AccessDeniedExceptions are being thrown, but on the CI IOExceptions and not only for the now disabled 2 test cases but another 4. I was assuming that the TempDir annotation and its lifecycle is doing some magic behind it but as I read it's javadoc and it is clear that each method is having its own temporary directory, cause we have no BeforeAll only BeforeEach method in the BaseTest.
I assume that in the CI when the build is running and the linking is happening after the temporary directory can not "delete" itself, but as I do not see the full stack trace I can not tell if that is the case.

As I read, hard linking folder on Windows required admin privileges or different policies, so this current folder/directory linking might not be the best solution for this particular feature, but on Linux it works fine.

Should I come up with some other alternative ?

- Changes in test and reason for disabling test on Windows
@maxandersen
Copy link
Collaborator

i'll need to test on my windows setup but I thought we used linking for jdks - but if we dont we might need a "jdklocations.json" instead of 100% file path based solution.
@quintesse wdyt?

@nandorholozsnyak
Copy link
Contributor Author

I was also thinking on a config file to locate JDKs, it should work everywhere. Let me know if I should rework it

@maxandersen
Copy link
Collaborator

I think that could be useful.

- CI config change to check what is the output of the tests, will get reverted
@nandorholozsnyak
Copy link
Contributor Author

BTW, so if you are using a privileged powershell or command line then the whole feature works on Windows too. The symbolic linking on Windows has been already announced as it could only work with admin privileges only. Knowing this as an already announced limitation on Windows, should it be still reworked to the config file based JDK location?

I just changed the CI build config to see the more information in the logs, I will revert it as soon as I just understand the situation within the CI build.

- Retesting old version with custom JBANG_CACHE_IDR
- Retesting old version with custom JBANG_CACHE_IDR
- Retesting old version with custom JBANG_CACHE_IDR
@maxandersen
Copy link
Collaborator

symbolic link was supposed to only affect jbang edit; for jdk location that is used for running i think we need to find a way to support it across platforms.

@nandorholozsnyak
Copy link
Contributor Author

Allright, I was not able to reproduce the error that was happening in multiple builds... I'm going to rework it with a config file.

@maxandersen
Copy link
Collaborator

a reason we didnt do config file before was that the jdk mechanism needs to work in the various bash/powershell/cmd script too - so we need to have that in mind how that will work.

@nandorholozsnyak
Copy link
Contributor Author

@maxandersen
By the way, I just tested the jbang jdk default on Windows, and it is also not working, but it is stated in the documentations as well: https://www.jbang.dev/documentation/guide/latest/javaversions.html#managing-jdks

This config based change could be very "big", I would be totally fine to use admin privileges to set and install JDKs because of that limition, so what is going to be the final decision? Keep it as is, cause it has been stated earlier or should I do the refact on it? I have no Mac to test, but I can imagine it does not have this limitation only Windows.

@maxandersen
Copy link
Collaborator

hmm okey - if windows already has this issue then maybe better we add this now (with its known limitations) and look at config file based approach separately.

@quintesse wdyt ? you know the nastiness around these

- Solution using BaseTest prepared JBANG_CACHE_DIR
- Removing --info from gradle build
@quintesse
Copy link
Contributor

I'll take a look tomorrow. But a quick remark would be that we do indeed use symlinks on all platforms and that has always worked. Using any kind of "index" will make the whole "default" system not work. (You could never use a path like that for PATH or JAVA_HOME, you'd have to change them each time you switch the default JDK).

@maxandersen
Copy link
Collaborator

Yes, but there are two separate but related features here.

Jbang jdk install to point to a specific alternate java vm to use when resolving java version.

Jbang jdk default install location.

We can do the first via config file and second can keep using symlink and be limited

@quintesse
Copy link
Contributor

Sure, we could, but we're already using the linking on Windows and we already accept that in certain cases the linking on Windows isn't ideal. So why introduce a new system just to work around one of those issues and not both? It seems to me that a better investment of time for the future would be to figure out how others are able to reliably make linking work on Windows (eg. Scoop).

@quintesse
Copy link
Contributor

Btw, for me this PR works perfectly on Windows 👍 🙂

throws IOException {
if (force || !JdkManager.isInstalledJdk(version)) {
JdkManager.downloadAndInstallJdk(version);
if (StringUtils.isNotBlank(path)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer the usage of https://github.com/jbangdev/jbang/blob/238ef90b0f65aba896b241b06bc8cde796364382/src/main/java/dev/jbang/util/Util.java#L1339_L1349 over the use of external libraries. (This might change in the future but at least that way it would be easy to find all occurrences)

quintesse
quintesse previously approved these changes Mar 7, 2022
Copy link
Contributor

@quintesse quintesse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The comments I left are mostly clean-up.

@@ -31,10 +31,11 @@
@BeforeEach
void initEnv(@TempDir Path tempPath) throws IOException {
jbangTempDir = Files.createDirectory(tempPath.resolve("jbang"));
jbangTempCacheDir = jbangTempDir.resolve("cache");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file are not necessary (and unwanted really). See my comments on TestJdk.


final File testCache = initJBangCacheDir();
final File jdkPath = new File(testCache, "jdks");
final File jdkPath = new File(jbangTempCacheDir.toFile(), "jdks");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use JdkManager.getJdksPath()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hehe, that is a private method. I can make it public to make it work.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, probably okay to do that 👍

void testJdkInstallWithLinkingToExistingJdkPathWhenJBangManagedVersionDoesNotExist(@TempDir File javaDir)
throws IOException {
initMockJdkDir(javaDir);
final File jdkPath = new File(jbangTempCacheDir.toFile(), "jdks");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, use JdkManager.getJdksPath()

@quintesse
Copy link
Contributor

quintesse commented Mar 7, 2022

Personally I'd merge this PR as-is because it works great, even on Windows as long as the user has made the appropriate policy changes (which were already necessary on Windows). We can then think about how to improve the linking issue later.

@nandorholozsnyak
Copy link
Contributor Author

I will fix the findings after work. Thank you.

- Removing StringUtils usage in Jdk
- Rework test to use JdkManager's method
@nandorholozsnyak
Copy link
Contributor Author

Everything should be fixed an the builds should be working.

@quintesse quintesse merged commit c8bcd40 into jbangdev:main Mar 8, 2022
@quintesse
Copy link
Contributor

Good job, thanks @nandorholozsnyak !

@nandorholozsnyak nandorholozsnyak deleted the feature/1248-install-jdk-with-path branch March 8, 2022 07:24
@maxandersen
Copy link
Collaborator

@all-contributors please add @nandorholozsnyak for code

@allcontributors
Copy link
Contributor

@maxandersen

@nandorholozsnyak already contributed before to code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants