Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gradle download broken #1218

Closed
hohwille opened this issue Jun 27, 2023 · 9 comments · Fixed by #1222
Closed

gradle download broken #1218

hohwille opened this issue Jun 27, 2023 · 9 comments · Fixed by #1222
Assignees
@jan-vcapgemini @alfeilex
Labels
bug Something isn't working urls ide-urls repo and related processes and features
Milestone
release:2023.07.001

Comments

@hohwille
Copy link
Member

It seems that gradle downloads got broken - maybe the URLs have been changed by the tool vendor.
Our github action already noted that the downloads broken immediately after the metadata was created as you can see in the status.json files.
It seems we have to fix the GradleUrlUpdater and also delete the entire gradle folder from ide-urls so it will be recreated in the night.

p.s.: I hope such effects will not recur often as this sucks and causes us maintenance effort that we actually do not want to have. The idea of #941 was actually to reduce the maintenance effort as much as possible.
@hohwille hohwille added bug Something isn't working urls ide-urls repo and related processes and features labels Jun 27, 2023
@hohwille hohwille added this to the release:2023.06.001 milestone Jun 27, 2023
@hohwille
Copy link
Member Author

Example for current download link leading to error:
https://services.gradle.org/distributions/gradle-9.95-bin.zip

@hohwille hohwille mentioned this issue Jun 27, 2023
Closed
@hohwille
Copy link
Member Author

It seems to be different. The download link structure is OK.
https://github.com/devonfw/ide-urls/blob/master/gradle/gradle/8.1.1/urls
https://services.gradle.org/distributions/gradle-8.1.1-bin.zip works

As you can see on the download website of gradle there are no 9.x release versions listed. So just tracking git tags does not seem to be the proper approach here. Still I guess that deleting the gradle folder and rebuilding it over night will not rebuild the broken releases.
My assumption is that the broken versions have been created from the very old code of url-updater that did not follow redirects. So as a first step, I will simply delete gradle folder.

hohwille added a commit to devonfw/ide-urls that referenced this issue Jun 27, 2023
@hohwille
devonfw/ide#1218: remove all but latest gradle metadata to rebuild ni…
999c93f 
…ghtly
@jan-vcapgemini jan-vcapgemini self-assigned this Jun 27, 2023
@hohwille
Copy link
Member Author

Even the latest official version failed:

$ devon gradle setup
Updating gradle from version 8.0.2 to version 8.1.1...
Success: run command mkdir
Starting installation of gradle in version 8.1.1 to /d/projects/test/software/gradle
Deleting old installation as it is a symlink: /d/projects/test/software/gradle
Trying to download gradle-8.1.1.zip from https://services.gradle.org/distributions/gradle-8.1.1-bin.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 18  118M   18 22.3M    0     0   188k      0  0:10:44  0:02:01  0:08:43     0
curl: (56) Recv failure: Connection was reset
Failed to download https://services.gradle.org/distributions/gradle-8.1.1-bin.zip with exit code 56

@hohwille
Copy link
Member Author

This is nothing we can change. The vendor of gradle has to provide a reliable service for downloading its tool.

@jan-vcapgemini
Copy link
Contributor

jan-vcapgemini commented Jun 27, 2023 •
edited
Loading

When I try to generate the checksums locally, I get following exception:

14:16:34.915 [main] [C:] - ERROR - c.d.t.i.u.updater.AbstractUrlUpdater - For tool gradle we failed to add version 6.9.1.
java.lang.IllegalStateException: Failed to read body of download https://services.gradle.org/distributions/gradle-6.9.1-bin.zip
at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doGenerateChecksum(AbstractUrlUpdater.java:324)
at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.checkDownloadUrl(AbstractUrlUpdater.java:280)
at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:190)
at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:164)
at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:149)
at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:136)
at com.devonfw.tools.ide.url.updater.gradle.GradleUrlUpdater.addVersion(GradleUrlUpdater.java:33)
at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.update(AbstractUrlUpdater.java:452)
at com.devonfw.tools.ide.integrationtests.JsonUrlUpdaterIT.testGradle(JsonUrlUpdaterIT.java:78)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.io.IOException: closed
at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.current(ResponseSubscribers.java:368)
at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.read(ResponseSubscribers.java:403)
at java.base/java.io.InputStream.read(InputStream.java:205)
at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doGenerateChecksum(AbstractUrlUpdater.java:310)
... 78 common frames omitted
Caused by: java.io.IOException: fixed content-length: 107669361, bytes received: 26966177
at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Utils.java:330)
at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.onReadError(Http1Response.java:758)
at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Http1AsyncReceiver.java:297)
at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:263)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: connection closed locally
at java.net.http/jdk.internal.net.http.SocketTube.signalClosed(SocketTube.java:157)
at java.net.http/jdk.internal.net.http.PlainHttpConnection.close(PlainHttpConnection.java:277)
at java.net.http/jdk.internal.net.http.AsyncSSLConnection.close(AsyncSSLConnection.java:111)
at java.net.http/jdk.internal.net.http.Http1Exchange.cancelImpl(Http1Exchange.java:490)
at java.net.http/jdk.internal.net.http.Http1Exchange.cancel(Http1Exchange.java:426)
at java.net.http/jdk.internal.net.http.Exchange.cancel(Exchange.java:228)
at java.net.http/jdk.internal.net.http.MultiExchange.cancel(MultiExchange.java:240)
at java.net.http/jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:71)
at java.net.http/jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1248)
at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:877)

@jan-vcapgemini
Copy link
Contributor

When I manually download the zip files with my browser, every download succeeds.

jan-vcapgemini added a commit to jan-vcapgemini/ide that referenced this issue Jun 29, 2023
@jan-vcapgemini
devonfw#1218: optimized gradle url-updater
a19bfd9 
made version regex pattern more distinct
added hashsum param to doAddVersion
added doGetHashSumForVersion method (can extract a valid hashsum for a given version)
moved regex patterns and URLs into constants
@jan-vcapgemini jan-vcapgemini mentioned this issue Jun 29, 2023
Merged
@hohwille
Copy link
Member Author

hohwille commented Jun 30, 2023 •
edited
Loading

Interesting: It seems to be unstable, but not entirely broken:

$ devon gradle setup
Success: run command mkdir
Starting installation of gradle in version 8.1.1 to /d/projects/devonfw/software/gradle
Trying to download gradle-8.1.1-gradle.zip from https://services.gradle.org/distributions/gradle-8.1.1-bin.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 41  118M   41 49.7M    0     0   411k      0  0:04:56  0:02:03  0:02:53     0
curl: (56) Recv failure: Connection was reset
Failed to download https://services.gradle.org/distributions/gradle-8.1.1-bin.zip with exit code 56

******** ATTENTION ********
Download of gradle in version 8.1.1 failed after trying 1 url(s).
We are sorry for the inconvenience. Please check the above errors, resolve them and try again.

hohwille@CE49454 MSYS /d/projects/devonfw
$ devon gradle setup
Success: run command mkdir
Starting installation of gradle in version 8.1.1 to /d/projects/devonfw/software/gradle
Trying to download gradle-8.1.1-gradle.zip from https://services.gradle.org/distributions/gradle-8.1.1-bin.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  118M  100  118M    0     0  10.5M      0  0:00:11  0:00:11 --:--:-- 10.6M
Download of gradle-8.1.1-gradle.zip from https://services.gradle.org/distributions/gradle-8.1.1-bin.zip succeeded.
No checksum found at /d/projects/devonfw/urls/gradle/gradle/8.1.1/urls.sha256
Success: run command unzip
Successfully extracted archive gradle-8.1.1-gradle.zip to /d/projects/devonfw/updates/extracted/gradle
Success: run command mv
Successfully installed gradle
The software gradle has been added. You need to rerun 'devon' command without arguments or restart your terminal to update your PATH so the newly installed software will be found.

Welcome to Gradle 8.1.1!

Here are the highlights of this release:
 - Stable configuration cache
 - Experimental Kotlin DSL assignment syntax
 - Building with Java 20

For more details see https://docs.gradle.org/8.1.1/release-notes.html


------------------------------------------------------------
Gradle 8.1.1
------------------------------------------------------------

Build time:   2023-04-21 12:31:26 UTC
Revision:     1cf537a851c635c364a4214885f8b9798051175b

Kotlin:       1.8.10
Groovy:       3.0.15
Ant:          Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM:          20 (Eclipse Adoptium 20+36)
OS:           Windows 10 10.0 amd64

The good thing is that devonfw-ide does download caching so once the download succeeded further requests to the same tool and version will reuse the already downloaded file.
This can also be used as a workaround if CURL cannot download. You can download with your browser and manually put the file under the right name in the download cache (~/Downloads/devonfw-ide/«filename»). You can get the proper «filename» from the log-message before the download starts ("Trying to download «filename» from «URL»").

jan-vcapgemini added a commit to jan-vcapgemini/ide that referenced this issue Jun 30, 2023
@jan-vcapgemini
devonfw#1218: implemented requested changes
04a2996 
added lazy loading of responseBody
added extra check for empty checksum
moved download Url into constant
hohwille pushed a commit that referenced this issue Jul 3, 2023
@jan-vcapgemini
#1218: optimized gradle url-updater (#1222)
54bf838
hohwille added a commit to devonfw/ide-urls that referenced this issue Jul 3, 2023
@hohwille
devonfw/ide#1218: ensure clean rebuild of gradle metadata
d0fb7c4
@hohwille
Copy link
Member Author

hohwille commented Jul 3, 2023

Was auto-closed from PR but not entirely solved.

@hohwille hohwille reopened this Jul 3, 2023
@hohwille
Copy link
Member Author

hohwille commented Jul 6, 2023

So there were two issues:

  • ide-urls metadata for gradle we incorrect - fixed with PR #1218: optimized gradle url-updater #1222
  • curl was (sometimes) unable to download gradle even though the URL was correct - retested with and without VPN and works fine now so it seems that was only a temporary problem on the file servers providing gradle releases.
    Therefore all is resolved and done.

@hohwille hohwille closed this as completed Jul 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jan-vcapgemini jan-vcapgemini

@alfeilex alfeilex

Labels
bug Something isn't working urls ide-urls repo and related processes and features
Projects
None yet
Milestone
release:2023.07.001
Development

Successfully merging a pull request may close this issue.

#1218: optimized gradle url-updater jan-vcapgemini/ide
3 participants
@hohwille @jan-vcapgemini @alfeilex