Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wrong checksum in ide-urls #1142

Closed
hohwille opened this issue May 5, 2023 · 9 comments
Closed

wrong checksum in ide-urls #1142

hohwille opened this issue May 5, 2023 · 9 comments
Assignees
@moritzLanger @alfeilex
Labels
bug Something isn't working SCM software-configuration-management (github actions, internal processes, git or github utilization) urls ide-urls repo and related processes and features
Milestone
release:2023.07.001

Comments

@hohwille
Copy link
Member

hohwille commented May 5, 2023

Expected behavior

Our update process populating ide-urls shall compute correct sha256 checksums for each download file so that devonfw-ide will be able to verify that checksum. Therefore when I download the according file and compute the checksum with sha256sum, I should get the exact same hash.

Actual behavior

For some tools the checksums are computed wrong. According to my analysis some download URLs lead to HTML websites that themselves contain some JavaScript redirecting to the actual download on a mirror. This especially applies to sourceforge.net.
Tools like curl can magically handle this and do the actual download. However, our according UrlUpdaters cannot handle this situation. As a result they consider the HTML content as the actual download file. Therefore they compute the SHA256 checksum of the HTML content instead of the binary download file.
I have already changed the code such that the SHA256 checksum computation will fail if the content type header is present and its value starts with text (such as text/html).

Steps to reproduce (bug) / Use Case of feature request (enhancement)

$ sha256sum ~/Downloads/devonfw-ide/npm-9.6.0-windows.tgz
628f7650e50a4081746a681c74aeca48f9d1cebb333d4d69fdffaa1a487c3ada *...

$ cat urls/npm/npm/9.6.0/windows_x64.urls.sha256
5f1da1ef6ba776449900c931e7acc3b60e47f3080425bf0e6a1c0c5a07dbd6fd

https://github.com/devonfw/ide-urls/blob/master/npm/npm/9.6.0/windows_x64.urls.sha256

Related/Dependent Issues

#1009

Comments/Hints:

Affected version:

2023.04.001
@hohwille hohwille added the bug Something isn't working label May 5, 2023
@hohwille hohwille added this to the release:2023.05.001 milestone May 5, 2023
@hohwille hohwille added SCM software-configuration-management (github actions, internal processes, git or github utilization) urls ide-urls repo and related processes and features labels May 5, 2023
@hohwille
Copy link
Member Author

For npm this is just a simple and stupid bug here:

ide/url-updater/src/main/java/com/devonfw/tools/ide/url/updater/npm/NpmUrlUpdater.java

Line 21 in eed1f40

String baseUrl = "https://nodejs.org/dist/v${version}/node-v${version}";

@moritzLanger moritzLanger mentioned this issue May 22, 2023
Closed
@moritzLanger moritzLanger mentioned this issue May 30, 2023
Merged
@hohwille hohwille mentioned this issue May 16, 2023
Closed
@hohwille
Copy link
Member Author

hohwille commented Jun 6, 2023

If I am not mistaken then only problem left that I could find is gcviewer. Example URL: https://sourceforge.net/projects/gcviewer/files/gcviewer-1.36.jar

@hohwille
Copy link
Member Author

hohwille commented Jun 6, 2023 

$ curl https://sourceforge.net/projects/gcviewer/files/gcviewer-1.36.jar
<html>
<head>
<title>301 Moved Permanently</title>
</head>
<body>
<h1>301 Moved Permanently</h1>
The resource has been moved to <a href="https://sourceforge.net/projects/gcviewer/files/gcviewer-1.36.jar/">https://sourceforge.net/projects/gcviewer/files/gcviewer-1.36.jar/</a>;
you should be redirected automatically.
<script defer src="https://static.cloudflareinsights.com/beacon.min.js/v52afc6f149f6479b8c77fa569edb01181681764108816" integrity="sha512-jGCTpDpBAYDGNYR5ztKt4BQPGef1P0giN6ZGVUi835kFF88FOmmn8jBQWNgrNd8g/Yu421NdgWhwQoaOPFflDw==" data-cf-beacon='{"rayId":"7d30eb96de622c21","version":"2023.4.0","b":1,"token":"05ab2f27910c4db284f4fcdcd6948338","si":100}' crossorigin="anonymous"></script>
</body>
</html>

@hohwille
Copy link
Member Author

hohwille commented Jun 6, 2023

more redirects:

$ curl https://sourceforge.net/projects/gcviewer/files/gcviewer-1.36.jar/download
<html>
<head>
<title>302 Found</title>
</head>
<body>
<h1>302 Found</h1>
The resource was found at <a href="https://downloads.sourceforge.net/project/gcviewer/gcviewer-1.36.jar?ts=gAAAAABkfzGNjOgXzn2nquU6JxLNKs9E7Lr6WUhZ3OzLwV2QbviZiYE1AISsPAunMVuF-FHeteuy0S1JR3bpH8TQogfMWx7Bxw%3D%3D&amp;use_mirror=nav&amp;r=">https://downloads.sourceforge.net/project/gcviewer/gcviewer-1.36.jar?ts=gAAAAABkfzGNjOgXzn2nquU6JxLNKs9E7Lr6WUhZ3OzLwV2QbviZiYE1AISsPAunMVuF-FHeteuy0S1JR3bpH8TQogfMWx7Bxw%3D%3D&amp;use_mirror=nav&amp;r=</a>;
you should be redirected automatically.
<script defer src="https://static.cloudflareinsights.com/beacon.min.js/v52afc6f149f6479b8c77fa569edb01181681764108816" integrity="sha512-jGCTpDpBAYDGNYR5ztKt4BQPGef1P0giN6ZGVUi835kFF88FOmmn8jBQWNgrNd8g/Yu421NdgWhwQoaOPFflDw==" data-cf-beacon='{"rayId":"7d30ed50df4bbb3e","version":"2023.4.0","b":1,"token":"05ab2f27910c4db284f4fcdcd6948338","si":100}' crossorigin="anonymous"></script>
</body>
</html>

@hohwille
Copy link
Member Author

hohwille commented Jun 6, 2023 •
edited
Loading

With the last URL finally curl fails but if I open in the browser, I still get sourceforge HTML but can download.
The final download came from
https://nav.dl.sourceforge.net/project/gcviewer/gcviewer-1.36.jar
However, if you open this link, you again get HTML from sourceforge in the browser.
They invested a hell of hacks to make downloads hard.

@hohwille
Copy link
Member Author

hohwille commented Jun 6, 2023

https://sourceforge.net/p/forge/documentation/Downloading%20files%20via%20the%20command%20line/

@alfeilex
Copy link
Member

When testing gcviewer, the checksums created were correct.

I just got an error in version 1.34.1

java.lang.IllegalStateException: Failed to read body of download https://sourceforge.net/projects/gcviewer/files/gcviewer-1.34.1.jar/download
	at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doGenerateChecksum(AbstractUrlUpdater.java:324)
	at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.checkDownloadUrl(AbstractUrlUpdater.java:280)
	at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:190)
	at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:164)
	at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:149)
	at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:136)
	at com.devonfw.tools.ide.url.updater.gcviewer.GcViewerUrlUpdater.addVersion(GcViewerUrlUpdater.java:19)
	at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.update(AbstractUrlUpdater.java:452)
	at com.devonfw.tools.ide.url.updater.UpdateManager.updateAll(UpdateManager.java:82)
	at com.devonfw.tools.ide.url.UpdateInitiator.main(UpdateInitiator.java:34)
Caused by: java.io.IOException: closed
	at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.current(ResponseSubscribers.java:460)
	at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.read(ResponseSubscribers.java:499)
	at java.base/java.io.InputStream.read(InputStream.java:218)
	at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doGenerateChecksum(AbstractUrlUpdater.java:310)
	... 9 common frames omitted
Caused by: java.io.IOException: fixed content-length: 466036, bytes received: 409354
	at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Utils.java:351)
	at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.onReadError(Http1Response.java:760)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Http1AsyncReceiver.java:302)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:268)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:205)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.IOException: connection closed locally
	at java.net.http/jdk.internal.net.http.SocketTube.signalClosed(SocketTube.java:159)
	at java.net.http/jdk.internal.net.http.PlainHttpConnection.close(PlainHttpConnection.java:372)
	at java.net.http/jdk.internal.net.http.AsyncSSLConnection.close(AsyncSSLConnection.java:111)
	at java.net.http/jdk.internal.net.http.Http1Exchange.cancelImpl(Http1Exchange.java:492)
	at java.net.http/jdk.internal.net.http.Http1Exchange.cancel(Http1Exchange.java:427)
	at java.net.http/jdk.internal.net.http.Exchange.cancel(Exchange.java:238)
	at java.net.http/jdk.internal.net.http.MultiExchange.cancel(MultiExchange.java:260)
	at java.net.http/jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:71)
	at java.net.http/jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1270)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:899)```

but the error does not occur in every run. Sometimes it works. Maybe sourceforge has some server problems.

Version 1.31 does not work either, because its not a `jar` but a `zip` file and the url does not work. But this version is very old and can be ignored in my opinion,

@hohwille
Copy link
Member Author

So the final summary is that we can now close this issue.
Excellent!

@hohwille
Copy link
Member Author

For the record also for Docker-Desktop and other "latest" download urls this bug was present.
The fixed have not always been connected with this issues so you cannot find all commits directly from here.

hohwille added a commit to devonfw/ide-urls that referenced this issue Aug 11, 2023
@hohwille
devonfw/ide#1300: devonfw/ide#1142: fixed gcviewer urls (#10)
d7009a8
hohwille added a commit to devonfw/ide-urls that referenced this issue Aug 31, 2023
@hohwille
devonfw/ide#1300: devonfw/ide#1142: Update GCViewer urls with checksu…
4d9a5df 
…ms (#9)

Update GCViewer urls with checksums
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@moritzLanger moritzLanger

@alfeilex alfeilex

Labels
bug Something isn't working SCM software-configuration-management (github actions, internal processes, git or github utilization) urls ide-urls repo and related processes and features
Projects
None yet
Milestone
release:2023.07.001
Development

No branches or pull requests
3 participants
@hohwille @moritzLanger @alfeilex