Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mirrors don't provide sha256 #521

Closed
mite-user opened this issue May 9, 2022 · 18 comments
Closed

Mirrors don't provide sha256 #521

mite-user opened this issue May 9, 2022 · 18 comments
Labels
enhancement New feature or request

Comments

@mite-user
Copy link
Contributor

aqtinstall requires .sha256 hashes to verify Updates.xml and 7z archives.

Only https://download.qt.io/ provides .sha256, .sha1, .md5 and .meta4 files for every visible file.
Mirrors provide only visible files, i.e. .sha1 for 7z archives which aren't *meta.7z.

If https://download.qt.io/ is not available, it's impossible to download any archive with aqtinstall.

There should be an option to use .sha1 and skip verification of Updates.xml.

@ddalcino
Copy link
Contributor

ddalcino commented May 9, 2022

Thanks for letting us know about the lack of sha256 hashes on the mirrors.

IMHO, we should be very careful about allowing aqt to use sha1 or md5 hashes in any way. It’s not hard for an attacker to create a .7z file with a hash collision with a legitimate sha1 or md5 hash; this is not true of sha256. If an attacker has control over any of our mirror sites, that attacker can convince aqt that their malicious copy of the .7z file is the real thing, and suddenly the attacker can get their code to run in your Qt program. This is a very big deal.

If you want to add the ability to use sha1, you will need to be able to prove without any doubt that there is no code path that uses sha1 by mistake. This feature will add attack surface to aqt that will need lots of developer attention.

@ddalcino
Copy link
Contributor

There are other potential solutions to this problem, other than using insecure checksums. We could host our own copies of the expected sha256 checksums here on Github, and use our copies as a fallback when download.qt.io is unavailable.

I have a project at https://github.com/ddalcino/qt-repo-cache that could be modified to do this job; right now, it hosts copies of the Updates.xml files at download.qt.io, stored as json. This project runs a workflow once a day that scans 8 HTML files at download.qt.io: 3 for the Windows targets, 2 for the Linux targets, and 3 for the Mac targets. It reads the 'Last Modified' column for each folder to decide whether or not anything has been updated. If a folder has been updated since the last time the workflow has run, it fetches the Updates.xml file and caches it; otherwise, it skips the folder. This approach will only make 8 HTTP GET requests per day if nothing has changed, and if something has changed, it makes the minimum number of requests required to fetch the updated files.

We could do something similar: Check the 'Last Modified' column for each folder, and if anything has changed recently, go into that folder and fetch all the sha256 files for modules that have been updated.

I must note that this approach also increases attack surface. I have the workflow at https://github.com/ddalcino/qt-repo-cache set up to commit changes automatically, rather than by pull request. The repo changes so often that I would not be able to keep up with it if I had to approve every change manually. If an attacker could somehow convince the workflow to download a bad checksum (via MITM, etc), there would be no way to prevent the workflow from committing those values, and no way to know what had happened afterward.

I don't know if this is a good solution, but IMHO it's better than using sha1. If download.qt.io is unavailable a lot of the time, maybe we should consider something like this.

@mite-user
Copy link
Contributor Author

As a temporary workaround, I wrote this https://github.com/mite-user/mite-qtinstall
The idea is to prefetch SHA256 hashes as @ddalcino suggested.
Hope this helps.

@miurahr miurahr added the enhancement New feature or request label Sep 15, 2022
@ddalcino
Copy link
Contributor

ddalcino commented Sep 15, 2022

Last night I tried to install qt 6.4.0 in CI, but the job failed because there’s no sha256 hashes for any of the files in that directory, on the download.qt.io server. Hopefully this is just a temporary situation, and they will upload the hashes later. However, there’s a very real possibility that the qt repo will stop uploading sha256 hashes for new versions. We don’t have any guarantees here.

In this case, caching sha256 hashes downloaded from download.qt.io will not work.

We may have no choice but to add an option to use one of the insecure hashes like md5/sha1 that appear to exist on both download.qt.io and the mirrors. I propose that we call it something like --insecure to discourage people from using it unless absolutely necessary, and to warn them about what they’re doing.

@ddalcino
Copy link
Contributor

ddalcino commented Sep 17, 2022

Quick update on the sha256 situation: There are now sha256 hashes available for Qt 6.4.0. The CI job that I observed as failing (due to missing hash) now passes: https://dev.azure.com/miurahr/github/_build/results?buildId=5742&view=logs&j=b86c6ba8-325a-58b4-9c91-e9a8f44b4793

@bam80
Copy link

bam80 commented Sep 24, 2022

It’s not hard for an attacker to create a .7z file with a hash collision with a legitimate sha1 or md5 hash; this is not true of sha256. If an attacker has control over any of our mirror sites, that attacker can convince aqt that their malicious copy of the .7z file is the real thing, and suddenly the attacker can get their code to run in your Qt program.

I'm not security expert but I don't see a problem here.
Even if attacker creates a hash collision, that "malicious" copy will contain a garbage, so no code can be injected that way?
Just my 2 cents, sorry if I'm wrong.

@ddalcino
Copy link
Contributor

I'm not security expert but I don't see a problem here. Even if attacker creates a hash collision, that "malicious" copy will contain a garbage, so no code can be injected that way? Just my 2 cents, sorry if I'm wrong.

No, the malicious copy may contain whatever code the attacker wants. It could be padded with garbage to make the hash come out right, but the padding would likely be hidden in a way that you would not see it.

You can search for hash collision generators on any search engine to find tools that allow you to do this; several are hosted on GitHub.

@bam80
Copy link

bam80 commented Sep 24, 2022

Thanks, didn't know that.
Does it mean all the mirrors can be considered insecure right now?

@ddalcino
Copy link
Contributor

No, the mirrors are fine, as long as the binaries they host match the sha256 hashes hosted on download.qt.io. It’s relatively easy to fake a sha1 hash, and extremely difficult to fake a sha256 hash.

@miurahr
Copy link
Owner

miurahr commented Sep 25, 2022

FYI: qt.download.io uses MirrorBrain for its distribution server.
MirrorBrain is a download redirector and generates cryptohashes, and metalinks.

Aqtinstall depends on metalinks which MirrorBrain generated, and detect blacklisted mirror site, not just redirect by 304 response.

MirrorBrain automatically generates Updates.xml.meta4 .sha256 .sha1 .md5 and even .torrent

@mardy
Copy link
Contributor

mardy commented Nov 13, 2022

Hi! Is it OK then if I propose a PR that adds one --insecure option which disables checksum verification?

@regs01
Copy link

regs01 commented Jul 19, 2023

Reading all issue tickets related to the problem. But as i get it, there is no solution? It's not possible to use aqtinstall, if https://download.qt.io/ is not available?

@miurahr
Copy link
Owner

miurahr commented Jul 20, 2023

Reading all issue tickets related to the problem. But as i get it, there is no solution? It's not possible to use aqtinstall, if https://download.qt.io/ is not available?

You can help #684 to be a state of quality that pass all the CI checks.

@regs01
Copy link

regs01 commented Jul 20, 2023

Reading all issue tickets related to the problem. But as i get it, there is no solution? It's not possible to use aqtinstall, if https://download.qt.io/ is not available?

You can help #684 to be a state of quality that pass all the CI checks.

That's what i did. I took head from here, added edits from that pull request, added lines to settings.ini and it worked.

[requests]
hash_algorithm: sha1
INSECURE_NOT_FOR_PRODUCTION_ignore_hash: False

@regs01
Copy link

regs01 commented Oct 22, 2023

Lost that executable. Now trying to compile a new one.
python build_standalone.py ends with no errors
But compiled exe showing error

./aqt
Traceback (most recent call last):
  File "launch_aqt.py", line 2, in <module>
  File "PyInstaller\loader\pyimod02_importers.py", line 385, in exec_module
  File "aqt\__init__.py", line 31, in <module>
ModuleNotFoundError: No module named 'aqt.version'
[31404] Failed to execute script 'launch_aqt' due to unhandled exception!

Pure master branch shows same

@miurahr
Copy link
Owner

miurahr commented Oct 22, 2023

aqt/version.py is generated when python -m pip install . by setuptools_scm.

you can see CI/CD script.
https://github.com/miurahr/aqtinstall/blob/master/.github/workflows/upload-release-artifacts.yml#L36C11-L36C34

setuptools_scm configuration is here.
https://github.com/miurahr/aqtinstall/blob/master/pyproject.toml#L90-L96

@regs01
Copy link

regs01 commented Oct 23, 2023

It turned it wanted .git folder. I was just downloading in zip archive.

@miurahr
Copy link
Owner

miurahr commented Nov 28, 2023

#684 merged.
#730 allow build from git export, github's zip download.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

6 participants