treewide: use new cargo fetcher

[Bot-wxt1221]

Already done. We should run nixpkgs-review to check if some package have different Cargo.lock when building and in nixpkgs, like veloren.

I have writen a script to update it automatically.

Now It can solve:

cargoLock = {
  lockFile = xxx;
  outputhahes = {xxx};
};

cargoLock.lockFile = xxx;
cargoLock.outputHashes = {xxx};

Script: https://github.com/Bot-wxt1221/cargo-rename

Usage:

Compile with gcc. Make sure fetch-cargo.py is in the same path. Exec with a xx/pkgs/by-name/xx/xx/package.nix

cc #327063

#349360

Step to reduce:

1. Generate a file list with cargoLock:

rg "cargoLock" --files-with-matches > filewithcargoLock

cat filewithcargoLock | rev | cut -d / -f 2|rev > packagename

2. run update-all

useFetchCargoVendor

• Squash all commit

• Built on platform(s)

  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin

• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

  • sandbox = relaxed
  • sandbox = true

• Tested, as applicable:

  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages

• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage

• Tested basic functionality of all binary files (usually in ./result/bin/)

• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module

• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[Bot-wxt1221]

@ofborg build libconfig hocon xdg-desktop-portal-cosmic

[JohnRTitor]

See above

[Bot-wxt1221]

@JohnRTitor We really need a quicker fetcher. I've found that it would be OK if we set max-retries time to 10. And allow run 100 threads in one time.

[JohnRTitor]

Unfortunately more concurrent jobs will just seen as a bot spam by GitHub and other servers, so the requests will error out (403).

This was the motivation behind 9fd5a8d as Ofborg wasn't able to fetch the git dependencies properly hitting 403.

[Bot-wxt1221]

@JohnRTitor I don't mean we should allow it run at 100 or similar thread. But I find that if we allow it retry for 10 times, even though I run 100 thread, I doesn't meet error when running script.

[JohnRTitor]

if we set max-retries time to 10

CC @TomaSajt, could this fix the issues encountered in #356385 (comment)?

[TomaSajt]

I wouldn't be against a retry system, but I don't think we should up the thread numbers by much.

Speed was not really a concern for me, since you're only going to build the FOD once or twice in normal circumstances.

[TomaSajt]

Looks like the current script can't handle cases where upstream doesn't have the lockfile. This means that we can't just blindly assume that having outputHashes present means that it can be migrated.

[Bot-wxt1221]

We can run nixpkgs-review to check what package can't be built and remove them.

[TomaSajt]

@Tomahna We can run nixpkgs-review to check what package can't be built and remove them.

Probably pinged the wrong person.

Also, these would probably result in eval errors.
If you look at the nixpkgs-vet CI jobs you can see which packages fail.

[Bot-wxt1221]

@TomaSajt Actually still 50 package are planned to move. So I'll do it after it. And that's the reason why it is draft.

[Bot-wxt1221]

The path should be different now. But not really bad, I'm running nixpkgs-review.

changes addressed

[JohnRTitor]

Could you seperate into 6905f1e another PR?

This way we can just squash merge this PR, without having you squash the commits yourself.

Or you can just squash all commits except that one and force push.

[emilazy]

Mixing substantive changes and formatting changes in the same commits in a treewide PR this huge is not acceptable.

Also, treewide replacements of opaque FOD hashes are among the most dangerous changes that can be made in Nixpkgs, given the existence of FOD injection attacks; just building packages doesn’t avoid that risk. I’d really like to see someone do an automated comparison of the resulting vendor directories, or at least write a script that can locally reproduce the hashes here.

Edit: Okay, it seems like the steps in the PR description may be sufficient to reproduce these commits? In that case I would really like to see someone do a full review of the script’s source code and confirm that they obtained the exact same tree after running it from a pristine state. This is the kind of change that would make it extremely easy to insert backdoors into Nixpkgs.

[JohnRTitor]

Please do not include formatting changes in this PR.

Keep only the new fetcher related changes.

[Bot-wxt1221]

Look above.

[TomaSajt]

To be honest, I'm also more in favor of multiple smaller PRs. We're not rushing anywhere.
It would also be much easier to review.

[Bot-wxt1221]

About nixfmt:

I can split the nixfmt commit into one.

About reduce:

The script was in https://github.com/Bot-wxt1221/cargo-rename. Anyone can reduce it.

About split into smaller PRs:

It won't make it more easier. We still need to recompile every package by hand or by nixpkgs-review in smaller PR.

I will provide any thing to reduce this PR.

@TomaSajt @JohnRTitor @emilazy

[Mic92]

@Bot-wxt1221 Maybe you can do a pull request chain? I.e just branch off every after X commits and make a new pull request depending on the previous one? That hopefully is an acceptable management overhead for both sides. Maybe 20 packages per pull request could work, which would be 10 pull requests.

[Bot-wxt1221]

Will split into multi PR. I will open an issue to track it.

[Bot-wxt1221]

#357257