Handle ambigous name/version splits in nix #854
Comments
That's great! How much of it made its way into unstable dump? I'm already seeing some
It turns out that I'm already using it, it's even mentioned in the source comment that references https://github.com/NixOS/nix/blob/master/src/libexpr/names.cc#L19.
Unfortunately problems support is currently quite limited, however this information can be dumped to update log (available from https://repology.org/repositories/updates#nix_unstable). |
|
Update: I've tried to use "version" in the parser There are some inconsistencies:
However much more packages were fixed: -nix_unstable fuse 7z-ng-git-2014-06-08
+nix_unstable fusefs:7z-ng 2014-06-08-nix_unstable lisp-trivial-utf 8-20111001-darcs
+nix_unstable lisp-trivial-utf-8 20111001-darcs-nix_unstable wmii-hg 2012-12-09
+nix_unstable wmii hg-2012-12-09Some packages with version not starting with number are now parsed (like
It doesn't make sense to dump all potentially ambiguous names - these are basically everything with two or more hyphens, and there is too many of them. |
… fixed version parsing (repology/repology-updater#854)
- Switch to PackageMaker context manager - Try to use new "version" field to reliably split package name and version - Improve logging
Well as of now it's all of them, so it doesn't make much sense. It makes sense though to report most suspicious ones. I've found that most of packages I've had to add exceptions for fall under '-[0-9]+[a-z]' regexp. I'm logging these now. I'm deploying update with |
|
Note that pname: when it should be: pname: However, if you follow link to nix package from https://repology.org/project/uhubctl/versions, it has |
Not in json which repology uses. @volth ping? It also haven't updated since Aug 30th. % curl --silent https://nixos.org/nixpkgs/packages-unstable.json.gz | gunzip | jq .packages.uhubctl
{
"name": "uhubctl-unstable-2019-07-31",
"system": "x86_64-linux",
"meta": {
"available": true,
"description": "Utility to control USB power per-port on smart USB hubs",
"homepage": "https://github.com/mvp/uhubctl",
"license": {
"fullName": "GNU General Public License v2.0 only",
"shortName": "gpl2",
"spdxId": "GPL-2.0-only",
"url": "http://spdx.org/licenses/GPL-2.0-only.html"
},
"maintainers": [
{
"email": "pavol@rusnak.io",
"github": "prusnak",
"githubId": 42201,
"keys": [
{
"fingerprint": "86E6 792F C27B FD47 8860 C110 91F3 B339 B9A0 2A3D",
"longkeyid": "rsa4096/0x91F3B339B9A02A3D"
}
],
"name": "Pavol Rusnak"
}
],
"name": "uhubctl-unstable-2019-07-31",
"outputsToInstall": [
"out"
],
"platforms": [
"aarch64-linux",
"armv5tel-linux",
"armv6l-linux",
"armv7l-linux",
"mipsel-linux",
"i686-linux",
"x86_64-linux",
"powerpc64le-linux",
"riscv32-linux",
"riscv64-linux",
"x86_64-darwin",
"i686-darwin",
"aarch64-darwin",
"armv7a-darwin"
],
"position": "pkgs/tools/misc/uhubctl/default.nix:23"
}
} |
|
Perhaps nix can add 2 more fields to json output: |
|
It's mentioned in the discussion above, these fields are (or at least should be) present in the json. For instance, {
"name": "abcl-1.5.0",
"system": "x86_64-linux",
"meta": {
"available": true,
"description": "A JVM-based Common Lisp implementation",
"homepage": "https://common-lisp.net/project/armedbear/",
"license": {
"fullName": "GNU General Public License v3.0 only",
"shortName": "gpl3",
"spdxId": "GPL-3.0-only",
"url": "http://spdx.org/licenses/GPL-3.0-only.html"
},
"maintainers": [
{
"email": "7c6f434c@mail.ru",
"github": "7c6f434c",
"githubId": 1891350,
"name": "Michael Raskin"
}
],
"name": "abcl-1.5.0",
"outputsToInstall": [
"out"
],
"platforms": [
"aarch64-linux",
"armv5tel-linux",
"armv6l-linux",
"armv7l-linux",
"mipsel-linux",
"i686-linux",
"x86_64-linux",
"powerpc64le-linux",
"riscv32-linux",
"riscv64-linux"
],
"position": "pkgs/development/compilers/abcl/default.nix:34",
"version": "1.5.0"
}
}Not sure why's it not present for Also, notably, there are no https://repology.org/log/1418803 |
|
Can you use attribute name as package name, and work out version from it? Note that full json https://nixos.org/nixpkgs/packages-unstable.json.gz has a field package name in its hierarchy, e.g: This way your script can know that actual package name is |
|
You'd still need to extract version part from |
This is great, awaiting it's properly exported via the dump. I wonder, could it be possible that |
We need different names as those are used to distinguish packages by |
|
I believe those predate the introduction of the the I consider our introduction of By the way, in many packages we currently have Maybe we should discuss these issues in Nixpkgs issue tracker instead. |
I am aware that there is no actual |
I belive that since pname/versions were just recently introducted,
If it can't be |
We could make
As I explained above, I am not very fond of this idea since
We could to redefine and the values as follows:
Alternately, we could consider dropping the variant names from package names, as they are primarily used by the slow legacy
I agree. Something like |
For Repology this is the most suitable (as long as |
|
Yeah, we have two parallel ways for selecting packages: You can let Nix traverse the whole nixpkgs and try to find a derivation (think JSON document with the package data) with matching Or you can get the derivation directly if you know the attribute path that leads to it (e.g. Obtaining any attribute from the derivation is easy, but getting the derivation from pname is inefficient. Updaters will want to know the attribute path or canonical derivation itself (Nix is lazily evaluated) to spare themselves of the slowness of the first method. |
|
Actually, what about using the attribute paths (the keys in the JSON file) instead of derivation names for Repology? |
As I've mentioned above, these contain even more garbage than |
|
So, I've somehow resolved this issue - I've refactored the parser a bit, it's now simpler and more straightforward in detecting bad data. The update is no longer blocked on separate While here, I'd like to mention that the new package names (in all their variety) handling mechanism (#931) is now in place, and it could make use of attribute name mentioned by @jtojnar, so I wanted to ask about it's meaning (and difference to pname). In short, Repology now stores a set of names of different purposes for each package:
I wonder if attribute name would be useful as 3 or some of 4. |
Nixpkgs is structured as a nested (in Nix parlance) attribute sets (very similar to Python dictionaries or JSON objects). The attribute name (or rather attribute path) denotes along which attributes (keys) you need to traverse to get to a desired package. Most packages are in the top-level attribute set but some things like Haskell, Python libraries, part of GNOME platform packages, Qt libraries… are nested under a common attribute set in the top-level (e.g. The attribute path is used to unambiguously refer to a package in the package set. There are aliases (e.g. For technical reasons, we sometimes also have multiple variants of a single project built with different configure flags, for example:
Those will all be listed in the generated JSON file and may not necessarily have a different So I agree that attribute name would be potentially useful as 3. But as we do not have a notion of a canonical attribute path, the JSON dump chooses And attribute paths can still change. For example, Nixpkgs did not allow dashes in attribute names in the past, so we used dashes instead. Now that the policy is somewhat relaxed, we are gradually moving packages to paths more in line with uppstream name. But yeah, the attribute path should still be the most stable identifier. Regarding Since the hash of the package expression and the transitive closure of expressions of its dependencies is the primary disambiguator, the Of course the heuristic for extracting version from derivation name (basically a equivalent to regex like For 4, yeah, the attribute patches would fit the generic name, as it is what we use to refer to packages within nixpkgs. The closest thing we have to a source packages is a |
Two items in resholve's mkDerivation are causing trouble for some ecosystem tools: 1. I didn't pass through the original package's meta, which breaks the ability of at least nixos package search and r-ryantm to find the right source file (in the latter case breaking auto updates). 2. I was prepending "resholved-" to the pname, which at least nixos package search picks up as the package's name. Repology also tries to do this, but their current nix updater will prefer to get this data from the name. For now, this means changing to name will not stop repology from picking up the `resholved-<package>` names. Repology's code makes it clear that they *want* to use the pname/ version, so I was inclined to settle with what I've got for now, but thiagokokada clarified that we aren't just waiting for nixpkgs fixes, but because Nix itself isn't exporting the pname/version in its JSON. See also: - repology/repology-updater#854 - repology/repology-updater@9313110121df5 For now, at least, I'll switch to appending "-unresholved" to the inner derivation's pname.
Two items in resholve's mkDerivation are causing trouble for some Nix ecosystem tools: 1. I didn't pass through the original package's meta, which breaks the ability of at least nixos package search and r-ryantm to find the right source file (in the latter case breaking auto updates). 2. I was prepending "resholved-" to the pname, which at least nixos package search picks up as the package's name. Repology also tries to do this, but their current nix updater will prefer to get this data from the name. For now, this means changing to name will not stop repology from picking up the `resholved-<package>` names. Repology's code makes it clear that they *want* to use the pname/ version, so I was inclined to settle with what I've got for now, but thiagokokada clarified that we aren't just waiting for nixpkgs fixes, but because Nix itself isn't exporting the pname/version in its JSON. See also: - repology/repology-updater#854 - repology/repology-updater@9313110121df5 For now, at least, I'll switch to appending "-unresholved" to the inner derivation's pname. Closes #86.
and others
Nix metadata format does not split package names from versions (e.g.
"name": "foo-1.0"). On top of that, both name and version may have hyphens in them. So there's no way to split name and version reliably, and though we use some heuristics (take largest part from the right which starts with the number), there are cases which are processed incorrectly (liblqr-1-VER,python3.6-3to2-VER,polkit-qt-1-qt5-VERetc.). I'm running into a lot of these lately, so this needs to be fixed.@ryantm @volth could nix dump format be extended to provide separate name and/or version field(s)?
The text was updated successfully, but these errors were encountered: