Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: variables to check for OS, distribution, etc. #3058

Closed
AltGr opened this issue Sep 14, 2017 · 19 comments
Closed

RFC: variables to check for OS, distribution, etc. #3058

AltGr opened this issue Sep 14, 2017 · 19 comments

Comments

@AltGr
Copy link
Member

AltGr commented Sep 14, 2017

Here is a proposal for new pre-defined variables, with examples. It's mostly inspired from the current state of opam-depext.

First some examples:

host\variables arch os distribution dist-version distribution-family
Debian stretch x86_64 linux debian 9 debian
OSX Yosemite x86_64 darwin homebrew 10.10.5 homebrew
OpenBSD 6.1 x86_64 bsd openbsd 6.1 openbsd
multiarch/debian-debootstrap:armhf-jessie container armv7l linux debian 8 debian
32bit/ubuntu:16.04 container x86_64 (known docker issue) linux ubuntu 16.04 debian
centos:7 container x86_64 linux centos 7 rhel
Android 5.1.1 armv7l linux android 5.1.1 android
resin/rpi-raspbian container armv6l linux raspbian 8 debian
  • arch is the output of uname -m, with i?86 rewritten to x86, amd64 rewritten to x86_64. We might want some normalisation on arm too ? Note: any solution for Windows ? Assume x86_64 ?
  • os is either the output of Sys.os_type, or the lowercased output of uname -s when the former is Unix, except for bsd in the case of *BSD and DragonFly (this differs from depext). Note: should the win32/cygwin distinction rather be made at the distribution level ? Do we want to translate darwin to osx ?
  • distribution is (originally following code from opam-depext):
    • on OSX, the first of homebrew or macport that has a binary installed (resp. brew and port) , or undefined otherwise
    • on Linux, the field ID, from /etc/os-release, or /usr/lib/os-release, or the output of lsb_release -i -s if that exists, or the first word of any of /etc/{redhat,centos,gentoo}-release or /etc/issue. The result is always lowercased. We could also define distribution-family using the (first word of the) field ID_LIKE from os-release, falling back to distribution if that is not defined (useful to get the package manager, for example).
    • lowercase uname -s when os is bsd, yielding freebsd, openbsd, dragonfly...
    • maybe win32/cygwin on Windows ?
    • assumed to be android if getprop ro.build.version.release exists and returns 0
    • undefined otherwise
  • dist-version is
    • On Linux, defined from os-release field VERSION_ID (see above), or some custom parsing of the different other cases that needs to be investigated
    • On OSX, the output of sw_vers -productVersion (note: this is the OS version and not the distribution version — versions of macports/homebrew don't really make sense here. Maybe find a better name for this variable ?)
    • On Windows, something like the output of ver ? There are solution using greps on the output of systeminfo, but that's extremely slow.
    • On Android, the output of getprop ro.build.version.release
    • On other systems, defined from uname -r (this is in general the kernel version, but it seems to match OS releases on *BSD ? Is it correct ?)

The fact that it is a big zoo was my main reason to do this as a plugin first, so that it could evolve and be fixed more rapidly than opam :)

Note: I am not sure distribution is the best term in general, as that seems Linux-specific; but had no idea for a better name. dist-version isn't a perfectly accurate name, either. Also, a OSX user may have both homebrew and macports, so in that case we assume they want to use homebrew...

Should we add os-version, which would make more sense on non-Linux OSes, and point to the kernel version on Linux ? Or just rename dist-version to os-version, since we don't care about the kernel version ?

Please if you have access to / need support for any specific system, check the above, extend, propose, and give feedback! Thanks.

@AltGr
Copy link
Member Author

AltGr commented Sep 14, 2017

Some detail on the usage: these variables could be built-in, and resolved lazily by opam, which means they would be accessible anywhere in opam files. With the proposed extension of depexts (still being designed), standard filters would be used rather than the current sets of tags, and these variables would be accessible from there, e.g.:

depexts: ["libfoo-dev"] { distribution = "debian" & dist-release >= "7" | distribution = "ubuntu" }`

@hannesm
Copy link
Member

hannesm commented Sep 14, 2017

to extend your above table, let me show you the output on some FreeBSD systems:

host\variables uname -m uname -s uname -r uname -K uname -U
FreeBSD-CURRENT (from September) amd64 FreeBSD 12.0-CURRENT 1200040 1200040
FreeBSD-CURRENT (from April) amd64 FreeBSD 12.0-CURRENT 1200028 1200028
FreeBSD-11.0 amd64 FreeBSD 11.0-RELEASE-p8 1100122 1100122
FreeBSD-11.1 amd64 FreeBSD 11.1-RELEASE 1101001 1101001
FreeBSD-10.3 amd64 FreeBSD 10.3-RELEASE-p19 1003000 1003000
FreeBSD-10.3 amd64 FreeBSD 10.3-RELEASE-p11 1003000 1003000

I'd propose to use uname -s as os (where Sys.os_type = Unix, and do not try to unify the BSDs here -- after all, isn't MacOSX as well a BSD?).
The uname -r output on FreeBSD includes both the release number and also the patchlevel. If you want to have release numbers, uname -K reports the kernel version (and -U the user environment version, which should usually be in sync) - this is most useful as dist-version AFAICT!

@AltGr
Copy link
Member Author

AltGr commented Sep 15, 2017

@samoht @avsm you probably want to get involved in this design process :)

@AltGr
Copy link
Member Author

AltGr commented Sep 15, 2017

@hannesm ok, so that would mean the following changes:

  • For all *BSD, os = distribution
  • for FreeBSD, dist-version is either
    • the output of uname -r, truncated before the first non [0-9.] character, e.g. 10.3, which looks nicer (although this will be compared using our version comparison function, so the full one may be OK). Note that doing the truncation for uname -r in all cases wouldn't hurt either, rather than special-casing FreeBSD.
    • the output of uname -K: that looks weird to me, and the flag is non-standard, but whatever is easier to figure out by FreeBSD users!

EDIT: and thanks a lot for the input :)

@hannesm
Copy link
Member

hannesm commented Sep 15, 2017

@AltGr I'd suggest to use uname -U for dist-version, as I outlined above, the output of uname -r for the trunk of FreeBSD isn't very precise. With uname -U (/-K) we have the ability to specify precisely "FreeBSD after importing clang-5.0.0" or "FreeBSD with binutils-2.13".
Certainly "10.3" is easier to read than "1003000", but it is less exact.

@samoht
Copy link
Member

samoht commented Oct 2, 2017

That looks pretty good to me! I am not an expert in the wild jungle of distribution, but that proposal seems to improve the current state of opam-depext :-)

@AltGr
Copy link
Member Author

AltGr commented Oct 5, 2017

So, I have a prototype implementation for this in this gist. Try it out with the following command and report results !

curl -sL https://gist.githubusercontent.com/AltGr/5bfc8cea6f01e74b95de79ceaba39369/raw/opam_analyse.ml | ocaml unix.cma -stdin

Please test on any exotic systems you might want to check depexts on, and tell wether the results would be accurate/convenient enough for you.
Thanks!


Some details

The prototype has some changes compared to the above:

  • os is normalised to "osx" on macs (instead of "darwin"). Not normalised on BSDs
  • os-distribution replaces distribution, and is equal to os on non linux or osx
  • os-release replaces dist-release, and is defined as before, taking into account the comments above for FreeBSD
  • os-family replaces distribution-family, and is bsd on *bsd or dragonfly, ID_LIKE on linuxes when that is defined, windows for win32 or cygwin, and equal to os otherwise.

Examples

My preliminary examples:

# opam-analyse v.1
arch:            x86_64
os:              linux
os-distribution: debian
os-release:      9.1
os-family:       debian

# opam-analyse v.1
arch:            x86_64
os:              osx
os-distribution: homebrew
os-release:      10.10.5
os-family:       osx

# opam-analyse v.1
arch:            x86_64
os:              win32
os-distribution: win32
os-release:      10.0.15063
os-family:       windows

# opam-analyse v.1
arch:            x86_64
os:              openbsd
os-distribution: openbsd
os-release:      6.1
os-family:       bsd

# opam-analyse v.1
arch:            armv7l
os:              linux
os-distribution: debian
os-release:      8
os-family:       debian

# opam-analyse v.1
arch:            aarch64
os:              linux
os-distribution: debian
os-release:      8
os-family:       debian

@AltGr
Copy link
Member Author

AltGr commented Oct 5, 2017

@ygrek @hannesm @avsm @thomasga -- it would help if you could run the above command:

curl -sL https://gist.githubusercontent.com/AltGr/5bfc8cea6f01e74b95de79ceaba39369/raw/opam_analyse.ml | ocaml unix.cma -stdin

@ygrek
Copy link
Contributor

ygrek commented Oct 5, 2017

    # opam-analyse v.1
    arch:            x86_64
    os:              linux
    os-distribution: debian
    os-release:      9.1
    os-family:       debian

    # opam-analyse v.1
    arch:            aarch64
    os:              linux
    os-distribution: android
    os-release:      8.0.0
    os-family:       android

@AltGr
Copy link
Member Author

AltGr commented Oct 6, 2017

Thanks for the confirmation that it works on android, I had trouble getting OCaml to work on my phone :)

@dbuenzli
Copy link
Contributor

dbuenzli commented Oct 6, 2017

os is normalised to "osx" on macs (instead of "darwin"). Not normalised on BSDs

This system is nowadays known under the name macos.

@dbuenzli
Copy link
Contributor

dbuenzli commented Oct 6, 2017

Here are values from my laptop and and a rpi2 running raspbian.

   # opam-analyse v.1
    arch:            x86_64
    os:              osx
    os-distribution: homebrew
    os-release:      10.12.6
    os-family:       osx

   # opam-analyse v.1
    arch:            armv7l
    os:              linux
    os-distribution: raspbian
    os-release:      9.1
    os-family:       debian

@dbuenzli
Copy link
Contributor

dbuenzli commented Oct 6, 2017

Note: I am not sure distribution is the best term in general, as that seems Linux-specific; but had no idea for a better name.

os-distribution seems fine, maybe os-pkg-distribution but it's longer and not sure the distinction is useful. At first sight it feels a bit strange on macos but I think it all makes sense in the end since a bare system has no notion of os package manager.

Should we add os-version, which would make more sense on non-Linux OSes, and point to the kernel version on Linux ? Or just rename dist-version to os-version, since we don't care about the kernel version ?

I think the way you handle version across systems under os-release is what a piece of software wanting to configure itself expects and would like to check: on windows and macos you are usually interested in testing an OS release and on linux a distribution release. So unifying them in the same field seems very sensible.

@AltGr
Copy link
Member Author

AltGr commented Oct 6, 2017

Ok, thanks for the input. I'll rename osx to macos then

@AltGr
Copy link
Member Author

AltGr commented Oct 10, 2017

Some more changes:

  • renamed os-release to os-version
  • decided to do more normalisation on arch, and make 32 bits archs explicit: it's now x86_32, x86_64, ppc32, ppc64, arm32, arm64, etc.
  • os-distribution is now equal to os rather than undefined when it couldn't be detected. So for example that would be macos when neither homebrew nor macports has been found.

@dbuenzli
Copy link
Contributor

dbuenzli commented Oct 10, 2017

I think it would be useful to get @dra27's input w.r.t windows identification. Currently we will have according to Sys.os_type:

# Sys.os_type = "cygwin"
os: cygwin
os-distribution: cygwin
os-family: windows

# Sys.os_type = "Win32"
os: win32
os-distribution: win32
os-family: windows

In the cygwin case, I'm wondering about treating cygwin as a proper os identifier and whether we should not rather use windows for the os and cygwin for the distribution (and get rid of that 32). That is have something like this for these two examples:

# Sys.os_type = "cygwin"
os: windows
os-distribution: cygwin
os-family: windows

# Sys.os_type = "Win32"
os: windows
os-distribution: windows
os-family: windows

Also @dra27 what are your toughts w.r.t. to depexts on windows ? Can cygwin be used to install depexts ? Or something else like chocolaty should be ? Are chocolaty and cygwin mutually exclusive ?

Btw. @AltGr on my side I have used wmic os get Version /value to get a windows version on Win32 whose output (key=value) is a bit less involved to parse than cmd /C ver.

AltGr added a commit to OCamlPro/opam that referenced this issue Oct 11, 2017
This closes ocaml#2919

This replaces the current obscure and verbose depexts format by lists of
system package names and filters based on opam variables, e.g. :

    depexts: [
      ["libssl-dev"] { os-distribution = "debian" | os-distribution = "ubuntu" }
      ["openssl-devel"] { os-distribution = "centos" & os-version >= "7.2" }
    ]

The old format still parses, and is rewritten by the package migration
function (older packages in 2.0 format won't be rewritten and may get
broken depexts, as we don't want to retain compatibility with beta
format versions)

All tags currently present in `opam-repository` are supported, except
for `source`, which is discarded (they were in the same namespace and
now need to be classified between `arch`, `os`, etc.)

Note that the rewrite needs to know about the variables defined and
their values. This is based on ocaml#3058 and
https://gist.github.com/AltGr/5bfc8cea6f01e74b95de79ceaba39369, version
4, but may still change.

Opam itself does not define these variables at the moment (except for
`arch` and `os`, but not with the same normalisation yet), and the
command-line format changes too, which means that current `opam-depext`
will be broken by the changes. The new format is

    opam list --external --vars arch=x86_64,os=linux,os-distribution=debian

instead of

    opam list --external=x86_64,linux,debian

which was more ambiguous and less generic, if shorter. The idea is that
the variables will later be built in opam, so the user won't need to
declare them for normal usage.
@AltGr AltGr mentioned this issue Oct 11, 2017
AltGr added a commit to AltGr/opam-depext that referenced this issue Oct 13, 2017
This is not backwards compatible. The system analysis and generation of
the variable is done according to
ocaml/opam#3058 (https://gist.github.com/AltGr/5bfc8cea6f01e74b95de79ceaba39369),
but is intended to be moved to opam at some point.

When that happens, this tool will be limited to handling the various
package managers, deciding from the opam variables using e.g. `opam var
os-distribution`.

Note that this drops support for "source" tags, which aren't supported
by beta5 anymore.
@hannesm
Copy link
Member

hannesm commented Oct 13, 2017

For FreeBSD systems above:

CURRENT from September:
    # opam-analyse v.4
    arch:            x86_64
    os:              freebsd
    os-distribution: freebsd
    os-version:      1200040
    os-family:       bsd

11.0
    # opam-analyse v.4
    arch:            x86_64
    os:              freebsd
    os-distribution: freebsd
    os-version:      1100122
    os-family:       bsd

11.1
    # opam-analyse v.4
    arch:            x86_64
    os:              freebsd
    os-distribution: freebsd
    os-version:      1101001
    os-family:       bsd

the other systems don't have any OCaml installed. The opam-analyse.ml uses String.lowercase_ascii which only exists since 4.03.00.

@AltGr
Copy link
Member Author

AltGr commented Oct 16, 2017

@hannesm thanks. Indeed I replaced them by String.lowercase while porting into opam-depext, for compat, ignoring the deprecation warnings (I would have recoded it, but then you run into String/Byte issues, and older OCamls don't even have String.map or equivalent).

AltGr added a commit to ocaml-opam/opam-depext that referenced this issue Oct 25, 2017
This is not backwards compatible. The system analysis and generation of
the variable is done according to
ocaml/opam#3058 (https://gist.github.com/AltGr/5bfc8cea6f01e74b95de79ceaba39369),
but is intended to be moved to opam at some point.

When that happens, this tool will be limited to handling the various
package managers, deciding from the opam variables using e.g. `opam var
os-distribution`.

Note that this drops support for "source" tags, which aren't supported
by beta5 anymore.
@UnixJunkie
Copy link
Contributor

Ubuntu LTS 16.04.3 on a 64bits machine:

    # opam-analyse v.4
        arch:            x86_64
        os:              linux
        os-distribution: ubuntu
        os-version:      16.04
        os-family:       debian

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants