1bin - easily build “static” binaries of any CLI tool

1bin lets you easily build “static” binaries of pretty much any command line program.

It's companion website, 1bin.org hosts curl/wget friendly links that you can use like so: wget 1bin.org/$(uname)/<some-program>. These links are updated on each push to master to reflect any new tools or deletions. Pull requests to add new tools welcome.

Using the scripts here you can build pretty much anything that can be packaged with conda, which is... pretty much everything.

Current status: working POC. It does what it’s supposed to, but is held together with virtual duct tape — use at your own risk

Usage

If you just want to run programs without having to do any installation, head over to 1bin.org, find the program you’re after, and then paste the code snippet into a terminal on your machine to run it instantly (or at least as instantly as your ISP allows), no fuss no muss.

All binaries are also available on this project's Github Releases page for direct download if you prefer.

What is this?

An unholy agglomeration of some hairy bash scripting, github actions, conda, conda-pack and warp to convert most anything that can be packaged with conda into a self-extracting and relocatable “static” binary (static in scare-quotes, because it’s actually a self-extractor not a true static binary)

Some potential use cases:

• easily use popular command-line tools from inside restricted environments such as CI pipelines or docker containers on both Mac and Linux.
• easily install command-line tools for one off use-cases on someone else’s machine.
• make more portable shell scripts that can download any needed third party tools on the fly, regardless of what package managers the OS has provided.
• (relatively) easily create self-contained and easily deployable packages of your own software for distributing to your end users.

How does it work?

Conda is mostly known as a python package manager built to help install python packages with complicated native dependencies (a common use case when using python for machine learning and data science work) without having to recompile them yourself. However, a lesser known fact about conda is that in the process of solving this problem, they built an entire generic package management system that’s actually closer in spirit to tools like homebrew or apt-get than it is to other python env managers like poetry or pipenv.

Conda is able to create virtualenvs in the same way as most python env managers do, but unlike other python env management tools, conda environments are self-contained and can easily include other (non-python) binary tools. Most binary packages rely on hardcoded absolute paths, and as a result the packages they create not be easily relocated to run on another user’s machine. Conda however takes a different approach, and through some fairly arcane patchelf and rpath black magic is able to convert these hardcoded absolute paths into relative paths that can be easily transported to different machines and that don't depend on anything from the host machine (except for glibc).

They also released the wonderful conda-pack, which can convert an installed conda environment into a tarball. By combining conda and conda-pack with a modified version of fintermobilityas/warp, (a rust program to package a directory into a single executable) it’s possible to build fully self-contained binaries that can be run anywhere.

Under the hood, when you run a 1bin app, it uses my fork of warp to extract a conda-pack-ed environment into a local cache directory, and then runs the specified executable from there as a subprocess, passing any unix signals and path information down to the called subprocess.

1bin takes care of automating this process so that any package that is available on conda-forge (and there are a lot of them!) can be converted into a single self-extracting binary for macOS or Linux. For now it’s x86_64 only, but this approach could relatively easily be extended to build arm or mac M1 self-extracting executables in the same way, and with some work even support building Windows packages this way!

Contributing

PRs are very welcome! Adding new packages that already exist on conda-forge should be as easy as adding them to the list in build-all.sh. If the package name on conda-forge differs from the executable name, add them as a ‘complex’ package.

Once they’ve been added there and I’ve verified that the built binary works correctly on both macOS and Linux I’ll merge the PR and rebuild 1bin.org and they should be available for all to download.

There’s some nascent support for custom conda recipes as well, so if you have a tool that you’d like to have packed up as a 1bin but isn’t yet in conda-forge feel free to send in PRs with new conda-recipes and I’ll see that they get built.

Development / making your own packages for personal use

Install prereqs

Make sure you have the prerequisites installed:

• install conda
• tell conda to use conda-forge (pasting this shell snippet should do the trick):

  cat >$HOME/.condarc <<EOF
  always_yes: true
  channels:
      - conda-forge
      - defaults
  EOF

• install conda-pack: conda install conda-pack
• install my fork of warp-packer: (you can download it from https://github.com/nikvdp/warp/releases)
• if you are using an M1 mac, make sure you have the intel version of conda installed, this approach does not yet support M1/Applie Silicon natively
• make sure you have a recent (> 5.0) version of bash installed (NOTE: the version of bash provided by default on macOS is too old!) because build-all.sh uses associative arrays which are not supported on older versions of bash. (You can of course install a recent version of bash simply by doing wget 1bin.org/$(uname)/bash && chmod +x bash , and putting the new bash binary somewhere on your PATH 🙂)

Build your own executable locally

To build a package that’s available in conda-forge or the standard anaconda repos just do:

./build.sh <some-conda-package-with-an-executable>

eg. ./build.sh jq will put a jq executable in the ./out folder.

In some cases the package name may be different from the executable name. In this situation you need to pass the package name as the argument to build.sh and tell it which command to call from inside the package via the CMD_TO_RUN environment variable.

As an example, ripgrep is available under the package name ripgrep, but the executable name for ripgrep is rg. If you wanted to build a 1bin of ripgrep, you would run ./build.sh like so:

CMD_TO_RUN=rg ./build.sh ripgrep

This would leave an rg executable in the out/ folder.

Building private packages

If you are familiar with conda, it’s also possible to use 1bin to build binaries from private conda repositories or private source code. More detail to come on this, but the tl;dr:

• you first need to make a valid conda recipe for your package. Take a look at the conda-build documentation, or the custom-recipes folder for some examples
• because build.sh uses the --use-local flag when building, once you’ve built and installed your own conda package, you can use it with build.sh as normal eg build.sh <some-package-you-built>

Further work

• Use caching in the Github Actions workflows. Right now it works reliably but takes > 1h to build all packages. Most of that time is spent rebuilding packages which don't need to be rebuilt since they haven't changed since the last build
• Publish a build manifest with the package and version numbers of each package on each release. This could be used to provide metadata (rather than just the cli name) to 1bin.org
• Support for building (and downloading) different versions of each CLI tool when conda-forge has more than one version present in it's repos
• (experimental) Look into writing an adapter to allow re-packaging Nix packages as Conda packages. Nix generally has a wider and more reliable selection of packages than conda-forge, but they aren't relocatable. However, Nix does allow you to initialize a new Nix store at any location (nix run --store $PWD/mac-nix-store nixpkgs.nix), so theoretically a base Nix store could be created inside a conda-build environment, which would then theoretically allowing the building of anything in nixpkgs as a (relocatable) conda package that could be bundled up with 1bin.

FAQs

• Sometimes nothing happens or it takes a while for my CLI tool to run, what gives? This can happen the first time you run a particular 1bin on a new machine. Since under the hood 1bin is extracting a compressed copy of all the dependencies used for that particular CLI tool, the initial run can take a few seconds while it extracts (see “Where do the extracted files go?” below). Subsequent runs will be much snappier as it will then run the already extracted cache instead. In some cases (eg if you hit ctrl-c during the extraction process) warp may get confused and try to run the partially extracted version, leading to very strange errors. If this happens, try going to the warp/packages directory (see “Where do the extracted files go?” below) and deleting the partially extracted version so that it’ll be (fully) extracted next time you run the 1bin. (I’ll be updating warp in the future to be smarter about this kind of thing so that this doesn’t happen)
• Where do the extracted files go? warp caches them under ~/Library/Application Support/warp/packages on macOS and ~/.local/share/warp/packages on Linux. If you delete any of the folders under there they will be recreated the next time you run a 1bin.
• Is this safe? Depends on your definitions of safe. You can check the code, this repo is just a way to repackage the great work done by the wonderful community over at conda-forge into convenient to use binaries, but you are definitely trusting both me and the conda-forge communities to not do anything untoward. If you are concerned about this I’d recommend using another tool
• Is it just me or are these binaries pretty huge?? It’s not just you, they’re pretty big. Warp does compress the input, but for each of these programs the binary has to contain both the program itself and a full copy of every dependency required for it to run, and no attempts have (yet) been made to optimize this. For example, the docker-compose 1bin contains a full Python interpreter and every library that docker-compose itself uses inside that file, and so comes in at around 55mb for the Linux version. Given that disks are cheap and networks are fast these days, seems like a good trade to me, ymmv.