Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Packaging and distributing apps #8

Open
killercup opened this issue Feb 20, 2018 · 45 comments
Open

Packaging and distributing apps #8

killercup opened this issue Feb 20, 2018 · 45 comments
Assignees
Labels
A-distribution Area: Licensing, packaging, etc C-tracking-issue Category: A tracking issue for an unstable feature

Comments

@killercup
Copy link
Collaborator

killercup commented Feb 20, 2018

(moderated summary by @epage)

Context:

  • "Rust makes writing crossplatform, tested, modern command line applications frictionless while incorporating industry best practices and providing great documentation." (goal)
  • Focus is on CLI apps (see README for definition)

Assumptions and Implications

  • Majority of CLIs will at more than just Rust developers (for targeting rust-developers, see Improving binary distribution through cargo. #20)
  • Users of CLIs will be somewhat technical considering they are using a terminal
  • Use case priorities
    1. Using the app on the command line via the path
    2. Glued together with other apps in a script
    3. Drag and drop files onto an icon for the CLI
  • Examples of content to distribute

Other solutions

Plan of attack

  1. "How to package" documentation
  1. Automation
  • Implement stager
  • Reach out to binary packaging tools to consolidate effort with stager

In-ecosystem resources

Open Issues

Open Questions

  • Prioritized list of package formats per platform?
    • Linux: deb, rpm, flatpak, snap
      • Are flatpak / snap's apps added to PATH?
    • Windows: MSI via wix, chocolatey
    • Mac: homebrew
      • How easily are pkg's available from command line?
  • PPAs?
  • Use case, challenges, and priority of source packages?
  • Importance of remote administration and provisioning?

@killercup's original post

In the first meeting, we identified packaging and distributing CLI apps a something we should improve.

We need to have a best-practice solution for getting from "I have a crate with a binary" to "I can ship this app and don't have to worry about cross-platform installation stuff".

Edit 2018-03-19: This issue is about distribution of rust binaries without using cargo-install.

@killercup
Copy link
Collaborator Author

@killercup
Copy link
Collaborator Author

Distribute documentation:

@kbknapp
Copy link
Contributor

kbknapp commented Feb 20, 2018

@killercup
Copy link
Collaborator Author

From the etherpad:

  • Guides/How-Tos
    • Deb, RPM, AUR, Homebrew, Snap, Flatpak, AppImage, nixpkgs
  • Automation
    • cargo subcommands
    • Using CI

If cargo-install is an important use case for dev tools, how much should we look at improving it?

@XAMPPRocky
Copy link

If cargo-install is an important use case for dev tools, how much should we look at improving it?

I think this is part of a lot broader area of "crates.io's treatment of executables.". Which is something that previously has been very vague. Part of distribution should be codifying how crates.io handles and promotes executables or lack thereof.

I'm unsure how we should go about this as there isn't really a process for requesting crates.io features.

@jaemk
Copy link

jaemk commented Mar 2, 2018

For distributing my cli apps I made https://github.com/jaemk/self_update to support a "self update" command that pulls the latest github release built using https://github.com/japaric/trust/

@spacekookie
Copy link
Collaborator

spacekookie commented Mar 2, 2018

@killercup When it comes to automatically generating debian packages (and there is already cargo-deb for that) I think it's even more important to generate source packages for distributions to include (or to build a PPA with). And that is something that cargo-deb can't yet to.

See cargo-deb#40

@matklad
Copy link

matklad commented Mar 4, 2018

For windows specifically, it would be nice to be able to add manifest to the binary: rust-lang/rfcs#721.

@sophiajt
Copy link

sophiajt commented Mar 4, 2018

A couple more thoughts after brainstorming with @killercup:

With CLI apps that are built by doing "cargo install", we have a pretty straightforward path we could go down with regards to building installers for people.

For example, what if we had a new cargo plugin, or separate tool, that could take a "cargo install"-able CLI app and then, using a bit of convention over configuration, build you an installable package for a distro or use a simple Windows installer .exe?

This seems doable because there are really only a couple steps when doing CLI work specifically: "build my binary" and "put it in my path", which distros and installers should be able to handle no problem. @killercup mentioned that we might even want to go a step further and offer some project templates that drop in whatever metadata is required for building .deb, homebrew, etc. From there maybe the cargo plugin can either be configured and/or searches for packaging tools it knows about and builds using them.

These combined with something like https://github.com/japaric/trust/ means folks could take an easy extra step to see what tweaks they'd need to make to get it to work across more platforms.

@Screwtapello
Copy link

This seems doable because there are really only a couple steps when doing CLI work specifically: "build my binary" and "put it in my path", which distros and installers should be able to handle no problem.

Eh, that kind of depends on the tool, and how polished an experience you want it to be. Even static binaries might want to include manpages, generic documentation, default config files, command-line completion integration, data files...

I suspect no single tool is going to be able to support many or varied platforms, because platform conventions vary so widely, and because so few tool-developers are familiar with all possible deployment platforms. However, it would be great to have a set of base conventions that platform-specific tools can extend in different ways. For example, there could be a base convention that documentation lives in a docs/ subdirectory in Markdown format, and the build-posix-package tool could have an extra convention that a file named docs/manpage.md will be installed as the manpage for the crate's main executable.

@russel
Copy link

russel commented Mar 5, 2018

I got pointed at this issue by @killercup on https://users.rust-lang.org/t/cargo-and-installation-of-ancilliary-files/15974

My specific problem is installing a GNOME application. So not a CLI application per se, but very similar. Whilst many of the resources needed by the application are embedded in the executable, there are some that need installing. Manual pages and user documentation certainly, but also some image files and a GNOMEShell launcher.

Whilst my need is very specific, I am sure it acts as a microcosm of the more general situation.

Meson makes it very easy to install application – my experience is with D and C++, I have not tried using Meson with Rust. It is also fairly straightforward, but not always easy, to set up installation using SCons and CMake – but again my experience is with D and C++, to date I have only used Cargo with Rust.

@kbknapp
Copy link
Contributor

kbknapp commented Mar 5, 2018

@Screwtapello: I suspect no single tool is going to be able to support many or varied platforms, because platform conventions vary so widely, and because so few tool-developers are familiar with all possible deployment platforms.

This is my exact concern. I agree very much with the rest of your comment that having separate tools or conventions that people can pick/choose/extend would go a lot further than an "all in one" tool. This is with the caveat that those "separate tools" are well known, widely supported, well documented, etc. Not just one offs.

@jaemk: For distributing my cli apps I made https://github.com/jaemk/self_update to support a "self update" command that pulls the latest github release built using https://github.com/japaric/trust/

I love this! It may not work for everyone, or for complex use cases, but making it easy to have a "self update" feature of Rust CLI applications for "small utilties and binaries" would probably go a long way. Of course this doesn't solve the initial installation piece, but I still think it'd be great to include in a full blown solution. This particular piece could just be a standard guide on how to do this, such as a cookbook like recipe.

[about cargo-install in general as an install tool]

I'm on record saying I don't really like leaning on cargo-install for the general public, or general purposes. I think it's great for dev tools, or things that make sense to use during Rust development, but I personally don't want to expand it's use case to a general purpose installer.

I think there's too many nuances, and too much complexity to do everything correctly for it to all be contained in cargo. I'd like to keep cargo a Rust build tool, not a general purpose package manager. This isn't to say we shouldn't add certain features to cargo-install such as post build scripts...I'd just like to have proper guides/tools doing things the correct way first so that people don't just hit the easy button of cargo install which is what people are starting to do already.

Why?

Here's a few of the things I think would need to be sorted out (conversely why I think they shouldn't sorted out). This isn't meant to be an all encompassing list, just a quick search of some relevant issues. There may be more I can't think of off the top of my head, but here's a few:

Advanced or Post Build Steps

My primary concern with using cargo install too generally is that because of all the added complexity of correctly installing software, cargo-install takes the easy route and doesn't support complex/advanced use cases. Which is perfectly fine to me because it's not a general purpose installer. Until recently cargo-install didn't even support use of Cargo.lock files. As far as I know it still doesn't (easily) support more advanced build steps (like setting RUSTFLAGS, or like many have mentioned post build steps).

Crate Size

Including things like docs, manpages, completion scripts, licenses, axillary files, etc. which will be required for installation of end user products will bloat the crate packages. I'd actually like to do the opposite of de-bloating crates

Binary Only Features/Deps

Until cargo supports binary only features and deps, crates that include both a lib and bin will suffer from the above points when used as a lib (compile times, crate size, etc.)

Also, being able to specify binary dependencies is huge for a general purpose package manager.

Code Signing

This is another pretty big hole if we'd want to properly use cargo install as a general purpose installer.

@vitiral
Copy link

vitiral commented Mar 7, 2018

Just to include packagers, nixpkgs has pretty good support for rust ATM. I managed to get the novault nixpkg created, which required custom external dependencies.

The advantage of nixpkgs is that they can run alongside other package managers on any Unix distribution (including Mac) and I believe they are trying to support windows as well. This allows for at least some installation path for almost any platform/distribution, which is very valuable.

The process is pretty straightforward, with carnix creating the package from the Cargo.lock file. I think it's a workflow worth emulating for other packaging scripts as well.

@matthiasbeyer
Copy link
Member

Just to include packagers, nixpkgs has pretty good support for rust ATM.

Tell me more about this! The last time I tried to package something for nixpkgs, I failed hard. I used carnix and it created a package expression which OOM-Errored with Nix!

From what I think, Rust packaging with nixpkgs is hell!

@vitiral
Copy link

vitiral commented Mar 8, 2018

@matthiasbeyer this might fix your problem (just came across it): NixOS/nix#358

Edit: link to PR

@luser
Copy link

luser commented Mar 16, 2018

cargo install is nice for developers who already have Rust + cargo installed and are likely to have whatever -dev packages are required to build the app, but it's a bit heavyweight otherwise. Especially since you can wind up pulling in a lot of dependent crates to build a useful tool, which means the build time for your tool creeps up...

It would be nice to have best practices for alternatives as well. For sccache we've got binary releases being built in Travis + Appveyor for Mac/Windows/Linux:
https://github.com/mozilla/sccache/releases

Getting this right involved a bunch of fiddling which I'd be happy to help document somewhere:
https://github.com/mozilla/sccache/blob/master/.travis.yml
https://github.com/mozilla/sccache/blob/master/appveyor.yml

@epage
Copy link
Contributor

epage commented Mar 16, 2018

@luser I'm planning on documenting creating binary releases on my crate-ci docs

See https://crate-ci.github.io/

I put it on hold for the moment because I felt like documenting the process was a bit much. Because we need to know what the target is to find the binary, we have to retrofit the CI configuration in a fairly invasive way.

My hope is soon(ish) to start a tool to make it easier to get the files needed to make it easier to document
See crate-ci/meta#1

@XAMPPRocky
Copy link

XAMPPRocky commented Mar 18, 2018

@kbknapp I think what you've listed as reasons cargo shouldn't be used as a build, is exactly the reasons this working group should push for solving those problems as anything that hampers general development also hampers the development of developer tools. Saying "I'd just like to have proper guides/tools doing things the correct way first so that people don't just hit the easy button of cargo install which is what people are starting to do already." is honestly a bit elitist, if people don't have an easy way they won't work on the tools. No amount of documentation or templates can compete with a single line command.

The point of this working group is provide the easy way for developers to release and distribute their apps saying "Oh well you need to integrate in with this CI template, and add this dependency to be able for users to upgrade your app." isn't an easy way to release apps, it's tedious busy work. So what if the program is only available to Rust developers? That's a great way to get feedback, you can release a 0.1 through cargo and go to r/rust or IRC and ask them to try it out.

Unless you can somehow automate the process of releasing to all package managers at once with zero configuration any other option is worse than releasing through cargo-install. Saying that because cargo lacks features so it shouldn't be a package manager is a self fulfilling prophecy as we're the CLI working group and if we choose not to try to improve cargo as a package manager that will be used as reasons to not improve cargo-install.

@spacekookie
Copy link
Collaborator

@Aaronepower Well, that's all well and good and yea maybe there are some things in the cargo install pipeline that aren't very nice right now. But to target that at end-users is the wrong way to go about it.

Writing apps and CLIs specifically in Rust is so great because you're not dependent on any interpreter or runtime or toolchain. And then to say: install this toolchain to install the application is a bit…weird.

If there are any hangups with cargo-install then yea, we should of course fix them. For example a super-duper awesome feature would be that sys- package automatically install the required -dev packages on your platform so that compilation doesn't fail if they're not there.

But as an end-user experience, I don't want them touching cargo at all

@XAMPPRocky
Copy link

XAMPPRocky commented Mar 19, 2018

@spacekookie I feel like that is bit too much of generalisation on end users and the applications being made. Obviously if you're making a tool that helps write python applications it'd be a pain point to require cargo, and if you're a project that as big as ripgrep you need to provide other avenues of installation.

A lot of projects are Rust focused and even if they aren't when they start out they need a way to easily distribute their application to get feedback. I'm saying that even just as a quick way to get dev tools or small cli apps cargo-install needs to be held to a higher standard as I think improving that will have the most impact on developers releasing their applications.

@artem-zinnatullin
Copy link

Should we split this issue into two separate issues then?

One issue could focus on making Cargo more friendly for installing CLI tools written in Rust with Rust-dev community as target userbase.

Another issue could focus on documenting distribution of CLI tools written in Rust with different package managers (+ OSes and architectures) like Homebrew, apt, etc with regular non-Rust dev audience as target userbase.

@killercup
Copy link
Collaborator Author

This issue was never meant to cover cargo-install 😅

So, yeah, please go ahead and make that issue! Ideally with quotes from the comments here :)

@artem-zinnatullin
Copy link

@killercup if it wasn't, maybe you could update issue description to specify that it tracks distribution through package managers (or other solutions) to final non-Rust-dev users?

That'll help avoid confusion we had here :)

@ayosec
Copy link

ayosec commented Mar 19, 2018

For Linux systems, it may be good to choice either Snap or Flatpak, instead of trying to support all available package managers.

@spacekookie
Copy link
Collaborator

@ayosec We don't have to support all package managers.

Creating source deb files, source rpm files would be a great start. Yea sure, making snap and flatpak packages is also nice. But not everybody likes those approaches (me included 😜), so supporting "classic" Linux packages is also good.

Besides, deb (for bin packages) already exists 👍

@joshtriplett
Copy link

I do think that it's worth separating out two separate considerations here, as a few people have suggested.

We want to support building and installing CLI tools and all the accompanying bits that people expect, including manpages, completions, etc.

We also want to support building policy-compliant source packages for Linux distributions, so that distributions can ship Rust software just as easily as they can ship anything else.

@tblair
Copy link

tblair commented Mar 22, 2018

I know I'm a bit late to this discussion, but I looked into what it would take to build a Cargo extension to package up a CLI app a while back and thought I'd add a bit of what I came up with after researching the issue.

  • It's much easier to create an installer for the platform on which you're compiling. Cross compiling is a much more complex process since you can't rely on the tools that ship with the OS.

  • Creating a very basic installer is pretty easy on most platforms, but for more complex options that make installers feel professional like adding a logo or installing files beyond the single executable, the developer will need to supply additional metadata that's currently awkward to put in Cargo.toml.

  • Homebrew should not be looked at as an acceptable Mac install solution, or at least not the only solution. It's much more analogous to cargo install in that it's targeted at developers and other more-technical users. The likely best/default answer on Mac is a .pkg file assembled with the pkgbuild tool.

  • Creating a .msi installer on Windows will probably mean requiring developers install something like Wix since the VC++ toolchain seemed to require using the GUI. Documentation on this was a bit confusing, so I could be wrong on this one, but Wix support might still be necessary to support the MinGW use case.

  • Windows .msi and Mac .pkg are particularly important because they fit in with the first-party remote administration tools that IT departments use to provision their fleets of machines and keep them up to date.

  • I'm purposefully not saying much about the Linux side of this since there are a bunch of comments here that already cover everything I found for producing debs and rpms.

It's great that this issue is getting attention and I wish I had more time to pitch in and help.

@epage
Copy link
Contributor

epage commented Mar 31, 2018

@soc pointed out https://sbt-native-packager.readthedocs.io/en/stable/

It has some similar concepts to what I am looking at creating. Initially, I planned for the user-configuration to be siloed by the packaging targets but combining information, where possible, is possible in the my scheme and might be considered in the future.

If we instead adopted sbt, we'd still need to develop a way to make it easy to find the binary (see crate-ci/meta#1)

@soc
Copy link

soc commented Mar 31, 2018

@epage I'm not recommending adopting sbt, but rather having a look at the concepts and figure out which parts work for Rust. I think people are largely happy with how sbt-native-packager works, so it could make sense to check any design decision against it and evaluate whether its an improvement, a different approach, or something that could be improved upon.

@epage
Copy link
Contributor

epage commented Mar 31, 2018

@soc ok. My companies installers are actually similar and what I was modeling things after already. I do need to actually look to see whats different rather than just assume they are the same.

@berkus
Copy link

berkus commented Apr 2, 2018

flatpak all the things maybe? :)

@soc
Copy link

soc commented Apr 2, 2018

@berkus I'd rather prefer avoiding Flatpak, or at least supporting the "standard" means of distributing packages (deb, rpm, ...) as a preferred option.

It really irritates me is that the Flatpak developers not only knowingly violate the XDG base directory spec, but also try to obscure the fact that they are doing it.

It took half a decade to get to a state where one can reasonably assume that applications follow the standards of the operating system they run on. I have zero appetite for new applications regressing on that.

YMMV, though.

@gibix
Copy link

gibix commented Apr 5, 2018

There is also also a gentoo support for generating ebuild recipe from a cargo project: https://github.com/cardoe/cargo-ebuild.

@spacekookie
Copy link
Collaborator

spacekookie commented Apr 5, 2018

(slightly off topic) @soc not to mention that things like GUI applications packaged with flatpak will just ignore your gtk theme and use whatever the developers bothered to include (usually the windows 95 fallback)

@soc
Copy link

soc commented Apr 6, 2018

(offtopic) @spacekookie +1. The current state is just sad, and Flatpak doesn't do much to improve it, despite hijacking XDG environment variables – which is just a terrible thing to do.
Things would be so much nicer already if some distribution just declared "we are making $HOME read-only, fix your applications and stop dumping random files into the folder", and then people could use this as a starting point to further sandbox applications, making them easier to package and distribute.

@epage
Copy link
Contributor

epage commented Apr 20, 2018

For those interested, I've created a doodle for getting a real time conversation going about how we can align and share within the different packaging tools
https://doodle.com/poll/6x2wgpxznaf7b7g5

@epage
Copy link
Contributor

epage commented Apr 21, 2018

RE homebrew

@uwearzt pointed out https://github.com/Sean1708/cargo-brew but it just wraps cargo install which is limited.

@rcoh has a template for brews
https://github.com/rcoh/rust-brew-template

We talked about turning that into a crate-ci example and adding documentation for it in crate-ci docs.

Ideally, we'd also create a cargo brew (name pending since one exists) that will generate and update the tap files as needed.

@kornelski
Copy link

I've not used Mac pkgs […]. Do they easily support running from the command line, like putting their bin in the path?

On macOS there's command-line installer that can install .pkg headlessly without user intervention. It has parseable output. Mac Sparkle updater uses it to apply complex updates.

@eberkund
Copy link

eberkund commented Jan 9, 2019

Are flatpak / snap's apps added to PATH?

I know snaps are.

@jayvdb
Copy link

jayvdb commented Oct 28, 2019

In Open Build System, there are currently 282 crates already packaged. There wiki page about Rust only documents the bootstrap, not how to build crates, but looking at the spec it seems they are using rust2rpm.

Fedora has ~820 rust-* specs, so they are even further along, and that should be nearly enough that each CLI app only needs to add packaging for one or two missing deps before the CLI can be built.

@matu3ba
Copy link

matu3ba commented Aug 4, 2020

(offtopic) @spacekookie +1. The current state is just sad, and Flatpak doesn't do much to improve it, despite hijacking XDG environment variables – which is just a terrible thing to do.
Things would be so much nicer already if some distribution just declared "we are making $HOME read-only, fix your applications and stop dumping random files into the folder", and then people could use this as a starting point to further sandbox applications, making them easier to package and distribute.

That may only work, if you can like google with Android force them to simplify/let the user decide how to organize their (user)configurations, (user)data, (user)install setups, caching and the central lookup of these information.
This would mean 1.having a standard how to write these data (as file format) + 2.sandbox enforcing it + 3.tool for import/export/moving of datafolders.

Android/Apple etc are quite inflexible regarding structure, so something similar should not work.

I will suggest something like this to distri.

paulRbr pushed a commit to paulRbr/meta that referenced this issue Mar 29, 2021
@settings settings bot removed the tracking issue label Aug 23, 2022
@epage epage added C-tracking-issue Category: A tracking issue for an unstable feature A-distribution Area: Licensing, packaging, etc labels Aug 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-distribution Area: Licensing, packaging, etc C-tracking-issue Category: A tracking issue for an unstable feature
Projects
None yet
Development

No branches or pull requests