Reduce build times for both nannou and projects created via nannou-new

[mitchmindtree]

After running a few workshops, we've run into a couple of serious bottlenecks for new users getting started with nannou.

Note: This issue does not refer to automation of the "getting started" process, just the build times themselves. I'll open a separate issue for this.

1. Initial build time of nannou

After installing the rust toolchain, the first step is to clone the nannou repo and build it in order to run the examples. For users with relatively new machines, this takes around 3-5 minutes from scratch, which is already a long time in itself, however for users with older machines it can take up to 20mins (maybe longer)? This is a serious time-sink for workshops where we just want to get users running the examples and playing as quickly as possible.

Possible Solution

Ideally, we'd distribute pre-built versions of the library that are hashed against:

• the target triplet (architecture, OS)
• the latest, stable version of rustc and cargo used to build them.
• the version of nannou itself

Ideally, if a build does not exist with this hash, or if some sort of unexpected error occurs when attempting to link this crate, the build process would then fall back to the default cargo procedure of building from source.

An even more awesome solution would be if cargo itself decided to start shipping pre-built binaries, however this would obviously massively increase the cost due to requiring time and access to all platforms necessary for pre-building and hashing the binaries (whereas currently they just need to store and serve source code).

Having a system for building nannou for multiple platforms, storing and serving these binaries is also something that we will also have to solve, though is obviously much more achievable than doing so for all 10k+ crates or whatever the number is now currently on cargo.

Related issues:

• Support for pre-built dependencies rust-lang/cargo#1139

2. Build times of new sketches/apps generated via nannou-new

Cargo currently requires re-building the entire nannou package for each newly created "sketch"/"app" generated via the nannou-new tool. This is obviously a massive time-sink for all the reasons described above, though is particularly troublesome for a couple reasons:

• The idea of nannou-new is to allow the user to get started with sketching an idea as quickly and efficiently as possible. Waiting 20 minutes is easily enough time to forget about your original idea plus 20 others in the time it just takes to compile nannou again.
• The frequency of building new sketches is far more common than building nannou itself. Users are much more likely to just open a new processing file if just creating the nannou project itself take 20+ mins.

Possible Solution

This one could obviously be cleanly solved by the same solution as above (serving a pre-built crate). However, this problem could also be solved by re-using pre-built nannou crates that the user themself have already built. E.g. there should be no need to re-build nannou version X if a built version already exists on the user's system that was built using the same version of rustc and cargo. This is perhaps something we could do ourselves within the nannou-new call - e.g. search the user's system for an existing version and copy it to the target dependencies directory. I'm sure there might be other steps involved to do this properly and validly, but you get the gist.

Ideally this would also be a feature of cargo itself - allowing to share dependencies between unrelated crates.

Related issues:

• Allow to share versions of dependencies between unrelated Cargo packages rust-lang/cargo#5332
• Add an "airplane" flag to Cargo, altering it's dependency resolution behavior rust-lang/cargo#4686
• Offline mode rust-lang/cargo#5655

cc @freesig just thought I'd get all my thoughts down on this in case it helps! Would be such a massive win to get these solved.

[freesig]

SCCACHE might be relevant. I can't get it working on my system though. Installs but I can't get it to register any compile requests.
There is a good discussion on integrating into build systems that has a lot of relevant points.

This doesn't look that easy because currently Cargo has no support for linking prebuilt rlibs. On the other hand if we go straight to rustc then we loose all the good things about cargo. I think a major win could be had with sscache because it caches builds locally and on servers. It also integrates with cargo. It might need some improvements to get this to be reliable though.

Another option is to build all dependancies seperatly with cargo then combine them with our own tool. Not sure how hard this would be but it would allow us to still take advantage of cargo.

Todo

• Get sscache working
• Research other possible ways to cache builds
• Look into compiling everything separately with cargo and then combining with another tool.
• Look into cargo workspaces (as a temporary solution to doing workshops this could work well because all dependencies are shared through a workspace and not rebuilt)
• Related RFC

[freesig]

sccache

__Link
So this is a really promising tool for speeding up the current build times. I have made an anecdotal clean build of nannou go from 9:00 to 2:00.
It caches builds of crates when you first build them and then next time you need that crate it searches for an existing cache and uses that. There is also support for caching builds on a server, so we could use that to avoid users ever having to build the crates.

Issues

1. You can't get cache hits from a project in a different directory. Which defeats the purpose of using it with nannou-new. This seems to be due to the absolute paths being included in the rlib hashes that are used for matching a cache hit.
2. Not all crates can be cached. Maybe we could contribute to this problem aswell. But I got about 50% cache on a nannou build already.

Solutions

• Here is a discussion on how to fixing this problem in sccache. @mitchmindtree could you please give this a quick read when you get a chance. Most of it goes over my head but with you're guidance I'd be happy to try for a PR on sccache that fixes this directory issue. It looks like they actually have it working in the last comment.
  Just ran a test and nannou-new results in a cache miss every time due to FileNotInCache. However they are getting compiler hits. This is further evidence that the hashes are different due to file paths because the crates are cached.
• Running sccache with logging shows that the libs that it can't cache are either binaries, incremental or proc-macro

[mitchmindtree]

This sounds really promising!

Not all crates can be cached. Maybe we could contribute to this problem aswell. But I got about 50% cache on a nannou build already.

So the nannou crate itself cannot yet be cached, but many of the dependencies can? This is probably already a big win! Although it's obviously not as nice as caching the whole crate, it's still a big step :) I wonder if the culprit crates have strange build scripts or compiler flags... I can imagine it being much easier to cache "pure rust" crates than those that depend on system libs for example. Does it give you a list of those crates that could not be cached and the reason for each?

Running sccache with logging shows that the libs that it can't cache are either binaries, incremental or proc-macro

Ahh, so does disabling incremental compilation help with this? I guess this is not really a feasible solution in the long run though as incremental compilation gives such massive wins when working on new projects, especially when continually recompiling and iterating, etc. We'd be trading in one time-sink for another.

Regarding the issue you linked:

We have definitely discussed making cargo able to do this caching internally. It ought to be able to do even better than sccache by checking the entire dependency graph and caching the output of the entire build, so that if you have previously built the exact same source it should be a quick cache hit. That bug belongs on cargo, though. :) this comment

I think this is a pretty potent point, and would also mean we can avoid having to add extra 3rd-party magic. I guess sccache will have to do though for now if we can't find any promising WIPs in cargo itself!

It looks like this PR attempted to fix the issue but was blocked on another rustc issue which has since been solved. Hopefully we can get an update from luser and get a clearer idea where it's at now. Ahh I see what you mean, it looks like luser already has a branch with this fix working! @freesig have you been able to clone that branch and see if it fixes the problem for you? If it does, you should be able to open a "New Pull Request" on the sccache github repo and select luser's branch that has the fix - that might get the ball rolling!

Nannou "Getting Started"

One thing that I'd like for us to seriously consider first is how this will affect the "Getting Started" process. Although this will speed up the cargo builds, this will also add the extra step of installing and configuring sccache. Perhaps it's worth getting started on automating the "Getting Started" process so that all of this happens behind the scenes anyways. I'll open up an issue for "Getting Started" automation now.

[freesig]

Ok so I ran that branch with the changes to hash with nannou-new and still got zero cache hits

It's something in this function that is changing between different nannou-new builds. I will compare two runs and try to find the difference

Ok so I found these differences in the hash for two different nannou-new

("CARGO_MAKEFLAGS", "--jobserver-fds=3,39 -j --jobserver-auth=3,39 -j"), ("CARGO_MANIFEST_DIR", "/Users/tomgowan/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.0.2"), ("CARGO_PKG_AUTHORS", "Marvin Löbel <loebel.marvin@gmail.com>"), ("CARGO_PKG_DESCRIPTION", "A macro for declaring lazily evaluated statics in Rust."), ("CARGO_PKG_HOMEPAGE", ""), ("CARGO_PKG_NAME", "lazy_static"), ("CARGO_PKG_VERSION", "1.0.2"), ("CARGO_PKG_VERSION_MAJOR", "1"), ("CARGO_PKG_VERSION_MINOR", "0"), ("CARGO_PKG_VERSION_PATCH", "2"), ("CARGO_PKG_VERSION_PRE", ""), ("DISPLAY", "/private/tmp/com.apple.launchd.5uRdvee48L/org.macosforge.xquartz:0"), ("DYLD_LIBRARY_PATH", "/Users/tomgowan/rust/urieruiefu/target/release/deps:/Users/tomgowan/.rustup/toolchains/nightly-x86_64-apple-darwin/lib:/Users/tomgowan/.rustup/toolchains/nightly-x86_64-apple-darwin/lib"),

VS

("CARGO_MAKEFLAGS", "--jobserver-fds=3,38 -j --jobserver-auth=3,38 -j"), ("CARGO_MANIFEST_DIR", "/Users/tomgowan/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.0.2"), ("CARGO_PKG_AUTHORS", "Marvin Löbel <loebel.marvin@gmail.com>"), ("CARGO_PKG_DESCRIPTION", "A macro for declaring lazily evaluated statics in Rust."), ("CARGO_PKG_HOMEPAGE", ""), ("CARGO_PKG_NAME", "lazy_static"), ("CARGO_PKG_VERSION", "1.0.2"), ("CARGO_PKG_VERSION_MAJOR", "1"), ("CARGO_PKG_VERSION_MINOR", "0"), ("CARGO_PKG_VERSION_PATCH", "2"), ("CARGO_PKG_VERSION_PRE", ""), ("DISPLAY", "/private/tmp/com.apple.launchd.5uRdvee48L/org.macosforge.xquartz:0"), ("DYLD_LIBRARY_PATH", "/Users/tomgowan/rust/ksbdksbdkfdsjk/target/release/deps:/Users/tomgowan/.rustup/toolchains/nightly-x86_64-apple-darwin/lib:/Users/tomgowan/.rustup/toolchains/nightly-x86_64-apple-darwin/lib"),

Looks like the job server changes from 38 to 39 --jobserver-auth=3,38

Also this directory changes /Users/tomgowan/rust/ksbdksbdkfdsjk/target/release/deps:/Users/tomgowan/.rustup/toolchains/nightly-x86_64-apple-darwin/lib:/Users/tomgowan/.rustup/toolchains/nightly-x86_64-apple-darwin/lib

Link to job server docs
cargo make

Maybe this is fixed in a future version. The next step will be to merge the branch im using with the latest sccache master and see if that get's the same hash

[freesig]

Looks like a lot of these issues have been solved in updates to sccache.
Here is some topics on rust sccache

• Add support for caching Rust crate compilation mozilla/sccache#42
• On caching bins dangerously

I want to try and cash bins because even if it doesn't work all the time it could be a huge time saver in the times it does work.

[kazimuth]

I use a single project with a set of examples for my sketches, which allows them to re-use compiled dependencies. A workspace would also work; you'd just need to make sure to modify the right

See also my comment on #387, allowing nannou-new to use lld should speed up initial as well as incremental compiles. It does require that the user install lld though.