Hermetic Builds: CI, bootstrapping and version pinnig

[vbfox]

This issue follow the twitter discussion on the subject of bootstrapping FAKE and how the global tool can be inadequate.

CC. @forki @matthid @BlythMeister @vivainio @Krzysztof-Cieslak @Rickasaurus @theimowski as you were involved in the discussion.

What do I/we want ?

Hermetic Builds citing the Google SRE Book :

Build tools must allow us to ensure consistency and repeatability. If two people attempt to build the same product at the same revision number in the source code repository on different machines, we expect identical results. Our builds are hermetic, meaning that they are insensitive to the libraries and other software installed on the build machine. Instead, builds depend on known versions of build tools, such as compilers, and dependencies, such as libraries. The build process is self-contained and must not rely on services that are external to the build environment.

It's hard to be perfect in this domain as at some point you might need to emulate hardware to ensure that a build is repeatable on a different system. But we should aim for most tools to work to enable that goal as much as possible.

FAKE as a tool that drive the build need to be aware of these problems and provide solutions for them. Even if they aren't the default as the needs of first time users and of more professional teams won't be the same there should be an easy, well documented path from one to the other.

There are 2 main aspects of FAKE affected :

• How FAKE itself come into the build, it's version and how we can ensure that from build to build it's a repeatable process.
• How FAKE search for tools and allow us to pin to specific versions of tools that are wrapped (By searching for local tools, by pinning versions in the build script, ...)

This issue will concentrate on the first problem as the changes between FAKE 4 and FAKE 5 affected it and I think that we need a discussion around it.

Where do we want it ?

Two main cases to distinguish :

• On build servers ideally the problem would be solved on build servers via a fully enclosed environment, the most typical being to use Docker build files. In such an environment even global, single version tools would do as we rebuild the environment from scratch each time.
  But realistically for technological and speed reasons build are often done on shared builders where having state affecting one build from the next is a big problem.
  Some machines even have multiple builders doing builds in parallel making it even worse.
• On developer machines where a developer can be expected to switch from project to project, or from branch to branch multiple time a day. Also potentially having builds running in parallel like long integration tests running on one branch while the developer is coding on another branch.
  Each of theses should be a repeatable, independent build. Other builds running at the same time or before shouldn't affect them and the build that will later happen on the build server for the same source code should produce the same result.

Where is FAKE now ?

In the FAKE 4 world the tool binary was provided from NuGet, typically via paket and it's version was pinned there, there was no problem with that approach as it fully moved the problem to paket itself (and it's bootstrapper).

FAKE 5 took the path of using a global tool and by doing that make version pinning quite hard, let's see the problems with the approaches presented :

• Global tool installation, used directly: A global mutable state, won't be repeatable on developer machines, the version isn't even stored in the repository making nothing repeatable. The good point is that the tool is available from any directory.
• Global tool locally installed with a script: It's the approach taken by the fake template tool, the version is set for first install but as it's not checked after it's not kept in sync.
• Chocolatey/system : Same problem as a global tool.
• Used as a dotnet tool (Either via paket or NuGet), placing a project file at the root and invoking via dotnet fake. The best solution for now. The potential problems are that it need a "dummy" project and only works on the root directory but that's not too much.

A (small) survey of the rest of the world

How are some other tools doing, and what solution do they offer :

• Npm, nuget, cargo have an easy way as they are delivered with their platform and just ride on it's version management. (Or absence of one in the case of npm)
• Yarn mainly propose a global install but also offer a yarn.js file that can be downloaded and placed in the repository. Upgrade is more annoying (And create a non-reviewable PR) but everyone uses the same version.
• Some npm tools like gulp offer a best of both world approach with a global install that use a local install if present or embed a fully functional version otherwise.

Proposed solution

I think that the long term solution when dotnet will get repo-tools is to do like gulp does. But in the interim we can still approach that level of integration.

A road toward that could look like this :

• De-emphasis or removal of most alternate routes like local scripting or chocolatey from the documentation to avoid confusing users.
• Place front and foremost the global tool approach as "How to start"
• Document a very specific way to do the local dotnet tool install especially specifying the filename (Build.proj or Repo.proj)
• Embed this convention in FAKE, making it search for this file in the parent hierarchy and if present (and containing the tool invocation) use dotnet fake instead of doing anything directly (It could also have a command line flag to generate it in the current folder).

Remark

I'll try to do a blog post on the subject but I also had some success by using fake libs in a project run via dotnet run it has some nice advantages and as I'm not the only one investigating here it could be worth it to have an advanced use page one the FAKE website documenting this practice.

But I don't think it should be cited prominently if it's one day suggested.

[Krzysztof-Cieslak]

TBF, I would just wait for making any more decisions and changes. MSFT is designing per-repo tools which should (I hope) solve many (all... if MSFT will design it correctly) mentioned problems.

[Krzysztof-Cieslak]

For reference- GH comment describing some plans for per-repo tools - https://github.com/dotnet/cli/issues/9782#issuecomment-410033494.

[vbfox]

Didn't see that recent comment by @KathleenDollard but it sounds great, repo tools in globals.json sure would bring the full gulp experience as a possibility (Or it could even be better if Microsoft choose to handle the recursive search for a local tool themselves as we won't even have to write that)

As Microsoft tooling is in an awkward position so are all global tools that interact with the build.

Waiting (Potentially with better docs) could be a solution but it depends on how much the wait is, and how easily people will be able to upgrade (For now core seem to be pretty upgrade-able)

Big advantage of well designed repo tools is that they would also solve the problem for a netcore-paket.

[matthid]

First: Thanks for the detailed write up. Now let me try to go into that:

FAKE 5 took the path of using a global tool and by doing that make version pinning quite hard

No we didn’t. What we did is provide additional installation option and they can ba very helpful in a lot of scenarios. And as they are “new” I guess people have the impression they are the “right” way to use fake 5 now.

Look at the linked fake-bootstrap repository. All of the suggested solution “lock” the fake version in one way or another. And one of them even allows you to version “dotnet-fake” in your paket lockfile.

The general workflow is (leaving out global tool for a second):

• git clone ...
• dotnet build // there is a small limitation in msbuild which makes us loose colorization

Or

• git clone ...
• dotnet restore
• dotnet fake build

It cannot really get easier than that. And the fake repository itself uses that.

Ok that said let me explain on how I see global installation. First, the term alone “global” clearly indicates that it isn’t what you want, so I’m not sure why you are using it?

Second, I think it is a new and easy way to get started with fake 5 and in particular its scripting capabilities. It allows us to use F# scripts pretty much everywhere without an existing fsi or dotnet sdk installation. This is why I feel we need to support chocolatey and the various native package managers.

Generally, I don’t think it is a huge problem using a global fake installation for building a repository on a local development maching as the impact of the runtime to your build script should be minimal (by design). Don’t forget that a global tool falling back to a local tool still has an nonzero impact as the fallback algorithm can fail or be faulty. The current approach with fake might be a bit higher but not too much. Let me explain a bit how fake 5 works:

• find dependencies (locked)
• compile the script (fsc)
• run the script in an issolated assembly context with the dependencies. The netstandard, FSharp.Core and one fake internal assemblies are overwritten by the runtime

So what can actually change between fake versions:

• fake chooses to compile with different assemblies -> paket bug?
• fsc produces different results -> f# bug
• runtime behaves differently -> netcore bug

Given the strict compatibility requirements from microsoft and the lockfile all are very very unlikely.

So I hope that enlightened my point of view on this.

[matthid]

So basically what's left to do:

• Wait for the dotnet sdk team to figure out repository tools
• Improve the documentation

What I'm probably not doing:

• Implement a local fallback to discover build.proj as there are too many possibilities for a local installation at the moment.

Any other things we can do to improve the situation?

[BlythMeister]

I personally think that using the global tool install with a pinned version in a script does exactly what it sounds like @vbfox wants.
With that script to get fake deleting the location where it will install to 1st.

That way you will always have the version you specified on your pin/lock.

It's not a true global tool, it's a repo level tool which is installed to a specific version

[BlythMeister]

Though I do agree with the improved documentation. I think we should agree on a couple of ways to install and we'll document it with examples (like @matthid examples repo)

I also think that the existing documentation is great I was able to follow it and I've never used the dot net cli before now.
But there was too much choice, so I tried every single example and worked out which fitted me best, which was rather cumbersome.
Some of the install methods are better and easier than others.

Imo, the best ways to install are:

• Paket cli
• dotnet tool
• Cinst

Each has a different use case, so if these are explained with the pros and cons, it gives people the best chance of choosing what fits their needs.

[kblohm]

We could also update the fake-template scripts to uninstall/reinstall in order to update. Sadly, dotnet tool update does not allow to specify a version...

[BlythMeister]

@kblohm yeah I found that issue with update, hence why I suggested using the script to purge the install location and then installing again.

[matthid]

Why not "ignore" global tools until we can properly use them. Why try to make them work and invest time in fixing the issues?

[BlythMeister]

Based on people consuming the modules from nuget, how much effort is it to push the cli into a nuget package which someone can consume via paket with a locked version?

[matthid]

@BlythMeister Isn't this exactly what I already linked above: https://github.com/FakeBuild/fake-bootstrap/tree/paket_clitool?files=1 ?

[BlythMeister]

Kinda, but I'm not sure using paket cli how you can lock to a specific version in the dependencies file...if that's possible, then I'm not really sure what the deal is with this issue.

[matthid]

It is:

• dependencies:

  https://github.com/FakeBuild/fake-bootstrap/blob/356a9f41daf6e83e45f6ccb60c48c8cd78810ddb/paket.dependencies#L3

• lockfile:

  https://github.com/FakeBuild/fake-bootstrap/blob/356a9f41daf6e83e45f6ccb60c48c8cd78810ddb/paket.lock#L10-L18

[matthid]

I'm not really sure what the deal is with this issue.

I can only assume that we need to fix/clarify the documentation. But I'm also not exactly sure what to do or what to write. Suggestions would be welcome.

[BlythMeister]

Ahh I used to use the paket cli route, but moved away...maybe that was better....the need for a paket reference file was a bit odd if I remember

[vbfox]

First, the term alone “global” clearly indicates that it isn’t what you want, so I’m not sure why you are using it?

One thing I want to be clear is that I don't think there is a problem of availability. There are ways to get hermetic builds, I know how to and I can isolate myself and just use them. But I think FAKE can do better than make it possible, it can make it easy and obvious.

The three things I wanted to discuss in this issue are:

• Fixing hermetic builds as a good thing and something FAKE should aim to help/encourage
• Better guidance on what are the good ways to install FAKE both when starting a new project from scratch, starting to use FAKE in an existing project and migrating from FAKE 4. With each time an hermetic build or a well documented way on how to get there.
• Potential ways to have a relatively simple evolution from first use to hermetic build or ideally first use being hermetic if the guidance is simple enough.

There has been an explosion of new ways to do things with cli tools, global tools, ... and FAKE can use a lot of them one way or another, there is more a problem of too much ways to do it than something missing (Except for simplicity maybe)

I shouldn't have ended my issue with a suggestion of change, and kept it on the guidance/documentation level. There are a few things that can be implemented but we can keep this discussion separate.

Wait for the dotnet sdk team to figure out repository tools

Yes I think that we can agree that going for a big complex solution, like the paket boostrapper would be overkill with Microsoft having a lot better one around the corner.

We could also update the fake-template scripts to uninstall/reinstall in order to update.

The problem with always reinstalling is that it isn't the fastest operation... far from it, having a local .proj file create bloat but is a much better thing to suggest until we get repo tools.

So what can actually change between fake versions:

• fake chooses to compile with different assemblies -> paket bug?
• fsc produces different results -> f# bug
• runtime behaves differently -> netcore bug
  Given the strict compatibility requirements from microsoft and the lockfile all are very very unlikely.

We had all 3 break one way or another, and we currently pin all 3. FAKE adding the possibility to break on either of these cases doesn't sound great (Granted it's better than if FAKE contained all it's libs)

I can only assume that we need to fix/clarify the documentation. But I'm also not exactly sure what to do or what to write. Suggestions would be welcome.

I'll try a PR for the website not sure you'll like it but maybe it will be more constructive 😉

[matthid]

I'm open for concrete PRs regarding documentation, guideline and implementation changes. However I'm closing this stale discussion, but feel free to open another issue if there are still some open points to discuss.

I'm fine with guiding users into a particular way of using FAKE as long as the other options are documented elsewhere and the way we guide users is "easy to follow" (whatever that means).