“The wand chooses the mage, remember.“

— Garrick Ollivander, Harry Potter and the Sorcerer’s Stone

A simple and powerful toolkit for Mage.

Features

wand is a toolkit for common and often recurring project processes for the task automation tool Mage. The provided API packages allow users to compose their own, reusable set of tasks and helpers or built up on the reference implementation.

• Adapts to any “normal“ or “mono“ repository layout — handle as many module commands as you want. wand uses an abstraction by managing every main package as application so that tasks can be processed for all or just individual commands.
• Runs any main package of a Go module without the requirement for the user to install it beforehand — Run any command of a Go module using the module-aware pkg@version syntax, or optionally cache executables in a local directory within the project root, using the gotool runner. See the “Command Runners“ sections below for details.
• Comes with support for basic Go toolchain commands and popular modules from the Go ecosystem — run common commands like go build, go install and go test or great tools like gofumpt, golangci-lint and gox in no time.

See the API and “Elder Wand“ sections for more details. The user guides for more information about how to build your own tasks and runners and the examples for different repositories layouts (single or “monorepo“) and use cases.

Motivation

Why Mage?

Every project involves processes that are often recurring. These can mostly be done with the tools supplied with the respective programming language, which in turn, in many cases, involve more time and the memorizing of longer commands with their flags and parameters. In order to significantly reduce this effort or to avoid it completely, project task automation tools are used which often establish a defined standard to enable the widest possible use and unify tasks. They offer a user-friendly and comfortable interface to handle the processes consistently with time savings and without the need for developers to remember many and/or complex commands. But these tools come with a cost: the introduction of standards and the restriction to predefined ways how to handle tasks is also usually the biggest disadvantage when it comes to adaptability for use cases that are individual for a single project, tasks that deviate from the standard or not covered by it at all.

Mage is a project task automation tool which gives the user complete freedom by not specifying how tasks are solved, but only how they are started and connected with each other. This is an absolute advantage over tools that force how how a task has to be solved while leaving out individual and project specific preferences.

If you would now ask me “But why not just use Make?“, my answer would be “Why use a tool that is not native to the programming language it is intended for?“. Make has somehow become a popular choice as task automation tool for Go projects and up to today I don‘t get it. Don‘t get me wrong: this is no bad talking against Make but a clarification that it is not intended for Go but rather for C projects, e.g. the Linux kernel, since Make is also written in C. Even Go itself is built using shell and Windows DOS scripts instead of Make. If you take a closer look, Make is nothing more than a DSL for shell commands so using shell/Windows DOS scripts directly instead is a way more flexible option. Therefore Make can not fullfil an important criteria: full cross-platform compatibility. The command(s) that each task runs must be available on the system, e.g. other tools must be installed globally and available in the executable search path, as well as requiring the syntax to be compatible with the underlying shell which makes it hard to use shell builtin commands like cd.

In my opinion, a task automation tool for a project should always be written in the same programming language that it is intended for. This concept has already been proven for many other languages, e.g. official tools like cargo for Rust and NPM for Node.js‘s or community projects like Gradle or Maven for Java. All of them can natively understand and interact with their target programming language to provide the widest range of features, good stability and often also ways to simply extend their functionality through plugin systems.

This is where Mage comes in:

• Written in pure Go without any external dependencies for fully native compatibility and easy expansion.
• No installation required.
• Allows to declare dependencies between targets in a makefile-style tree and optionally runs them in parallel.
• Targets can be defined in shared packages and imported in any Magefile. No mechanics like plugins or extensions required, just use any Go module and the whole Go ecosystem.

Why wand?

While Mage is often already sufficient on its own, I‘ve noticed that I had to implement almost identical tasks over and over again when starting a new project or migrating an existing one to Mage. Even though the actual target functions could be moved into their own Go module to allow to simply import them in different projects, it was often required to copy & paste code across projects that can not be exported that easily. That was the moment where I decided to create a way that simplifies the integration and usage of Mage without loosing any of its flexibility and dynamic structure.

Please note that this package has mainly been created for my personal use in mind to avoid copying source code between my projects. The default configurations or reference implementation might not fit your needs, but the API packages have been designed so that they can be flexibly adapted to different use cases and environments or used to create and compose your own wand.Wand.

See the API and “Elder Wand“ sections to learn how to adapt or extend wand for your project.

Wording

Since wand is a toolkit for Mage, is partly makes use of an abstract naming scheme that matches the fantasy of magic which in case of wand has been derived from the fantasy novel “Harry Potter“. This is mainly limited to the main “Wand“ interface and the “Elder Wand“ reference implementation. The basic mindset of the API is designed around the concept of tasks and the ways to run them.

• Runner — Components that run a command with parameters in a specific environment, in most cases a (binary) executable of external commands or Go module main packages.
• Tasks — Components that are scoped for Mage “target“ usage in order to run an action.

API

The public wand API is located in the pkg package while the main interface wand.Wand, that manages a project and its applications and stores their metadata, is defined in the wand package.

Please see the individual documentations of each package for more details.

Application Configurations

The app package provides the functionality for application configurations. A Config holds information and metadata of an application that is stored by types that implement the Store interface. The NewStore() app.Store function returns a reference implementation of this interface.

Command Runners

The task package defines the API for runner of commands. Runner is the base interface while RunnerExec interface is a specialized for (binary) executables of a command.

The package already provides runners for the Go toolchain and gotool to handle Go module-based executables:

• Go Toolchain — to interact with the Go toolchain, also known as the go executable, the golang.Runner can be used.
• gotool Go module-based executables — to install and run Go module-based main packages, the gotool.Runner makes use of the Go 1.16 go install command features.
  1. (Optional) Go Executable Installation & Caching — Go 1.16 introduced go install command support for the pkg@version module syntax which allows to install commands without “polluting“ a projects go.mod file. The resulting executables are placed in the Go executable search path that is defined by the GOBIN environment variable (see the go env command to show or modify the Go toolchain environment). The problem is that installed executables will overwrite any previously installed executable of the same module/package regardless of the version. Therefore only one version of an executable can be installed at a time which makes it impossible to work on different projects that make use of the same executable but require different versions.
  2. UX Before Go 1.16 — The installation concept for main package executables was always a somewhat controversial point which unfortunately, partly for historical reasons, did not offer an optimal and user-friendly solution until Go 1.16. The go command is a fantastic toolchain that provides many great features one would expect to be provided out-of-the-box from a modern and well designed programming language without the requirement to use a third-party solution: from compiling code, running unit/integration/benchmark tests, quality and error analysis, debugging utilities and many more. This did not apply for the go install command of Go versions less than 1.16. The general problem of tool dependencies was a long-time known issue/weak point of the Go toolchain and was a highly rated change request from the Go community with discussions like golang/go#30515, golang/go#25922 and golang/go#27653 to improve this essential feature. They have been around for quite a long time without a solution that worked without introducing breaking changes and most users and the Go team agree on. Luckily, this topic was finally resolved in the Go release version 1.16 and and golang/go#40276 introduced a way to install executables in module mode outside a module.
  3. UX As Of Go 1.17 — With the introduction in Go 1.17 of running commands in module-aware mode the (local) installation (and caching) of Go module executables has been made kind of obsolete since go run can now be used to run Go commands in module-aware by passing the package and version suffix as argument, without affecting the main module and not "pollute" the go.mod file 🎉 The pkg/task/golang/run package package provides a ready-to-use task implementation. The runner is therefore halfway obsolete, but there are still some drawbacks that are documented below. As of wand version 0.9.0 the default behavior is to not use a local cache directory anymore to store Gomodule-based command executable but make use of the module-aware go run pkg@version support! To opt-in to the previous behavior set the WithCache option to true when initializing a new runner.
  4. The Leftover Drawback — Even though the go install command works totally fine to globally install executables, the problem that only a single version can be installed at a time is still left. The executable is placed in the path defined by go env GOBIN so the previously installed executable will be overridden. It is not possible to install multiple versions of the same package and go install still messes up the local user environment.
  5. The Workaround — To work around the leftover drawback, the gotool package provides a runner that uses go install under the hood, but allows to place the compiled executable in a custom cache directory instead of go env GOBIN. It checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the node_modules directory of the Node package manager npm. Storing executables in a cache directory within the repository (not tracked by Git) allows to use go install mechanisms while not affect the global user environment and executables stored in go env GOBIN. The runner achieves this by temporarily changing the GOBIN environment variable to the custom cache directory during the execution of go install. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so the Validate method is a NOOP. This is currently the best workaround to…
     1. install main package executables locally for the current user without “polluting“ the go.mod file.
     2. install main package executables locally for the current user without overriding already installed executables of different versions.
  6. Future Changes — The provided runner is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the runner can be made opt-in or removed at all:
  • golang/go#44469 — tracks the process of making go build module-aware as well as adding support to go install for the -o flag like for the go build command. The second feature, mentioned in a comment, would make the "install" feature of this runner in (or the whole runner at all) obsolete since commands of Go modules could be run and installed using pure Go toolchain functionality.

Project Metadata

The project package defines the API for metadata and VCS information of a project. The New(opts ...project.Option) (*project.Metadata, error) function can be used to create a new project metadata.

The package also already provides a VCS Repository interface reference implementation for Git:

• VCS “Git“ — the git package provides VCS utility functions to interact with Git repositories.

Tasks

The task package defines the API for tasks. Task is the base interface while Exec and GoModule are a specialized to represent the (binary) executable of either an “external“ or Go module-based command.

The package also already provides tasks for basic Go toolchain commands and popular modules from the Go ecosystem:

• go-mod-upgrade — the gomodupgrade package provides a task for the github.com/oligot/go-mod-upgrade Go module command. go-mod-upgrade allows to update outdated Go module dependencies interactively. The source code of go-mod-upgrade is available in the GitHub repository.
• gofumpt — the gofumpt package provides a task for the mvdan.cc/gofumpt Go module command. gofumpt enforces a stricter format than gofmt and provides additional rules, while being backwards compatible. It is a modified fork of gofmt so it can be used as a drop-in replacement.
• goimports — the goimports package provides a task for the golang.org/x/tools/cmd/goimports Go module command. goimports allows to update Go import lines, add missing ones and remove unreferenced ones. It also formats code in the same style as gofmt so it can be used as a replacement. The source code of goimports is available in the GitHub repository.
• Go — The golang package provides tasks for Go toolchain commands.
  • build — to run the build command of the Go toolchain the task of the build package can be used.
  • env — to run the env command of the Go toolchain the task of the env package can be used.
  • install — to run the install command of the Go toolchain the task of the install package can be used.
  • run — to run the run command of the Go toolchain the task of the test package can be used.
  • test — to run the test command of the Go toolchain the task of the run package can be used.
• golangci-lint — the golangcilint package provides a task for the github.com/golangci/golangci-lint/cmd/golangci-lint Go module command. golangci-lint is a fast, parallel runner for dozens of Go linters that uses caching, supports YAML configurations and has integrations with all major IDEs. The source code of golangci-lint is available in the GitHub repository.
• gox — the gox package provides a task for the github.com/mitchellh/gox Go module command. gox is a dead simple, no frills Go cross compile tool that behaves a lot like the standard Go toolchain build command. The source code of gox is available in the GitHub repository.

There are also tasks that don‘t need to implement the task API but make use of some “loose“ features like information about a project application are shared as well as the dynamic option system. They can be used without a task.Runner, just like a “normal“ package, and provide Go functions/methods that can be called directly:

• Filesystem Cleaning — The clean package provides a task to remove directories in a filesystem.

Usage Guides

In the following sections you can learn how to use the wand reference implementation “elder wand“, compose/extend it or simply implement your own tasks and runners.

Elder Wand

The elder package is the reference implementation of the main wand.Wand interface that provides common Mage tasks and stores configurations and metadata for applications of a project. Next to task methods for the Go toolchain and Go module commands, it comes with additional methods like Validate to ensure that the wand is initialized properly and operational.

Create your Magefile, e.g magefile.go, and use the New function to initialize a new wand and register any amount of applications. Create a global variable of type *elder.Elder and assign the created “elder wand“ to make it available to all functions in your Magefile. Even though global variables are a bad practice and should be avoid at all, it‘s totally fine for your task automation since it is non-production code.

Note that the Mage specific // +build mage build constraint, also known as a build tag, is important in order to mark the file as Magefile. See the official Mage documentation for more details.

// +build mage

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/svengreb/nib"
	"github.com/svengreb/nib/inkpen"

	"github.com/svengreb/wand/pkg/elder"
	wandProj "github.com/svengreb/wand/pkg/project"
	wandProjVCS "github.com/svengreb/wand/pkg/project/vcs"
	taskGo "github.com/svengreb/wand/pkg/task/golang"
	taskGoBuild "github.com/svengreb/wand/pkg/task/golang/build"
)

var elderWand *elder.Elder

func init() {
	// Create a new "elder wand".
	ew, ewErr := elder.New(
		// Provide information about the project.
		elder.WithProjectOptions(
			wandProj.WithName("fruit-mixer"),
			wandProj.WithDisplayName("Fruit Mixer"),
			wandProj.WithVCSKind(wandProjVCS.KindGit),
		),
		// Use "github.com/svengreb/nib/inkpen" module as line printer for human-facing messages.
		elder.WithNib(inkpen.New()),
	)
	if ewErr != nil {
		fmt.Printf("Failed to initialize elder wand: %v\n", ewErr)
		os.Exit(1)
	}

	// Register any amount of project applications (monorepo layout).
	apps := []struct {
		name, displayName, pathRel string
	}{
		{"fruitctl", "Fruit CLI", "apps/cli"},
		{"fruitd", "Fruit Daemon", "apps/daemon"},
		{"fruitpromexp", "Fruit Prometheus Exporter", "apps/promexp"},
	}
	for _, app := range apps {
		if regErr := ew.RegisterApp(app.name, app.displayName, app.pathRel); regErr != nil {
			ew.ExitPrintf(1, nib.ErrorVerbosity, "Failed to register application %q: %v", app.name, regErr)
		}
	}

	elderWand = ew
}

Now you can create Mage target functions using the task methods of the “elder wand“.

func Build(mageCtx context.Context) {
	buildErr := elderWand.GoBuild(
		cliAppName,
		taskGoBuild.WithBinaryArtifactName(cliAppName),
		taskGoBuild.WithGoOptions(
			taskGo.WithTrimmedPath(true),
		),
	)
	if buildErr != nil {
		fmt.Printf("Build incomplete: %v\n", buildErr)
	}
}

See the examples to learn about more uses cases and way how to structure your Mage setup.

Build It Yourself

wand comes with tasks and runners for common Go toolchain commands, gotool to handle Go module-based executables and other popular modules from the Go ecosystem, but the chance is high that you want to build your own for your specific use cases.

Custom Tasks

To create your own task that is compatible with the wand API, implement the Task base interface or any of its specialized interfaces. The Kind() task.Kind method must return KindBase while Options() task.Options can return anything since task.Options is just an alias for interface{}.

1. If your task is intended for an executable command you need to implement the Exec interface where…
   • the Kind() task.Kind method must return KindExec.
   • the BuildParams() []string method must return all the parameters that should be passed to the executable.
2. If your task is intended for the main package of a Go module, so basically also an executable command, you need to implement the GoModule interface where…
   • the Kind() task.Kind method must return KindGoModule.
   • the BuildParams() []string method must return all the parameters that should be passed to the executable that was compiled from the main package of the Go module.
   • the returned type of the ID() *project.GoModuleID method must provide the import path and module version of the target main package.

For sample code of a custom task please see the examples section. Based on your task kind, you can also either use one of the already provided command runners, like for the Go toolchain and gotool, or implement your own custom runner.

Custom Runners

To create your own command runner that is compatible with the wand API, implement the Runner base interface or any of its specialized interfaces. The Handles() Kind method must return the Kind that can be handled while the actual business logic of Validate() errors is not bound to any constraint, but like the method names states, should ensure that the runner is configured properly and is operational. The Run(task.Task) error method represents the main functionality of the interface and is responsible for running the given task.Task by passing all task parameters, obtained through the BuildParams() []string method, and finally execute the configured file. Optionally you can also inspect and use its task.Options by casting the type returned from the Options() task.Options method.

1. If your runner is intended for an executable command you need to implement the RunnerExec interface where…
   • the Handles() Kind method can return kinds like KindExec or KindGoModule.
   • the Run(task.Task) error method runs the given task.Task by passing all task parameters, obtained through the BuildParams() []string method, and finally execute the configured file.
   • it is recommended that the Validate() error method tests if the executable file of the command exists at the configured path in the target filesystem or maybe also check other (default) paths if this is not the case. It is also often a good preventative measure to prevent problems to check that the current process actually has permissions to read and execute the file.

For a sample code of a custom command runner please see the examples section. Based on the kind your command runner can handle, you can also either use one of the already provided tasks or implement your own custom task.

Examples

To learn how to use the wand API and its packages, the examples repository directory contains code samples for multiple use cases:

• Create your own command runner — The custom_runner directory contains code samples to demonstrate how to create a custom command runner. The FruitMixerRunner struct implements the RunnerExec interface for the imaginary fruitctl application.
• Create your own task — The custom_task directory contains code samples to demonstrate how to create a custom task. The MixTask struct implements the Exec interface for the imaginary fruitctl application.
• Usage in a monorepo layout — The monorepo directory contains code samples to demonstrate the usage in a monorepo layout for three example applications cli, daemon and promexp. The Magefile provides a build target to build all applications. Each application also has a dedicated :build target using the mg.Namespace to only build it individually.
• Usage with a simple, single command repository layout — The simple directory contains code samples to demonstrate the usage in a “simple“ repository that only provides a single command. The Magefile provides a build target to build the fruitctl application.

Contributing

wand is an open source project and contributions are always welcome!

There are many ways to contribute, from writing- and improving documentation and tutorials, reporting bugs, submitting enhancement suggestions that can be added to wand by submitting pull requests.

Please take a moment to read the contributing guide to learn about the development process, the styleguides to which this project adheres as well as the branch organization and versioning model.

The guide also includes information about minimal, complete, and verifiable examples and other ways to contribute to the project like improving existing issues and giving feedback on issues and pull requests.

Copyright © 2019-present Sven Greb