Definition:
noun: mess; scramble; a confused conflict verb: to put into disorder; to make messy
Or if you prefer, an acronym:
Manage User Selected Services
muss is a command line application to allow users to choose what services
they want to run and how they want to run them.
muss is a wrapper around docker-compose.
When you run a muss subcommand it will:
- process (and validate) the config files
- generate a
docker-compose.yml - prepare any defined bind mounts (volumes)
- load secrets into the environment
- delegate to
docker-compose
Projects (and their dependent services) can be configured in a familiar style through a series of yaml files.
brew tap instructure/muss
brew install muss
muss is a CLI written in go.
make install to install it into your $GOPATH.
You can use make test to test all of the included go packages
or you can run any go commands directly (go test ./cmd).
muss help will describe available subcommands.
Many docker-compose commands are supported directly so that config files are always up to date and secrets are loaded into the environment.
The simplest possible usage is to call muss up
to start all the services and type Ctrl-C to shut it down.
$ muss help
Configure and run project services
Usage:
muss [command]
Available Commands:
attach Attach local stdio to a running container
build Build or rebuild services
config muss configuration
dc Call aribtrary docker-compose commands
down Stop and remove containers, networks, images, and volumes
exec Execute a command in a running container
help Help about any command
logs View output from services
ps List containers
pull Pull the latest images for services
restart Restart services
rm Remove stopped containers
run Run a one-off command
start Start services
stop Stop services
up Create and start containers
version Show version information
wrap Execute arbitrary commands
Flags:
-h, --help help for muss
Use "muss [command] --help" for more information about a command.
For less common docker-compose commands that muss does not define
you can use the dc subcommand, e.g. muss dc events --json.
You can also run any arbitrary commands using muss wrap.
This allows you to run a command after files have been generated and
the environment has been loaded.
muss has its own config subcommand (different from the docker-compose
config command).
muss config save will generate the files. This happens before running other
commands (like muss up) but this can be useful if you just want to inspect the
files.
muss config show will print out the whole configuration. The --format
parameter takes a go template string to allow you to limit or manipulate the
config (useful for scripting and debugging).
A muss project is configured via a series of yaml files:
- project config
- user config
- module definitions
The project configuration defines:
- default preferences for module configs
- secret commands
- status command
- the locations of additional config files
- docker-compose settings
This file should be committed into source control.
A muss user file allows you to specify preferences and customizations:
- specify a different module order than the project's default
- select specific module configs
- disable modules that you don't intend to use
- a compose override map to be merged in at the end
This file should be ignored by version control.
The project file can include the location of module definition files, each of which:
- has a name
- defines one or more config options for how to run the service (example use cases would be local repos, pre-baked docker images, or remote services).
The syntax and options for the main muss.yaml file:
---
# Path to user customization file.
user_file: muss.user.yaml
# If present will set COMPOSE_PROJECT_NAME (unless already set).
project_name: myproject
# Alternate target for compose config (default is docker-compose.yml).
# If present will set COMPOSE_FILE (unless already set).
# Note that when COMPOSE_FILE is set docker-compose will not automatically
# use docker-compose.override.yml.
# The override section of the muss user file will still work, however.
compose_file: "docker-compose.muss.yml"
# Define the order of which configuration option to use
# for any module that has multiple options.
default_module_order:
- registry
- repo
- internal
# Module files are yaml files containing module definitions.
module_files:
- ./dev/database.yml
- ./dev/microservice/service.yml
# Secret commands define aliases that can be used by module definitions.
secret_commands:
vault:
# Arguments will be prepended to the secret arguments.
exec: ["vault", "kv", "get", "-field"]
# Env Commands will be run once to setup the environment
# if any secrets are requested.
env_commands:
# When you specify a varname the output of the command
# will be set in that environment variable. If that variable
# is already set in the environment the command will not be run.
- varname: VAULT_TOKEN
exec: ["bin/vault-token"]
# A passphrase is required for local caching of the secrets.
# Use an env var representing your auth token.
secret_passphrase: $VAULT_TOKEN
# A status line will be fixed to the bottom of the screen during "up".
status:
# Stdout from this command will appear in the status line.
exec: ["bin/muss-status"]
# Each line of status output will be formatted with this spec:
line_format: "# %s"
# Specify how often to execute the command to update the status:
interval: 5sUsers can customize what they want to run with a user file:
---
# Users can set their own order for which module config to use.
# This list will be used before the default_module_order of the
# project config.
module_order:
- remote
- repo
# Modules can also be chosen specifically (to override order lists)
modules:
microservice:
config: remote
# ...or removed completely.
stats:
disabled: true
# An override section can be defined that will be merged onto the
# docker-compose config. By defining it here, muss extensions (like file
# volumes) can be utilized.
override:
services:
app:
environment:
HOW_I_LIKE_IT: niftyModule files contain module definitions that will generate chunks of a docker-compose file (which will all be merged on top of each other).
A module can define multiple configs.
The names of the configs are arbitrary;
These are the values used in the
module_order and default_module_order lists.
Use common names among your modules so that the
order options apply as consistently as possible.
In the example below:
- "local" implies "run from local directory"
- "registry" implies "an image from a docker registry"
- "edge" and "staging" are different options for external integrations
When multiple configs are defined the option will be chosen in this order:
MUSS_MODULE_ORDERenv var (split on commas)- a specific user choice
- the first of any
module_order - the first of any
default_module_order
The body of a module config can contain the following:
- "include" is a list of items to merge in before the rest of the map
and can be:
- a string referring to another config name
- a map of
file: pathto read in a file (relative to muss.yaml)
- "secrets" is a list of secrets to load
- "services" is a subset of the "services" section of a docker-compose configuration... it will be passed through.
- "volumes" is also just a piece of docker-compose syntax that will be passed.
---
# Module name.
name: microservice
configs:
# Configs with a leading underscore are private/internal
# and can be used as common bases for merging.
_base:
# The "services" map will be merged into the final docker-compose yaml.
# We define attributes of multiple services here that will all be merged
# on top of each other (the way a docker-compose override file works).
services:
app:
environment:
# The "app" service (defined more fully elsewhere)
# will only receive this environment variable
# if this service definition is enabled.
MICROSERVICE_ENABLED: '1'
microservice:
environment:
APP_ENV: 'development'
local:
include:
# A config can start by including other maps that will be merged in first.
# This can be the name of another config
- _base
# or the contents of another yaml file.
- file: compose-snippet.yml
# Any "volumes" will also pass to docker-compose.
volumes:
microservice-data: {}
services:
microservice:
# Paths will be relative to the project root.
build: ../microservice
volumes:
- microservice-data:/var/lib/microservice
registry:
include:
- _base
services:
microservice:
image: our-registry/microservice
_remote:
services:
# This service can define how another service will reach it.
app:
environment:
MICROSERVICE_URL:
MICROSERVICE_KEY:
# Note that there is no "microservice" here.
# By pointing to a remote instance
# we have eliminated the need to run it locally.
edge:
include:
- _base
- _remote
# A config can define secrets that will be loaded into the environment.
secrets:
MICROSERVICE_URL: {vault: ["MICROSERVICE_URL", "app/edge/common"]}
MICROSERVICE_KEY: {vault: ["MICROSERVICE_KEY", "app/edge/common"]}
staging:
include:
- _base
- _remote
secrets:
MICROSERVICE_URL: {vault: ["MICROSERVICE_URL", "app/staging/common"]}
MICROSERVICE_KEY: {vault: ["MICROSERVICE_KEY", "app/staging/common"]}Any "services" defined that do not contain a "build" or an "image" will not be included in the final docker-compose file. This allows one service to define what secrets are available for another service even if that service won't be used in the end.
Service integration can now be organized around how services will be selected. All of the docker-compose configuration for a given service can be placed in the same module file, including not only that service (or multiple services) but also how it integrates with another service.
For example, if a microservice can be local, remote, or completely optional, the definition file can point the environment variables for other services to this one. That way they update according to which ever option is chosen, or the service can be completely disabled and the other services won't receive the environment variables at all.
As another example, you could have a stats container that collects and prints stats from all other services. If the stats service file defines not only the stats service but also the environment variables for all the other services that will send stats to it, then you can simply disable the stats service from your user config file and all the other services will no longer be configured to send any stats.
Module definitions can specify secrets to be loaded by external commands.
The project configuration defines aliases that simplify the usage and define commands that setup the environment (for example, logging in to vault and returning the token).
When a module config is chosen that includes secrets:
- the setup commands are executed and the environment is populated
- the individual secret commands are then run and added as environment variables
- muss then delegates to the subcommand (docker-compose, etc).
Secrets are cached and encrypted with the secret_passphrase.
So if you use your auth token as your passphrase the secrets will be cached
for as long as your token is valid. When you get a new token it will force
fetching new secrets.
Secret commands can either specify a varname and the STDOUT of the script
will be assigned to that var.
Alternatively the commands can specify: parse: true
and the output will be parsed as lines of NAME=VALUE.
STDIN and STDERR will pass directly so that users can response to password prompts and see errors.
To provide a more concrete example:
muss.yaml:
secret_commands:
vault:
exec: ["vault", "kv", "get", "-field"]
env_commands:
- exec: ["bin/vault-token"]
parse: true
# The passphrase will be parsed and env vars will be interpolated.
passphrase: $VAULT_TOKEN
# The "cache" value can be "passphrase" (the default)
# "none" to disable caching, or a duration ("24h", "168h")
# to expire the cache (if the passphrase hasn't already changed by then).
cache: "passphrase"
# You can set a global passphrase that will be used for any secrets
# that do not define their own.
secret_passphrase: $VAULT_TOKEN
module_files:
- dev/microservice.yml
The exec command can be something global but will often be a command that is part of your project.
The bin/vault-token script would probably do something like this:
#!/bin/bash
# Setup any necessary env like (perhaps VAULT_ADDR).
export VAULT_ADDR=...
if ! token-is-valid; then
vault login ...
# persist the token and expiration so that next time
# the script runs it won't have to login again.
fi
# These lines will be parsed and set in the environment.
echo VAULT_ADDR="$VAULT_ADDR"
echo VAULT_TOKEN="$(< ~/.vault-token )"
The actual secrets in the module definition (dev/microservice.yml):
name: microservice
configs:
somewhere-far:
secrets:
SECRET_KEY: {vault: ["MICROSERVICE_KEY", "path/to/key"]}
services:
app:
environment:
SECRET_KEY: # no value, it will get it from the environment.
So when the "somewhere-far" option is chosen for the module it will load those secrets.
First bin/vault-token will be executed and VAULT_ADDR and VAULT_TOKEN
will be set in the environment.
Then vault kv get -field MICROSERVICE_KEY path/to/key will be executed
and the value stored in the SECRET_KEY environment variable.
Then muss will continue and delegate to docker-compose to run your services
and the populated environment variables will be passed along.
A few additional behaviors are defined beyond the normal docker-compose features.
When bind mounts (host volumes) are specified muss will attempt to ensure that they already exist. This helps prevent permission problems that can occur by letting the docker daemon create a directory under your project directory.
Additionally volumes can be specified to be files instead of directories
which can be useful for mounting config files into the container.
Just use the long syntax for volumes and add file: true:
volumes:
- target: /home/docker/.somerc
source: ~/.somerc
file: true
This way if the specified path isn't already a file muss will create it so that docker doesn't make it a directory and cause confusing errors later.
The file: true will be removed from the resulting docker-compose file.