Handle quotes in --env-file values consistently with Linux/WSL2 "source"ing

[markm77]

Hi there,

I use a .env file to supply env vars for my application both when running locally and in a Docker container.

When running my application in Linux/WSL2 (Ubuntu), I source the .env file before running.

Example:

set -o allexport
source ./myVars.env
set +o allexport

And when running my application in Docker Desktop on Windows I use the --env-file option to specify the .env file:

docker run --env-file  .\myVars.env <etc>

Unfortunately, some of the env var values in the .env file need to contain spaces. This means they must be double-quoted to work correctly when sourceing in Linux. However when quoted, the parsing in docker run does not remove the quotes making the values invalid as they contain literal quote marks at beginning and end.

The consequence of this is I need to maintain a second (mostly duplicate) .env file for Docker use.

I would much prefer if docker run could parse .env files in the same way as my Linux sourceing does and in line with the general shell concept that quotes are a guide to how a string should be interpreted - not part of the string itself.

From searching, it seems this has been fixed in docker compose (docker/compose#8388). Please can you also fix it in docker run?

[ToshY]

What's the status on this issue and the corresponding PR?

[ivan-kleshnin]

Swarm and Compose see the same value differently 😨

JWT_SECRET='{"type": "HS256", "key": "xxx"}'

$ docker compose up ... with .env_file

[container] $ echo $JWT_SECRET
{"type": "HS256", "key": "xxx"}

$ docker stack deploy ... with .env_file

[container] $ echo $JWT_SECRET
'{"type": "HS256", "key": "xxx"}'

[4nif]

Hopefully this can get fix soon

[TheodorRene]

Yes, this stumped me for a while as well. Worked nice locally, but not when dockerized. Even more confusing when compose would've handled the variables differently.

[ulbi]

Just run into the same problem, either the script works locally OR in the container, but never in both. This is really useless and forces keeping the same content twice. Please fix this soon.

[thaJeztah]

Let me copy my comment over from #4665 (comment)

The value should be unquoted as documented for docker compose (https://docs.docker.com/compose/compose-file/05-services/#env_file)

Unfortunately that format is a breaking change introduced in compose, and I wonder if that change was intentional or because they conflated the format for the .env file and --env-file, which are very different. The --env-file format was designed to be a plain NAME=value, no parsing, and no handling on quotes.

The only processing happening is if a NAME is provided without = / =value, the value of NAME is set from an $NAME env-var if present in the current environment; https://docs.docker.com/engine/reference/commandline/run/#env

[thaJeztah]

@glours do you know if the change in format for --env-file in compose was an intentional change (as it's diverging from docker run --env-file, and a backward-incompatible change)?

[0xGuybrush]

I've just spent a good while debugging an issue off the back of this, which is colouring my view, but I think this design choice is really confusing —

The --env-file format was designed to be a plain NAME=value, no parsing, and no handling on quotes.

At the moment, my natural inclination is to expect it to behave as per any other .env file parsing in other languages/frameworks.

The docs also make it sound as if -e and --env-file behave comparably (by grouping them together and not calling out the distinction), but the -e does parse quotes correctly (or at least they're parsed by the OS before assigning)

Personally, I'd much prefer if --env-file did unwrap quotes from the values, but if not then the 'principle of least surprise' would be to call it something else at least IMO.

[Elio-FairlyMade]

Any updates on this issue ?

[siriele]

I just ran into this bug today, and it feels like this command should be removed or corrected. It doesn't make sense as is. Especially not according to the documentation

[andersonbrands]

@thaJeztah @glours

Do we have any update on this?

[pveierland]

Issue is currently preventing use of the env_file feature with docker stack. Currently placing env file sourcing inside a startup script to avoid issue.

[phrfpeixoto]

I just found this annoyed with the fact that --env-file does not handle quotes. Now I can't use the same .env file for both docker compose and regular docker run commands.

Please fix this.

[pirafrank]

an alternative may be to mount the env file itself, instead of giving it to docker CLI, like this:

docker run -v ./.env:/usr/src/app/.env your:image

of course you need to customize path names to your env inside and outside the container.

[woutervh]

A .env-file in the root of a project can used by several different tools

• bash / zsh /...
• any dotenv-parser in any language (python, ...)

you can source a .env-file in bash via

set -a && source .env && set +a

Although not officially formalized, it is quite universally understood that a .env-file needs to have a bash-compatible syntax.
Nearly all dotenv-parsers correctly support quoting and variable substitution.
EXCEPT docker .... for over a decade.

Fixing the quotes only fixes some symptoms.

[mccolljr]

I second the notion that .env files are typically understood to support, at a minimum, quoted values. I use .env files with four or five different tools. I can use the same .env files for all of those tools. Docker is the odd one out, and it is certainly unexpected. If the existing behavior of --env-file needs to be preserved, can we have a separate option like --dotenv-file which can accept the same sort of .env files that other dotenv-themed tools accept?

[GLStephen]

This behavior is proving to cause all sorts of issues when moving between environments and interpretations of quotes.

[jlbribeiro]

To address Docker CLI's behavior I tried removing the quotes from my env file, which inevitably made all the other dotenv tools break due to some characters in the values. Not an option.

As a hackish workaround, I'm currently relying on the shell to interpret/import the env file as expected and, instead of using the --env-file flag, using multiple --env. Something along the lines of

# Docker is capable of passing individual environment variables by name.
# Assumes .env has already been sourced.

env_flags="$(cat .env | grep -v -e '^[[:space:]]*$' -e '^#' | cut -d '=' -f 1 | xargs -I{} printf "--env {} ")"
# env_flags=--env SOME_VAR --env ANOTHER_VAR 

# Usage:
docker run --rm -it \
  -v "$(pwd):/usr/src/app" \
  $env_flags \
  "my_image:latest"

# Explanation
# `grep -v -e '^[[:space:]]*$' -e '^#'` # removes empty lines and comments
# `cut -d '=' -f 1`                     # strips the value, only keeping the variable name
# `xargs -I{} printf "--env {} "`       # prefixes each variable name with `--env `

This is a very naïve idea (and the example implementation probably fails in many cases), but might be helpful to someone.
In case multiple calls across multiple scripts are necessary, I'd personally turn this into a function and/or individual script in PATH.

[kurtharriger]

I also ran into this issue when trying to run an expect script and found that my env vars had quotes in them

[ivanvmoreno]

Just ran into the same problem - Docker's .env parsing does not allow for non-escaped double quotes ", as opposed to Docker Compose's .env file syntax parser.

[xtaixe]

+1

[reorx]

welp, just fell into this trap out of nowhere

[brian316]

this approach is not intuitive. spent quite a while trying to debug this issue as the env file with quoted values works in docker compose but not regular docker run...

the following example demonstrates how it doesn't follow common sense:

.env

SECRET="mysecret"

works for docker compose using env_file

services:
  myimage:
    env_file:
      - .env

doesnt work for docker run using env-file

docker run --env-file .env myimage

[dscarmo]

Just adding to this, I ran into the same exact issue today, while trying to debug a service without running docker-dompose and just doing docker run.

If no breaking changes are being made on this, at least having an alternative command that parses "" would help!

[mou]

I fought for an hour against this feature. It feels strange that i can not pass files i use for user profiles.

[oalfred]

This behavior is really strange, docker run should be fixed.

[mou]

@thaJeztah A couple of years ago, you made a PR that didn't hit the mainline. Could you update this issue on its current status and explain why it is not fixed and looks like it never will be? I expect the Docker team to provide a workaround or reflect this issue in documentation. The perfect solution would be to have a warning emitted by docker CLI when it detects that the env file is in a traditional shell format.

[Lyanthropos]

Thank goodness for this thread, I thought I was crazy. Spent a week trying to figure out why an env file would cause a container crash and this turned out to be the issue. Having such a strong inconstancy between docker run and docker compose really breaks down the promise of portability. If there is no planned fix it would be nice to have a warning of it in both the CLI when --env-file is passed and in the docs.

[wimnat]

Importing an .env file when using Kubernetes behaves in a similar way. E.g. kubectl create secret generic my-secret --from-env-file=.env.

I just use sed to remove the quotes for me when running the docker or kubectl command.

Example

Here's my .env file.

❯ cat .env
MY_VAR="hello"
MY_SECOND_VAR="hello world"
MY_THIRD_VAR=noquote

Let's run nginx and confirm the problem.

❯ docker run --detach --name test --env-file .env nginx
❯ docker exec test env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=dc63cd8488db
MY_VAR="hello"
MY_SECOND_VAR="hello world"
MY_THIRD_VAR=noquote

❯ docker stop test && docker rm test

Now let's use sed on the way in.

❯ docker run --detach --name test --env-file <(sed 's/="\([^"]*\)"/=\1/' .env) nginx
❯ docker exec test env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=0725b2f337f1
MY_VAR=hello
MY_SECOND_VAR=hello world
MY_THIRD_VAR=noquote

❯ docker stop test && docker rm test

Much better! The .env file is left untouched for sourceing or whatever else you want to do.

While having to remember the sed syntax isn't the best ux, as I'm using this as part of a scripted environment, it works for me. Hopefully some others can find this useful.

Note that this is GNU sed. If you're using macos sed it should still work without modification.