src: add `--env-file-if-exists` flag #53060

BoscoDomingo · 2024-05-19T18:43:33Z

Add a new flag, --env-file-if-exists that allows loading .env files without throwing an error if they don't exist.

Achieved by returning an env_file_data struct instead of just the paths of the .env files. This tells the loader if the file itself is required or not, thus allowing to not throw an error if the file doesn't exist and is optional, without breaking existing behaviours with --env-file (we could maybe show a warning if an optional file is missing?).

Usage:

node --env-file A.env --env-file-if-exists B.env

If B.env exists, it will load its environment variables and overwrite those that conflict with A.env. If it doesn't, node will continue as normal.

Fixes: #50993
Refs: #51451

nodejs-github-bot · 2024-05-19T18:43:38Z

Review requested:

@nodejs/startup

anonrig

I think we shouldn't introduce a new flag. We should make the default --env-file not throw and revert the change.

BoscoDomingo · 2024-05-19T18:50:31Z

I think we shouldn't introduce a new flag. We should make the default --env-file not throw and revert the change.

@anonrig Are you sure? I am personally fine with #50588, and at this stage, we might be breaking people's CI processes and whatnot by allowing processes to run that will be missing environment variables. Multiple people in #49148 agree with this stance, too. I'd like to understand your reasoning

anonrig · 2024-05-19T18:54:15Z

@anonrig Are you sure? I am personally fine with #50588, and at this stage, we might be breaking people's CI processes and whatnot by allowing processes to run that will be missing environment variables. Multiple people in #49148 agree with this stance, too. I'd like to understand your reasoning

This is an "active development" non stable feature. therefore, breaking changes are expected.

IMHO, Instead of throwing an error, we can print a warning. I don't see why we should create a new flag rather than changing the current one.

BoscoDomingo · 2024-05-19T20:00:39Z

@anonrig Are you sure? I am personally fine with #50588, and at this stage, we might be breaking people's CI processes and whatnot by allowing processes to run that will be missing environment variables. Multiple people in #49148 agree with this stance, too. I'd like to understand your reasoning

This is an "active development" non stable feature. therefore, breaking changes are expected.

IMHO, Instead of throwing an error, we can print a warning. I don't see why we should create a new flag rather than changing the current one.

@anonrig I do see value in having a "hard" and a "soft" version, to allow users to decide what behaviour they desire. Plus the underlying implementation will be the same and thus barely any complexity is added while greatly improving UX/DX.

In my case, there are multiple instances where I'd want the node process to be aborted if the .env file is not present.

Is there any team we should tag to get their input on this as well?

anonrig · 2024-05-19T20:21:37Z

Is there any team we should tag to get their input on this as well?

There is no specific team responsible for dotenv. Originally, I implemented the dotenv parser and but we can always ask @nodejs/tsc for their opinion.

I think #50588 was a mistake. @mcollina and several others have expressed their opinion on this as well, but it never hurts to ask for their opinion one more time :-)

FYI, I didn't block this pull-request, any collaborator can come in and approve your pull-request, and land this pull-request.

BoscoDomingo · 2024-05-19T22:51:41Z

Is there any team we should tag to get their input on this as well?

There is no specific team responsible for dotenv. Originally, I implemented the dotenv parser and but we can always ask @nodejs/tsc for their opinion.

I think #50588 was a mistake. @mcollina and several others have expressed their opinion on this as well, but it never hurts to ask for their opinion one more time :-)

FYI, I didn't block this pull-request, any collaborator can come in and approve your pull-request, and land this pull-request.

Agree on asking others. I'm not an expert on this matter either, so happy to hear other POVs. I do err on the side of keeping --env-file as is for now, though

GeoffreyBooth · 2024-05-20T03:27:18Z

If we add this, the flag name needs to start with env-file like the others.

I feel pretty strongly that the current behavior is correct. We should not be changing --env-file. I'm also against adding this new flag but I feel less strongly about that if there's a reasonable need for it. I don't consider "it's annoying to make my command understand the difference between dev and prod" a persuasive enough reason.

mbrevda · 2024-05-20T05:53:16Z

I'm also against adding this new flag

There seems to be a consensus that:

No new flags should be added
There should be a mechanism to signal not to error if the env file isn't found

Is there any precedent for loader-style arguments? Perhaps prefixing (or suffixing) the path with a symbol (say, ?) can meet all the above requirements?

E.g. --env-file=?.optional.env

mcollina

lgtm

BoscoDomingo · 2024-05-20T08:09:58Z

@GeoffreyBooth Fair enough. You can see in #51451 the different workarounds we currently need to do. I believe for this to compete with dotenv, bun or other alternatives out there, the UX has to be at least matching, if not better. They all currently march on when the .env file is not found. We can do better.

In terms of changing the flag, good on my side 👍🏼

@mbrevda I feel like that's too much complexity both in use (I'd never seen such syntax) as in implementation (what happens with --env-file? path/to/file). It'd be a first AFAIK and I'm not sure such a small flag warrants such a divergence from the norm, but again, open to change my mind!

BoscoDomingo · 2024-05-21T22:15:35Z

Updated code according to the failed CI jobs. Should all be good now

@GeoffreyBooth Updated flag to start with --env-file

anonrig

I'm -1 on this change. It makes the existing implementation harder to maintain for almost no gain. We should not throw on env-file instead.

GeoffreyBooth · 2024-05-21T23:29:47Z

I will block on changing --env-file. We should not be silently ignoring failing to load a specified file, even if other tools do.

I'm also fine with no solution landing for this problem. I'm not convinced this is something that Node should solve, as opposed to users better managing their commands for production versus development.

anonrig · 2024-05-21T23:35:25Z

I will block on changing --env-file. We should not be silently ignoring failing to load a specified file, even if other tools do.

We can always convert it into a warning.

BoscoDomingo · 2024-05-22T10:59:10Z

I agree with @GeoffreyBooth on this one. I'd much rather Node doesn't allow the program to run than to start it with the wrong assumptions.

In my experience, startup warnings are rarely read in production environments, and usually not even properly marked as such by log services. It's far worse to let systems crash in prod due to missing files than to not start them to begin with.

However, the pain for users having to manage multiple scripts for whether they want to load x file or not will be a major pain (imagine all the possible combinations) when it's an easy fix for us. Therein lies the value IMHO.

Edit: @anonrig I like the idea of a warning for missing optional files

lewismoten · 2024-05-27T19:39:22Z

I would opt to have the flag not fail when the file is missing, have an option to say that the file is optional, or simply detect that a .env file exists without specifying a flag. With .env files, the industry standard is to proceed when its missing because environment variables may already exist. The main issue I see is with build servers that don't have the .env file. You end up adding extra steps in your build to create the file with each build in order to support projects that use it when environment variables are already available on the build server.

Shudrum · 2024-05-27T20:07:03Z

Hello there.

Basically, both of the solutions are going to achieve the same result. But I would also prefer to have the flag to not fail.

I think every people using .env files are currently using motdotla/dotenv, and its behavior is to not fail. Like said @lewismoten it's kind of an industry standard. The reasons I think of are:

It is strongly discouraged to commit .env files. So should not be mandatory.
It's named "environment" because well ... it should depends of the environment. And it should allow developpers to set their environments locally.
Many teams use environments to define default values for their applications, having a default one commited to their repositories. Having it missing is not frequent, and should be tested because...
...they frequently defines empty values, for the database for example. Throwing an error will not ensure of a proper configuration.
Having two dot files for overriding is frequent, making one of both optionnal.
I don't have the real statistics, but I'm sure at least 80% of deployed applications have their productions values defined from the running environment, like with Kube, AWS Fargate / ECS, GCP Run, Heroku, etc...
Secrets are rarely wrote into an env file.

I know it is still possible to use dot files for, maybe, all my examples, but with more boilerplates.

Personnaly, I never used dot env files outside of local developments. With this solution, for example, I cannot run the tests with the same npm script on my CI and locally. That's why I stay with dotenv.

I love to minimize libraries on my microservices, I really hope it will stop to throw.

BTW thank you for the contribution and all my thanks to the team!

BoscoDomingo · 2024-06-02T16:47:50Z

Just for reference: this issue will be discussed in the next TSC meeting since a consensus has not been reached. More info in the counterpart PR by @anonrig

Thank you all for the input, and we'll see how this one goes!

GeoffreyBooth · 2024-06-03T22:20:05Z

I assume the intended usage of this PR is:

node --env-file=maybe-exists.env --env-file-optional app.js

What if instead it’s:

node --env-file-optional=maybe-exists.env app.js

And if both --env-file and --env-file-optional are passed, Node throws an error that they can’t be used together.

BoscoDomingo · 2024-06-05T13:29:18Z

I assume the intended usage of this PR is:
node --env-file=maybe-exists.env --env-file-optional app.js
What if instead it’s:
node --env-file-optional=maybe-exists.env app.js
And if both --env-file and --env-file-optional are passed, Node throws an error that they can’t be used together.

@GeoffreyBooth No, the idea is to have --env-file-optional behave exactly like --env-file, but not throw an error when the file isn't found. E.g.

node --env-file this_exists.env --env-file-optional this_may_not.env

The flag was --optional-env-file originally, but I was asked to change it (so both show up together in the docs and to maintain consistency I assume)

aduh95 · 2024-06-05T14:26:31Z

I would be in favor of either non-boolean flag (e.g. --missing-env-file=throw), or a more explicit one (e.g. --throw-on-missing-env-file); my idea is that it would be good to have a flag to which we can flip the default while keeping it readable (because --no-env-file-optional is not very clear, --no-throw-on-missing-env-file is much better, and --missing-env-file=warn is shorter).
(I don't know if we are going to flip the default, but it's good to keep that option open)

BoscoDomingo · 2024-06-06T07:45:11Z

I would be in favor of either non-boolean flag (e.g. --missing-env-file=throw), or a more explicit one (e.g. --throw-on-missing-env-file); my idea is that it would be good to have a flag to which we can flip the default while keeping it readable (because --no-env-file-optional is not very clear, --no-throw-on-missing-env-file is much better, and --missing-env-file=warn is shorter). (I don't know if we are going to flip the default, but it's good to keep that option open)

@aduh95 I personally am not a big fan of flags that affect others, plus it doesn't really solve any of the issues why I proposed this change to begin with. One would still need multiple scripts for files that may or may not exist, and need to be aware of the context in which each is run to decide if the file will exist.

doc/api/cli.md

BoscoDomingo · 2024-09-01T09:33:57Z

@anonrig All ready to run CI. Should all be working fine now, except for the linter rule from #54663 👍🏼

src/node_options.cc

BoscoDomingo · 2024-09-01T22:07:51Z

Now that #54663 has landed I believe we can also merge this one in 👍🏼

aduh95 · 2024-09-01T22:38:38Z

Now that #54663 has landed I believe we can also merge this one in 👍🏼

You’d need to rebase your branch for the change to be picked up. It cannot land without a passing CI.

BoscoDomingo · 2024-09-02T15:22:57Z

Now that #54663 has landed I believe we can also merge this one in 👍🏼

You’d need to rebase your branch for the change to be picked up. It cannot land without a passing CI.

Done via GitHub for speed. See if it'll pass now 🙏🏼

aduh95 · 2024-09-02T15:28:26Z

It looks like the linter is still failing.

BoscoDomingo · 2024-09-03T19:44:55Z

It looks like the linter is still failing.

Murphy's Law strikes again... I mean the rule should've been removed if the other PR was merged, or am I missing something?

Edit: We were indeed missing something. See #54744

Fixes: nodejs#50993 Refs: nodejs#51451 test: remove unnecessary comment src: conform to style guidelines src: change flag to `--env-file-optional` test: revert automatic linter changes doc: fix typos src: change flag to `--env-file-if-exists` src: refactor `env_file_data` and `GetEnvFileDataFromArgs` test: clean up tests src: print error when file not found test: remove unnecessary extras

BoscoDomingo · 2024-09-11T07:53:13Z

@anonrig This one is (finally) ready to ship. Don't know if you have the power to merge it or if someone else should be tagged. Thanks for all the work!

nodejs-github-bot · 2024-09-11T09:30:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62306/

BoscoDomingo · 2024-09-15T21:24:30Z

Is there anything I have to do on my side or are those checks flaky and skippable?

nodejs-github-bot · 2024-09-15T23:07:40Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62466/

nodejs-github-bot · 2024-09-16T00:58:18Z

Landed in 53ede87

Fixes: #50993 Refs: #51451 test: remove unnecessary comment src: conform to style guidelines src: change flag to `--env-file-optional` test: revert automatic linter changes doc: fix typos src: change flag to `--env-file-if-exists` src: refactor `env_file_data` and `GetEnvFileDataFromArgs` test: clean up tests src: print error when file not found test: remove unnecessary extras PR-URL: #53060 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels May 19, 2024

BoscoDomingo mentioned this pull request May 19, 2024

.env file support issue tracker #49148

Open

8 tasks

anonrig reviewed May 19, 2024

View reviewed changes

mcollina approved these changes May 20, 2024

View reviewed changes

BoscoDomingo changed the title ~~src: add --optional-env-file flag~~ src: add --env-file-optional flag May 21, 2024

anonrig previously requested changes May 21, 2024

View reviewed changes

anonrig mentioned this pull request May 27, 2024

src: make --env-file return warning when not found #53177

Closed

BoscoDomingo commented Jun 6, 2024

View reviewed changes

doc/api/cli.md Outdated Show resolved Hide resolved

RedYetiDev added the dotenv Issues and PRs related to .env file parsing label Sep 1, 2024

aduh95 approved these changes Sep 1, 2024

View reviewed changes

src/node_options.cc Show resolved Hide resolved

aduh95 force-pushed the feature/optional-env-file-argument branch from f0b8ead to 5d2d76b Compare September 2, 2024 15:24

BoscoDomingo mentioned this pull request Sep 5, 2024

tools: remove readability/fn_size error throw #54744

Merged

aduh95 force-pushed the feature/optional-env-file-argument branch from 5d2d76b to 825c8de Compare September 7, 2024 23:45

aduh95 added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. request-ci Add this label to start a Jenkins CI on a PR. labels Sep 11, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Sep 11, 2024

anonrig approved these changes Sep 16, 2024

View reviewed changes

anonrig added the commit-queue Add this label to land a pull request using GitHub Actions. label Sep 16, 2024

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Sep 16, 2024

nodejs-github-bot merged commit 53ede87 into nodejs:main Sep 16, 2024
59 checks passed

RedYetiDev mentioned this pull request Sep 16, 2024

src: parse dotenv with the rest of the options #54913

Open

RafaelGSS mentioned this pull request Sep 16, 2024

v22.9.0 proposal #54966

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

src: add `--env-file-if-exists` flag #53060

src: add `--env-file-if-exists` flag #53060

BoscoDomingo commented May 19, 2024 •

edited

Loading

nodejs-github-bot commented May 19, 2024

anonrig left a comment

BoscoDomingo commented May 19, 2024

anonrig commented May 19, 2024

BoscoDomingo commented May 19, 2024

anonrig commented May 19, 2024

BoscoDomingo commented May 19, 2024

GeoffreyBooth commented May 20, 2024

mbrevda commented May 20, 2024

mcollina left a comment

BoscoDomingo commented May 20, 2024

BoscoDomingo commented May 21, 2024 •

edited

Loading

anonrig left a comment

GeoffreyBooth commented May 21, 2024

anonrig commented May 21, 2024

BoscoDomingo commented May 22, 2024 •

edited

Loading

lewismoten commented May 27, 2024 •

edited

Loading

Shudrum commented May 27, 2024 •

edited

Loading

BoscoDomingo commented Jun 2, 2024

GeoffreyBooth commented Jun 3, 2024

BoscoDomingo commented Jun 5, 2024

aduh95 commented Jun 5, 2024 •

edited

Loading

BoscoDomingo commented Jun 6, 2024

BoscoDomingo commented Sep 1, 2024

BoscoDomingo commented Sep 1, 2024 •

edited

Loading

aduh95 commented Sep 1, 2024

BoscoDomingo commented Sep 2, 2024

aduh95 commented Sep 2, 2024

BoscoDomingo commented Sep 3, 2024 •

edited

Loading

BoscoDomingo commented Sep 11, 2024

nodejs-github-bot commented Sep 11, 2024

BoscoDomingo commented Sep 15, 2024

nodejs-github-bot commented Sep 15, 2024

nodejs-github-bot commented Sep 16, 2024

src: add --env-file-if-exists flag #53060

src: add --env-file-if-exists flag #53060

Conversation

BoscoDomingo commented May 19, 2024 • edited Loading

nodejs-github-bot commented May 19, 2024

anonrig left a comment

Choose a reason for hiding this comment

BoscoDomingo commented May 19, 2024

anonrig commented May 19, 2024

BoscoDomingo commented May 19, 2024

anonrig commented May 19, 2024

BoscoDomingo commented May 19, 2024

GeoffreyBooth commented May 20, 2024

mbrevda commented May 20, 2024

mcollina left a comment

Choose a reason for hiding this comment

BoscoDomingo commented May 20, 2024

BoscoDomingo commented May 21, 2024 • edited Loading

anonrig left a comment

Choose a reason for hiding this comment

GeoffreyBooth commented May 21, 2024

anonrig commented May 21, 2024

BoscoDomingo commented May 22, 2024 • edited Loading

lewismoten commented May 27, 2024 • edited Loading

Shudrum commented May 27, 2024 • edited Loading

BoscoDomingo commented Jun 2, 2024

GeoffreyBooth commented Jun 3, 2024

BoscoDomingo commented Jun 5, 2024

aduh95 commented Jun 5, 2024 • edited Loading

BoscoDomingo commented Jun 6, 2024

BoscoDomingo commented Sep 1, 2024

BoscoDomingo commented Sep 1, 2024 • edited Loading

aduh95 commented Sep 1, 2024

BoscoDomingo commented Sep 2, 2024

aduh95 commented Sep 2, 2024

BoscoDomingo commented Sep 3, 2024 • edited Loading

BoscoDomingo commented Sep 11, 2024

nodejs-github-bot commented Sep 11, 2024

BoscoDomingo commented Sep 15, 2024

nodejs-github-bot commented Sep 15, 2024

nodejs-github-bot commented Sep 16, 2024

src: add `--env-file-if-exists` flag #53060

src: add `--env-file-if-exists` flag #53060

BoscoDomingo commented May 19, 2024 •

edited

Loading

BoscoDomingo commented May 21, 2024 •

edited

Loading

BoscoDomingo commented May 22, 2024 •

edited

Loading

lewismoten commented May 27, 2024 •

edited

Loading

Shudrum commented May 27, 2024 •

edited

Loading

aduh95 commented Jun 5, 2024 •

edited

Loading

BoscoDomingo commented Sep 1, 2024 •

edited

Loading

BoscoDomingo commented Sep 3, 2024 •

edited

Loading