[DotEnv] Add a new loadForEnv() method mimicking Ruby's dotenv behavior #28533

dunglas · 2018-09-21T06:56:31Z

Q	A
Branch?	master
Bug fix?	no
New feature?	yes
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets	helps for symfony/recipes#465, symfony/recipes#408
License	MIT
Doc PR	todo

This PR adds a new loadForEnv() method that mimics the behavior of Create React App, Rails' DotEnv and probably some other libs:

DotEnv::loadForEnv() will load the following files, starting from the bottom. The first value set (or those already defined in the environment) take precedence:

.env - The Original®
.env.dev, .env.test... - Environment-specific settings.
.env.local - Local overrides. This file is loaded for all environments except test.
.env.dev.local, .env.test.local... - Local overrides of environment-specific settings.

The plan is to use this method in the default SF installation (symfony/recipes#466).

Note: this feature can also be used in prod if the app doesn’t follow the 12factor methodology (discouraged, but sometimes mandatory in constrained hosting environments not allowing to set environment variables).

ogizanagi · 2018-09-21T08:44:33Z

src/Symfony/Component/Dotenv/Dotenv.php

+    /**
+     * Loads one or several .env and the corresponding env.local, env.$env and env.$env.local files if they exist.
+     *
+     * The env.test.local is always ignored because tests should produce the same results for everyone.


I don't think we should put any restriction and hardcode this. If that's really what the users want, that's on them. I'd remove it.

I was hesitating, but Create React App, Rails and Sinatra behave like this. Consistency is the key, it helps switching from one project to another. It especially matters when using both React and SF in a project (more and more common).

And, it's a good idea!

I really don't want to introduce something Symfony-specific here!

Why is .env.test.local mentioned in the following sentence, then ? 😄

.env.dev.local, .env.test.local, .env.prod.local... - Local overrides of environment-specific settings.

@ogizanagi the comment was wrong 😁. Fixed!

ogizanagi · 2018-09-21T11:23:28Z

src/Symfony/Component/Dotenv/Dotenv.php

+    /**
+     * Loads one or several .env and the corresponding env.local, env.$env and env.$env.local files if they exist.
+     *
+     * env.local is always ignored in test env because tests should produce the same results for everyone.


- env.local + .env.local // or + $path.local

? 😄

src/Symfony/Component/Dotenv/Dotenv.php

weaverryan · 2018-09-21T15:07:25Z

src/Symfony/Component/Dotenv/Dotenv.php

@@ -50,6 +51,33 @@ public function load(string $path, string ...$paths): void
        $this->doLoad(false, $path, $paths);
    }

+    /**
+     * Loads one or several .env and the corresponding env.local, env.$env and env.$env.local files if they exist.


Environment variables in later files override earlier files, correct?

Almost, see PR description.

Wirone · 2018-09-25T17:55:34Z

I don't get what's the purpose of having .env.prod while at the same time discuraging or even blocking DotEnv usage on production 🤔 Also, development instances shouldn't use production configuration. .env.prod should not be commited, nor included in built Docker image. So what is the use case for it?

I don't like the idea of this PR. It makes simple task (app configuration) much more complicated and IMO doesn't bring added value (the only one I see is configuring variables for different environments and use it only by changing APP_ENV, without need to edit .env by commenting/uncommenting lines per environment). I thought that YAML files' split (packages in prod/dev subdirectories) and .env file were supposed to fulfill all the config scenarios. This is a step back for me.

But maybe I don't understand something..

fbourigault · 2018-09-25T18:26:39Z

The first use case I have is to have on a local machine the dev and test environments running at the "same" time. I mainly have a different DATABASE_URL env variable which is probably different per developer and I don't want to use DATABSE_URL=... vendor/bin/simple-phpunit!

dunglas · 2018-09-25T19:21:21Z

@Wirone .env.prod is totally optional (for people not able to set environment variables in prod). This file will not be generated by the default skeleton (only .env.test will).
However it will make easier to use Dotenv in prod in cases were there is no alternatives (shared servers usually, and even for jobs in dedicated servers without Docker/K8S).

However, having .env.test, .env.panther, .env.otherenv on the development machine will be very useful (and it’s my main goal, as I use Kubernetes in prod, without env files obviously).

Wirone · 2018-09-26T06:30:09Z

src/Symfony/Component/Dotenv/Dotenv.php

+    /**
+     * @param bool $envMode Silently ignore not existing files in $paths (but not the one in $path)
+     */
+    private function doLoad(bool $overrideExistingVars, string $path, array $paths, bool $envMode = false): void


$envMode variable name does not correspond with what it does. Should be named $silent (as it's used internally) or $ignorePathErrors or something like that.

PS. phpDoc is missing the other arguments

$silent or $ignorePathErrors is less accurate: an error will be thrown of the main file (usually .env) doesn’t exist. Only files for specific environments are optional.

Regarding the PHPDoc, in Symfony we don’t add the useless lines.

IMO $envMode is not meaningful. If you don't look at implementation, only on signature, would you know what $envMode does? Honestly, it doesn't even look like bool variable, rather string or bitwise flag.

@Wirone I agree, it's why I added a PHPDoc description :) If you have better name in mind, I'll be very happy to switch ($silent and $ignorePathErrors were my first thoughts, but they are more confusing than $envMode).

What about $safe (determines if call is safe and will not throw an exception) or $required (determines if all paths should exist - it would require changing to $silent = !$required).

IMO they are not more explicit, and can be confusing. environment mode is exactly what it is, and $ignoreFilesThatNotExistExceptTheFirstOne is a bit long. Isn't the description in the PHPDoc enough? This method is private, it will never be used by the end user, only by Symfony contributors (we may expect that they read the PHPDoc :D).

doLoad(bool $overrideExistingVars, string $path, bool $ignoreMissingExtraPaths, string ...$extraPaths = array())?

Not sure that using the splat operator is a good idea in this private method (we may add extra parameters at some point), also it is already used without the splat operator in other functions.

parameter renamed

chalasr

Finally! This will allow me to drop an extra class from all my enterprise projects.
👍 Thanks for raising

teohhanhui · 2018-10-04T10:10:59Z

I think what's missing from this picture is where .dist files fit in. We should not follow blindly the Create React App / Rails way, because it conflicts with our conventions.

dunglas · 2018-10-04T22:49:09Z

@teohhanhui the whole point of this PR is to switch from “our” conventions to the ones other projects follow.
Basically, .env.dist is now .env, and .env is now .env.local.

teohhanhui · 2018-10-05T04:02:25Z

That sounds like a really bad thing to me. We shouldn't just switch conventions like that. What's wrong with sticking with .dist? It makes it more explicit that it's a distribution file. Plus, this would not go well with Docker Compose (.env is special for Docker Compose).

What I'm trying to say is, we could stick with our conventions while having all the benefits of an extended .env support like in the other projects.

dunglas · 2018-10-05T04:08:26Z

This has already been discussed a lot in symfony/recipes#466 and related tickets. It actually makes it easier to deal with Docker Compose.

fabpot · 2018-10-26T06:40:31Z

Thank you @dunglas.

…s dotenv behavior (dunglas) This PR was merged into the 4.2-dev branch. Discussion ---------- [DotEnv] Add a new loadForEnv() method mimicking Ruby's dotenv behavior | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | yes | BC breaks? | no  | Deprecations? |no  | Tests pass? | yes  | Fixed tickets | helps for symfony/recipes#465, symfony/recipes#408 | License | MIT | Doc PR | todo This PR adds a new `loadForEnv()` method that mimics the behavior of [Create React App](https://github.com/facebook/create-react-app/blob/master/packages/react-scripts/template/README.md#what-other-env-files-can-be-used), [Rails' DotEnv](https://github.com/bkeepers/dotenv#what-other-env-files-can-i-use) and probably some other libs: `DotEnv::loadForEnv()` will load the following files, starting from the bottom. The first value set (or those already defined in the environment) take precedence: - `.env` - The Original® - `.env.dev`, `.env.test`, `.env.prod`... - Environment-specific settings. - `.env.local` - Local overrides. This file is loaded for all environments _except_ `test`. - `.env.dev.local`, `.env.test.local`, `.env.prod.local`... - Local overrides of environment-specific settings. The plan is to use this method in the default SF installation (symfony/recipes#466). Commits ------- 774a78c [DotEnv] Add a new loadForEnv() method mimicking Ruby's dotenv behavior

Tobion · 2018-10-29T01:16:42Z

src/Symfony/Component/Dotenv/Dotenv.php

+    }
+
+    /**
+     * Loads one or several .env and the corresponding env.$env, env.local and env.$env.local files if they exist.


this seems to be missing the dots in front of the files like .env.$env

This PR was merged into the 4.2-dev branch. Discussion ---------- [DotEnv] Fix loadForEnv PHPDoc | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | no  | BC breaks? | no  | Deprecations? | no  | Tests pass? | yes  | Fixed tickets | #28533 (review) | License | MIT | Doc PR | n/a  Commits ------- c10710c [DotEnv] Fix loadForEnv PHPDoc

ro0NL · 2018-10-30T12:27:23Z

did we really need this?

Why .env.prod over config/packages/prod? if you cant define real env vars those locations are effectively the same already... and you can already import a git ignored local parameter file.

the behavior of Create React App, Rails' DotEnv and probably some other libs:

so yeah.. curious if the config/packages/ convention didnt already provide such behavior in SF.

jaikdean · 2018-10-30T14:14:17Z

Is there a well-defined use case for this to explain why it's useful? The only ones I can think up are already solved by using parameters and environment variables as they are in Symfony 4.1, but I could well be missing something.

In my projects we have phpunit.xml.dist and .php_cs.dist files committed and their non-dist versions ignored, so the convention adopted from Rails etc just brings another convention into play, rather than replacing one. It's understandable if a lot of others are already used to this convention, but it's the first time I've seen it.

Just my 2 pence!

weaverryan · 2018-10-30T14:41:07Z

This feature has a lot of options in it, but the original motivation was to help solve exactly one problem. Currently, you need to add all of your environment variables in phpunit.xml.dist. So, your environment variables are completely duplicated, even if you only want to change a few environment variables. This + this WIP recipe (symfony/recipes#466) will help make that happen (as you can just have a .env.dist). I think that, mostly, this change will not mean much to people - other than you don't need to duplicate your env vars in the test environment anymore and can instead create a .env.test with only the overrides.

Cheers!

ro0NL · 2018-10-30T14:47:42Z

still.. there's zero difference with doing

# config/packages/test/some.yaml
env(SOME): "default in test"

no?

nicolas-grekas · 2018-10-30T14:51:33Z

The value can be changed without invalidating the container, that's the diff :)

weaverryan · 2018-10-30T14:53:45Z

That's a VERY fair point @ro0NL :). But I think setting a default in that way is a bit less obvious than having a .env.test that's created for you when installing the simple-phpunit recipe.

nicolas-grekas · 2018-10-30T15:05:41Z

@weaverryan right!, that's the 2nd difference: flex automation!

ro0NL · 2018-10-30T15:06:47Z

understood. im just skeptical if the overhaul is worth it eventually, it's quite significant.

ryanotella · 2018-10-31T00:04:18Z

@weaverryan Thanks for this! We were starting to think we'd gone mad and had completely missed something.

…nv() (nicolas-grekas) This PR was merged into the 4.2-dev branch. Discussion ---------- [Dotenv] add loadEnv(), a smoother alternative to loadForEnv() | Q | A | ------------- | --- | Branch? | 4.2 | Bug fix? | yes | New feature? | no | BC breaks? | yes (4.2-only) | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - This PR replaces the `loadForEnv()` method introduced in #28533 by a new `loadEnv()` method. - It accepts only one mandatory argument: `$path`, which is the path to the `.env` file. - The 2nd argument is optional and defines the name of the environment variable that defines the Symfony env. This plays better with the current practice of defining the env in `.env` (`loadForEnv()` requires knowing the env before being called, leading to a chicken-n-egg situation that `loadEnv()` avoids.) - the possibility to load several files at once is removed. We don't have a use case for it and those who do can call `loadEnv()` in a loop anyway. In addition to $path (.env), the following files are loaded, the latter taking precedence in this order: .env < env.local < .env.$env < .env.$env.local Note that `loadForEnv()` used to give higher precedence to .env.local vs .env.$env. The new behavior is aligned with [the order used by create-react-app](https://github.com/facebook/create-react-app/blob/master/docusaurus/docs/adding-custom-environment-variables.md#what-other-env-files-can-be-used). It also allows overriding the env in .env.local, which should be convenient for DX. Last but not least, the "test" env has this special behaviors: - `.env.local` file is skipped for the "test" env (same as before and as in create-react-app) - ~vars defined in .env files **override** real env vars (similar to what Rails' dotenv does: you don't want your tests to randomly fail because of some real env vars)~. Commits ------- 0cf9acb [Dotenv] add loadEnv(), a smoother alternative to loadForEnv()

bouland · 2019-03-21T09:43:22Z

Hi,

This "Ruby's dotenv behavior" is BC for all tools using ".env traditional behaviour" which is to store a lot of command lines arguments value and store secrets.

My main difficulty is with this last point. Cf. What Changed Exactly?

B) The .env file is now commited to your repository.

Check this illustration with docker-compose to build the following Dockerfile :

FROM registry.access.redhat.com/rhel7/rhel
ARG PASSWORD

With the following tips Cf. docker compose args

You can omit the value when specifying a build argument, in which case its value at build time is the value in the environment where Compose is running.

services:
    web:
        build:
            context: docker/web
            args:
                - PASSWORD

And command line ".env traditional behaviour"explain Cf. docker compose env

Compose supports declaring default environment variables in an environment file named .env placed in the folder where the docker-compose command is executed (current working directory).

Will propose docker-compose to adopt "Ruby's dotenv behavior"

extra thoughts

a dot file is hidden on Unix FS
.env is a conventional name shared across multiple tools
best practice is to use command / service (.composer / .nvm / .java )
why not .symfony ?
I used to create local environment on my desktop and keep dev for integration on shared dev server, so weird result .env.local.local

dunglas requested a review from ogizanagi September 21, 2018 06:56

carsonbot added Status: Needs Review Feature labels Sep 21, 2018

dunglas changed the title ~~[DotEnv] Add a new loadForEnv() method mimicing Ruby's dotenv behavior~~ [DotEnv] Add a new loadForEnv() method mimicking Ruby's dotenv behavior Sep 21, 2018

dunglas force-pushed the loadForEnv branch 2 times, most recently from 1000890 to e0fecdd Compare September 21, 2018 06:57

ogizanagi reviewed Sep 21, 2018

View reviewed changes

dunglas mentioned this pull request Sep 21, 2018

Allow to override .env files for specific environments symfony/recipes#466

Closed

dunglas force-pushed the loadForEnv branch 2 times, most recently from 6173590 to d3b49fd Compare September 21, 2018 10:18

dunglas mentioned this pull request Sep 21, 2018

Change the EnvConfigurator to only handle the .env file symfony/flex#423

Closed

nicolas-grekas added this to the next milestone Sep 21, 2018

ogizanagi reviewed Sep 21, 2018

View reviewed changes

dunglas force-pushed the loadForEnv branch from d3b49fd to fd006a6 Compare September 21, 2018 12:28

maxhelias approved these changes Sep 21, 2018

View reviewed changes

carsonbot added Status: Reviewed and removed Status: Needs Review labels Sep 21, 2018

fbourigault reviewed Sep 21, 2018

View reviewed changes

src/Symfony/Component/Dotenv/Dotenv.php Show resolved Hide resolved

yceruto reviewed Sep 21, 2018

View reviewed changes

src/Symfony/Component/Dotenv/Dotenv.php Show resolved Hide resolved

weaverryan reviewed Sep 21, 2018

View reviewed changes

ogizanagi approved these changes Sep 22, 2018

View reviewed changes

Wirone reviewed Sep 26, 2018

View reviewed changes

chalasr approved these changes Sep 26, 2018

View reviewed changes

[DotEnv] Add a new loadForEnv() method mimicking Ruby's dotenv behavior

774a78c

nicolas-grekas force-pushed the loadForEnv branch from 95d821e to 774a78c Compare October 24, 2018 09:39

fabpot approved these changes Oct 26, 2018

View reviewed changes

fabpot merged commit 774a78c into symfony:master Oct 26, 2018

dunglas mentioned this pull request Oct 26, 2018

File api/.env should not be versionned api-platform/demo#65

Closed

Tobion reviewed Oct 29, 2018

View reviewed changes

dunglas mentioned this pull request Oct 29, 2018

[DotEnv] Fix loadForEnv PHPDoc #29012

Merged

dunglas deleted the loadForEnv branch October 29, 2018 07:13

javiereguiluz mentioned this pull request Oct 30, 2018

[DotEnv] Add a new loadForEnv() method symfony/symfony-docs#10607

Closed

This was referenced Nov 3, 2018

Release v4.2.0-BETA1 #29072

Closed

Release v4.2.0-BETA1 #29073

Merged

nicolas-grekas mentioned this pull request Nov 7, 2018

[Dotenv] add loadEnv(), a smoother alternative to loadForEnv() #29129

Merged

helhum mentioned this pull request Nov 22, 2019

[FEATURE] Use loadEnv instead of load to enable usage of .env.local f… helhum/dotenv-connector#15

Closed

teohhanhui mentioned this pull request Jan 2, 2020

[RFC][DX] Defaulting missing env variables to empty string #31497

Closed

kcassam mentioned this pull request Jun 19, 2020

Use the symfony mechanism for managing env files laravel/ideas#2269

Closed

helloitsjoe mentioned this pull request Jun 2, 2022

.env.local should have higher priority than .env.<environment> entropitor/dotenv-cli#73

Closed

[DotEnv] Add a new loadForEnv() method mimicking Ruby's dotenv behavior #28533

[DotEnv] Add a new loadForEnv() method mimicking Ruby's dotenv behavior #28533

Conversation

dunglas commented Sep 21, 2018 • edited Loading

Choose a reason for hiding this comment

dunglas Sep 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wirone commented Sep 25, 2018 • edited Loading

fbourigault commented Sep 25, 2018

dunglas commented Sep 25, 2018

Choose a reason for hiding this comment

dunglas Sep 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dunglas Sep 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chalasr left a comment

Choose a reason for hiding this comment

teohhanhui commented Oct 4, 2018

dunglas commented Oct 4, 2018

teohhanhui commented Oct 5, 2018

dunglas commented Oct 5, 2018

fabpot commented Oct 26, 2018

Choose a reason for hiding this comment

ro0NL commented Oct 30, 2018 • edited Loading

jaikdean commented Oct 30, 2018

weaverryan commented Oct 30, 2018

ro0NL commented Oct 30, 2018

nicolas-grekas commented Oct 30, 2018 • edited Loading

weaverryan commented Oct 30, 2018

nicolas-grekas commented Oct 30, 2018

ro0NL commented Oct 30, 2018

ryanotella commented Oct 31, 2018

bouland commented Mar 21, 2019 • edited Loading

dunglas commented Sep 21, 2018 •

edited

Loading

dunglas Sep 21, 2018 •

edited

Loading

Wirone commented Sep 25, 2018 •

edited

Loading

dunglas Sep 26, 2018 •

edited

Loading

dunglas Sep 26, 2018 •

edited

Loading

ro0NL commented Oct 30, 2018 •

edited

Loading

nicolas-grekas commented Oct 30, 2018 •

edited

Loading

bouland commented Mar 21, 2019 •

edited

Loading