Love it! Ship it!

@stof can you take a look at this too please?

In general, was the issue not that this approach would be problematic with bundles that used the config before the dumping or something? I seem to remember a discussion in this direction. It would be possible to have dynamic variable lookups before dumping, or from the loaded dumped cache, but not both?

This is an interesting approach, but it is not as backwards compatible as one would think. There are bundles out there today, which as a part of their extension actually manipulate parameters. SncRedisBundle e.g. splits DSNs in the extension. Passing one of these parameters there will at best cause some kind of syntax error in the bundle, or at worst cause old environment values to be cached.

I don't think bc is the right concept for what you're saying, at least there is no bc break here, unless one already has a parameter named env(...). So unlikely.

But you're right that this strategy is not compatible with any parameter transformations done by DI extensions.
Yet, this is not a blocker at all to me: if you opt into using an env param at a place that is incompatible with one of your extensions, you'll get a failure from that extension (or it is badly designed), which means an opportunity to fix it. Such extensions are already incompatible with unresolved parameters btw, so there isn't much new here. And I guess you can even workaround the issue most of the time by using several parameters concatenated by the separators that the extension expects.
The huge benefit of this approach is that it doesn't need any cooperation from DI extensions. It just needs a way to get through them. Which most allow.

The problem isn't so much for bundle authors, as you say, they can just fix it, but for users. I'm just worried this will give users weird and possibly subtile errors.

That's not to say I'm necessarily -1 though, this sort of approach (where you only have to send PRs to the few bundles that manipulate parameters) will get the entire ecosystem environment variable compatible a lot sooner.

Thank you for this ! Much needed

From @iamluc:

Is it possible to define "providers" instead of calling getenv directly ? This way, we could use key/value store (like consul, etcd, ...) or even files

The getEnv() method is protected. I think that's enough for you to build consul, etcd... integration by just changing the base class of your container, and specialize the implementation of getEnv() in your own base class. Not sure we need more nor want in Symfony core.

@iamluc I would avoid calling consul during the container's boot. Instead I would use consul-template.

Would it be a good idea to do one of the following:

A: Add a canHandleDynamicParameters(bool $canHandle) method to Symfony\Component\Config\Definition\BaseNode, which we default to false, and deprecate Symfony 3.4, so that we can require all config nodes to support dynamic parameters in Symfony 4.

B: Add a cannotHandleDynamicParameters() to Symfony\Component\Config\Definition\BaseNode, so that Config nodes that are processed in the extension can "opt out" of dynamic parameters, raising an early exception if the user tries to do so. This method could then also be deprecated for 3.4, requiring all Symfony 4 bundles to support dynamic parameters.

Talking with @lyrixx , I reverted the split of the getEnv() method. Yet my comment still holds true:

The getEnv() method is protected. I think that's enough for you to build consul, etcd... integration by just changing the base class of your container, and specialize the implementation of getEnv() in your own base class. Not sure we need nor want more in Symfony core.

@magnusnordlander I'd say no need, bundle authors don't necessarily need to know about that, and shouldn't be allowed to block what a user wants to do if it works for them IMHO. But let's see what others think of your proposal.

I see no reason to make dynamic parameters configurable like Magnus suggests. It shouldn't matter parameters come from the environment, and it should be totally up to the end user.

I'd only worry if the exception user gets is descriptive enough to see what's wrong.

@nicolas-grekas: Bundle authors do need to know, because it means that they basically cannot write extensions or compiler passes that inspects the values of configuration (or, basically anything in the DIC).

I don't imagine that bundle authors will block the user out of spite, but rather if the implementation of the extension or a compiler pass doesn't work with dynamic parameters. This isn't a hypothetical scenario, there are bundles out there today that do not work with this, bundles that by Symfony 3.1 standards do nothing wrong, and I think it's better that the user finds out due to an exception than subtle bugs.

if the implementation of the extension or a compiler pass doesn't work with dynamic parameters

Tell if I'm wrong, but such extensions are already kind of broken, by not allowing unresolved parameters, isn't it? If yes, then nothing new here to me.

@nicolas-grekas: To my knowledge, using unresolved parameters has never been possible? Parameters are resolved in MergeExtensionConfigurationPass, before Extension::load is called. and any non-existent parameters will cause a Symfony\Component\DependencyInjection\Exception\ParameterNotFoundException to be thrown.

It's slightly unusual (probably less than 10% of extensions, but not an insignificant number) to do a lot of processing of config values in the extension, but by no means broken according to Symfony 3.1 standards.

Do we actually need the env placeholder system ?

If the DI Configuration class accepts any string, the %env(...)% one is fine.

If it performs validation on the value (boolean field, number field, enum field, string with specific format...), the placeholder will also fail.

In fact, resolving %...% parameters happens both in ParameterBag and in PhpDumper. Leaving %env(...)% asis would mean making an exception to parameter resolving (it's recursive today). Or do you have something else in mind @stof ?

@mmucklo: I use https://github.com/vlucas/phpdotenv, and then add the following method to my AppKernel:

    public function boot()
    {
        if (file_exists(__DIR__ . '/../.env')) {
            $dotenv = new \Dotenv\Dotenv(__DIR__ . '/..');
            $dotenv->load();
        }

        return parent::boot();
    }

Works like a charm.

@magnusnordlander Does this allow you to add parameters into container via dotenv?

What I've done is made .env tracked just like parameters.yml (so that a change to a .env file will cause a re-compile of the cache) however I had to extend buildContainer and use a custom version of EnvParametersResource to do it.

I'm wondering if with this change there's perhaps now a better way? (i.e. to track .env and to also make any parameters in .env, such as SYMFONY__SOMETHING__OR__OTHER to appear as parameters in the container, such as something.or.other)...

@mmucklo: No, it allows me to add environment variables via dotenv. Those environment variables can then (now that this PR is merged) be referenced in your config using the %env(DATABASE_DSN)% syntax. No container re-compilation is necessary.

Can/will this change be back-ported to Symfony2?

@denormalizer No, new features are always only added to the current development version (i.e. the master branch).

@denormalizer,

Can/will this change be back-ported to Symfony2?

ecentria/dynamic-parameters-bundle provides %env()% parameters support for Symfony2 apps.

Implementation is forward compatible with Symfony 3.2 %env()% parameters, so make sure to upgrade to Symfony 3.2 when it's stable.

Hmm.. I might be a bit late to join this discussion, but since I am running into this issue............
I would like to clarify this (and my) problem, because I might not fully understand what is going on here.

Just for the sake of clearity (and to save myself headaches) we have a number of env variables in the config.yml and for those that are not critical in our application we have defined default values. I appreciate errors to point out problems, but in our case this is what we want. We have a set of defaults, and when needed we can overwrite the the value. And yes, some variables might not be loaded in every environment. But in this particular application and these specific variables we decided that this is OK and this is what we prefer. Partially to keep the config straight forward, prevent deployment issues over insignificant settings and to help new developers get up to speed quickly and that sorts of things.

But as I understand, Symfony is now creating the bug for us to point out that there might be a bug? I feel that this should be optional. There are cases in which you don't want total protection. Other factors might outweigh the gain. And what about refactoring? I might not want to remove parameters from which I know I will need them very soon. I should be able to keep them without breaking the application. Also I should be able to work on a ticket to create or refactor configurations and I should be able to release that (assuming its valid of course) and continue to work on the implementation later if that suits our schedule better. Not every application is the same. I don't think this is a good change.

Please correct me if I got the wrong idea.

Kind regards,
Daan