[DI] Validate env vars in config #23888

ro0NL · 2017-08-14T16:36:32Z

Q	A
Branch?	4.1/master
Bug fix?	no
New feature?	yes
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets	#22151, #25868
License	MIT
Doc PR	symfony/symfony-docs#8382

This PR registers the env placeholders in Config\BaseNode with its default value or an empty string. It doesnt request real env vars during compilation,

What it does is if a config value exactly matches a env placeholder, we validate/normalize the default value/empty string but we keep returning the env placeholder as usual. If a placeholder occurs in the middle of a string it also proceeds as usual.

The latter to me is OK as you need to expect any string value during runtime anyway, including the empty string.

nicolas-grekas · 2017-08-16T15:31:46Z

I suggest wait for #23901 before continuing on this topic - that will provide a much better base to fix this issue.

ro0NL · 2017-09-07T17:33:43Z

@nicolas-grekas could you briefly describe the path you have in mind for this, and how it leverages getProvidedTypes?

From a technical pov this approach was the only thing i could come up with, regarding decoupled state between di and config. I like how it works out-of-the-box for everyone :)

My idea here was to generate a better default value using getEnv() actually, probably calling it before being compiled (as that's allowed anyway ^^)

nicolas-grekas · 2017-09-08T08:33:49Z

The env placeholders contain the prefixes - so we can extract that, and use EnvPlaceholderParameterBag::getProvidedTypes() to get the type of each placeholder.
That should be enough, isn't it?

ro0NL · 2017-09-09T11:07:29Z

@nicolas-grekas first round, wdyt?

ro0NL · 2017-09-09T11:16:29Z

src/Symfony/Component/Config/Definition/BaseNode.php

+     */
+    public static function setPlaceholder($placeholder, $type)
+    {
+        switch ($type) {


IMO. can be done at the calling side..

ro0NL · 2017-09-09T12:09:21Z

src/Symfony/Component/DependencyInjection/Compiler/PassConfig.php

@@ -92,7 +91,7 @@ public function __construct()
    public function getPasses()
    {
        return array_merge(
-            array($this->mergePass),
+            array(new RegisterEnvVarProcessorsPass(), $this->mergePass),


now runs even earlier.. anything to do? Patched PhpDumper test for now.

Wait, I just realized that this may just make it impossible to register a new env processor in practice, isn't it?

ro0NL · 2017-09-09T12:13:53Z

src/Symfony/Component/DependencyInjection/Compiler/MergeExtensionConfigurationPass.php

+                        continue;
+                    }
+                    foreach ($placeholders as $placeholder) {
+                        BaseNode::setPlaceholder($placeholder, reset($envTypes[$prefix]));


What about getParam("env($env)") as default value instead, if available :) That justifies computing default values on the calling side.

Let's do. We also need to validate the type of the default value before calling setPlaceholder (its type needs to be in $envTypes).

BUT, calling "reset" here prevents taking advantage of the "multiple types provided" info. So maybe pass an array instead, $type=>$value?

nicolas-grekas · 2017-09-09T14:19:16Z

src/Symfony/Component/DependencyInjection/Compiler/RegisterEnvVarProcessorsPass.php

@@ -43,14 +44,23 @@ public function process(ContainerBuilder $container)
                foreach ($class::getProvidedTypes() as $prefix => $type) {
                    $processors[$prefix] = new ServiceClosureArgument(new Reference($id));
                    $types[$prefix] = self::validateProvidedTypes($type, $class);
+
+                    if (!$types[$prefix]) {


can this happen at all? explode always creates an array with at least one element, isn't it?

can this happen at all?

nope :)

ro0NL · 2017-09-09T17:04:47Z

@nicolas-grekas pushed latest changes real quick. Think this is heading towards what you have in mind; see BaseNode.

nicolas-grekas · 2017-09-09T17:26:44Z

src/Symfony/Component/Config/Definition/BaseNode.php

+            foreach (self::$placeholders[$value] as $placeholder) {
+                try {
+                    $this->normalize($placeholder);
+                    break;


should be return $value;
and the current return below should be throw $e;

now with test :)

ro0NL · 2017-09-10T08:44:11Z

Tests looking good. Tried to keep the big code blob at a minimum, but tweaks are welcome :)

Status: needs review

deps=high failure expected.

nicolas-grekas · 2017-09-10T10:16:14Z

src/Symfony/Component/Config/Definition/BaseNode.php

+            foreach (self::$placeholders[$leftSide] as $placeholder) {
+                try {
+                    $this->validateType($placeholder);
+                    break;


return + throw here also (and below)

or at least, the throw part is missing if return is not for here

ro0NL · 2017-09-10T11:01:25Z

@nicolas-grekas now with test for validating on merge :) see simplified BaseNode::resolvePlaceholderValues() + merge().

nicolas-grekas · 2017-09-10T16:08:15Z

src/Symfony/Component/DependencyInjection/ParameterBag/EnvPlaceholderParameterBag.php

@@ -79,6 +79,13 @@ public function mergeEnvPlaceholders(self $bag)
                $this->envPlaceholders[$env] += $placeholders;
            }
        }
+        foreach ($bag->getProvidedTypes() as $prefix => $types) {


I think this file doesn't need to be modified: merging happens after setProvidedTypes is called, and should not happen anymore after

They get lost in MergeExtensionConfigurationParameterBag::__construct as we overwrite $resolvingBag.

But agree; this method is called mergeEnvPlaceholders.. so doesnt imply it merges types. Now fixed.

ro0NL · 2017-09-10T18:22:44Z

@nicolas-grekas what about this;

node: 'value' > "process" > node: 'normalized-prefix value'

But then with env placeholders where "env_NAME_x" becomes "normalized-prefix env_NAME_x".

I think current behavior is OK as envs are normalized values (nothing happens with them after compile, i.e. no config normalization, except processing). Thus you'd have NAME="normalized-prefix value".

WDYT?

edit: hm that only applies if a type conversion happens, and we lose the placeholder. For string-to-string we could in fact return the processed value if it still contains the original placeholder (and perhaps if not also). Not sure who's exactly responsible here :)

nicolas-grekas · 2017-09-11T09:07:15Z

src/Symfony/Component/DependencyInjection/Compiler/MergeExtensionConfigurationPass.php

+                    $default = null;
+                    if ($hasEnv = (false === $i && $defaultEnvBag->has("env($env)"))) {
+                        switch ($type = gettype($default = $defaultEnvBag->get("env($env)"))) {
+                            case 'boolean': $type = 'bool'; break;


should we rename our allowedTypes to match those of gettype and remove that switch?

Think so 👍 preferring FQ over short names.

While at it should we infer boolean|integer|float|string from scalar while resolving? Nice to have perhaps.

nicolas-grekas · 2017-09-11T09:07:36Z

src/Symfony/Component/DependencyInjection/Compiler/MergeExtensionConfigurationPass.php

+                            case 'float': $values[$type] = 0.0; break;
+                            case 'int': $values[$type] = 0; break;
+                            case 'string': $values[$type] = ''; break;
+                            default: $values[$type] = null; break;


is this case possible? if not, default should be string, isn't it?

only if we forget to update it after adding types. Should be never, just a safety net.

default: throw then? In case of non-fatal behavior id say null.. why assume string?

no default case and let php crash? fine fore me :)

cdaguerre · 2018-03-19T10:08:54Z

Does the milestone change mean it won't make it into the 3.4 branch?

nicolas-grekas

(with minor comments)

nicolas-grekas · 2018-03-21T13:34:44Z

src/Symfony/Component/Config/Definition/BaseNode.php

+                return self::$placeholders[$value];
+            }
+
+            if (0 === strpos($value, self::$placeholderUniquePrefix, 0)) {


3rd arg not needed

nicolas-grekas · 2018-03-21T13:35:05Z

src/Symfony/Component/Config/Definition/BaseNode.php

+
+    private static function getType($value): string
+    {
+        switch ($type = gettype($value)) {


nicolas-grekas · 2018-03-21T13:37:34Z

src/Symfony/Component/Config/Definition/Processor.php

@@ -15,6 +15,8 @@
 * This class is the entry point for config normalization/merging/finalization.
 *
 * @author Johannes M. Schmitt <[email protected]>
+ *
+ * @final since version 4.1


unrelated to this PR? If yes, should be removed.

not sure :) https://github.com/symfony/symfony/pull/23888/files#diff-e321d9e71b12d99326a74385f020e2eeR66

see also #23888 (comment)

i think it's reasonable

then: missing changelog/upgrade entries

changelog entry added. Suggestion for an upgrade note?

yes!
in *4.1: The ... class has been made final
in *5.0: The ... class has been made final

:)

nicolas-grekas · 2018-03-21T13:49:10Z

src/Symfony/Component/DependencyInjection/ParameterBag/EnvPlaceholderParameterBag.php

+     */
+    public function getEnvPlaceholderUniquePrefix(): string
+    {
+        return $this->envPlaceholderUniquePrefix ?? $this->envPlaceholderUniquePrefix = 'env_'.bin2hex(random_bytes(16));


8 instead of 16 is enough (we use 7 bytes in ContainerBuilder::hash())

fabpot · 2018-03-22T07:30:04Z

src/Symfony/Component/Config/Definition/BaseNode.php

@@ -50,6 +56,39 @@ public function __construct(?string $name, NodeInterface $parent = null, string
        $this->pathSeparator = $pathSeparator;
    }

+    /**
+     * Register possible (dummy) values for a dynamic placeholder value. Matching configuration values will be processed


I would move all sentences but the first one as a phpdoc description.

You mean split into short & long description right? Did that for now.

fabpot · 2018-03-22T07:31:31Z

src/Symfony/Component/Config/Definition/BaseNode.php

+    }
+
+    /**
+     * Set a common prefix for dynamic placeholder values. Matching configuration values will be skipped from being


Sets + same comment as above about separating the first sentence from the other ones.

fabpot · 2018-03-22T07:31:52Z

src/Symfony/Component/Config/Definition/BaseNode.php

+    }
+
+    /**
+     * Reset all current placeholders available.


fabpot · 2018-03-22T07:33:23Z

src/Symfony/Component/Config/Definition/BaseNode.php

+    public static function resetPlaceholders(): void
+    {
+        self::$placeholderUniquePrefix = null;
+        self::$placeholders = array();


Having static properties like this is very dangerous as we need to make sure that they are reset properly. I can see the finally calls here and there to reset this, but is there is any other way? Avoiding static properties would be great.

It's not that dangerous: placeholders are unique strings so there is not really any way to mess up with them, even if someone forgets to reset them.
About removing the static state, that's really hard...
BUT, @ro0NL we should make these methods @internal for sure.

I think we can settle indeed, i dont see a elegant workaround to avoid static currently, that is without overhauling the config component itself :)

Now marked internal.

also we deal with it properly where needed; it's not scattered all over the environment.

fabpot · 2018-03-22T07:34:03Z

src/Symfony/Component/Config/Definition/BaseNode.php

@@ -363,4 +459,85 @@ public function getParent()
     * @return mixed The finalized value
     */
    abstract protected function finalizeValue($value);
+
+    /**
+     * Test if placeholder values are allowed for this node.


fabpot · 2018-03-22T07:35:35Z

src/Symfony/Component/Config/Definition/BaseNode.php

+    }
+
+    /**
+     * Get allowed dynamic types for this node.


nicolas-grekas · 2018-03-23T09:56:02Z

src/Symfony/Component/Config/Definition/BaseNode.php

+    /**
+     * Resets all current placeholders available.
+     */
+    public static function resetPlaceholders(): void


this one also needs to be internal

oops :) updated.

nicolas-grekas · 2018-03-27T06:02:51Z

Thank you @ro0NL.

mcfedr · 2018-03-27T14:00:23Z

src/Symfony/Component/DependencyInjection/Tests/Compiler/ValidateEnvPlaceholdersPassTest.php

+        $container = new ContainerBuilder();
+        $container->registerExtension(new EnvExtension());
+        $container->prependExtensionConfig('env_extension', array(
+            'simple_array_node' => '%env(json:FOO)%',


json: can return a simple_array, e.g. in the case the env is something like [1, 2] - of course json doesnt guarrentee that its like this, it could also be {"a": 1, "b": 2}, so its a bit of a question, is it valid or not - I think its more useful if it passes validation.

yeah i think we need to reassure / revise integration a bit. Or put different im curious if the change to json prefix affects the possibilities here.

the json change only added the possibility to accept null in addition to array so shouldn't change anything here - this was on purpose so that its still possible to do some validation based on it always being an array.

I wonder if it would make sense to add a simple_array: decoder to further help with validation

mcfedr · 2018-03-27T14:06:06Z

src/Symfony/Component/Config/Definition/ArrayNode.php

+     */
+    protected function allowPlaceholders(): bool
+    {
+        return false;


Why have you made it false by default for ArrayNode? It seems this will limit how much of a config you can change in other peoples bundles, i.e. you can only put env(json:FOO) if the owner things of you in advance.

comments on closed PRs are likely to be lost, please consider opening a new issue/PR if you think something is buggy or can be improved. Thanks

In this case i asked for it on another closed PR :P But yeah.. not sure if there's any real issue now (hence i asked :D)

@mcfedr see symfony/symfony-docs#8382 (comment) , tried to explain the general proces there.

I see, makes sense, so for simple_array nodes it will accept env(json:FOO) but not when it should validate the structure.

Majkl578 · 2018-05-31T17:02:48Z

FYI: Just got hit by this BC break when upgrading to 4.1.0:

In ValidateEnvPlaceholdersPass.php line 54:
                                                                                                              
  [Symfony\Component\DependencyInjection\Exception\LogicException]                                            
  Invalid type for env parameter "env(ABC)". Expected "string", but got "float".

for configuration with env defaults:

parameters:
    env(ABC): 1.2

Issue & failing test coming soon.

carsonbot added Status: Needs Review DependencyInjection Feature labels Aug 14, 2017

nicolas-grekas added this to the 3.4 milestone Aug 15, 2017

ro0NL mentioned this pull request Aug 18, 2017

[DI] Allow processing env vars #23901

Merged

nicolas-grekas added Status: Needs Work and removed Status: Needs Review labels Aug 22, 2017

ro0NL mentioned this pull request Aug 24, 2017

Environment Variables lack type - break built-in Symfony configuration #22151

Closed

ro0NL commented Sep 9, 2017

View reviewed changes

nicolas-grekas reviewed Sep 9, 2017

View reviewed changes

ro0NL changed the title ~~[DI] Validate env vars in config~~ [WIP][DI] Validate env vars in config Sep 9, 2017

nicolas-grekas reviewed Sep 9, 2017

View reviewed changes

ro0NL changed the title ~~[WIP][DI] Validate env vars in config~~ [DI] Validate env vars in config Sep 10, 2017

carsonbot added Status: Needs Review and removed Status: Needs Work labels Sep 10, 2017

nicolas-grekas reviewed Sep 10, 2017

View reviewed changes

ro0NL mentioned this pull request Sep 10, 2017

Update %env()% documentation symfony/symfony-docs#8382

Closed

symfony deleted a comment from ro0NL Sep 11, 2017

nicolas-grekas reviewed Sep 11, 2017

View reviewed changes

grifx mentioned this pull request Mar 14, 2018

Impossible to use dotenv with this bundle Algatux/influxdb-bundle#80

Closed

nicolas-grekas approved these changes Mar 21, 2018

View reviewed changes

carsonbot added Status: Reviewed and removed Status: Needs Review labels Mar 21, 2018

nicolas-grekas approved these changes Mar 21, 2018

View reviewed changes

carsonbot added Status: Needs Work and removed Status: Reviewed labels Mar 22, 2018

fabpot requested changes Mar 22, 2018

View reviewed changes

ro0NL added 6 commits March 22, 2018 17:53

[DI] Validate env vars in config

68ee20e

optimized

d33b1c8

feedback

e7b9656

added changelog entry

1e59cf7

phpdocs

a90d34b

mark internal

e5f02cd

nicolas-grekas reviewed Mar 23, 2018

View reviewed changes

Update BaseNode.php

c76adcd

nicolas-grekas closed this in c2a67aa Mar 27, 2018

ro0NL deleted the config/placeholders branch March 27, 2018 07:16

mcfedr reviewed Mar 27, 2018

View reviewed changes

nicolas-grekas mentioned this pull request Mar 29, 2018

[Configuration] environment variables don't work on booleanNodes #26704

Closed

sstok mentioned this pull request Apr 2, 2018

Unable to resolve env() in (Security) configuration #26747

Closed

stof mentioned this pull request Apr 5, 2018

[DI] Environment variables lack type #26812

Closed

fabpot mentioned this pull request May 7, 2018

Release v4.1.0-BETA1 #27181

Merged

jmikola mentioned this pull request Jul 6, 2018

Bundle doesn't support port configuration by environment variable doctrine/DoctrineMongoDBBundle#466

Open

alex-bacart mentioned this pull request Sep 25, 2018

Symfony Config Component does not recognize environment variables #28599

Closed

[DI] Validate env vars in config #23888

[DI] Validate env vars in config #23888

Conversation

ro0NL commented Aug 14, 2017 • edited Loading

nicolas-grekas commented Aug 16, 2017 • edited Loading

ro0NL commented Sep 7, 2017 • edited Loading

nicolas-grekas commented Sep 8, 2017 • edited Loading

ro0NL commented Sep 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ro0NL Sep 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ro0NL Sep 9, 2017 • edited Loading

Choose a reason for hiding this comment

ro0NL commented Sep 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ro0NL commented Sep 10, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ro0NL commented Sep 10, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ro0NL commented Sep 10, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ro0NL Sep 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdaguerre commented Mar 19, 2018

nicolas-grekas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ro0NL Mar 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolas-grekas commented Mar 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Majkl578 commented May 31, 2018

ro0NL commented Aug 14, 2017 •

edited

Loading

nicolas-grekas commented Aug 16, 2017 •

edited

Loading

ro0NL commented Sep 7, 2017 •

edited

Loading

nicolas-grekas commented Sep 8, 2017 •

edited

Loading

ro0NL Sep 9, 2017 •

edited

Loading

ro0NL Sep 9, 2017 •

edited

Loading

ro0NL commented Sep 9, 2017 •

edited

Loading

ro0NL commented Sep 10, 2017 •

edited

Loading

ro0NL commented Sep 10, 2017 •

edited

Loading

ro0NL commented Sep 10, 2017 •

edited

Loading

ro0NL Sep 11, 2017 •

edited

Loading

ro0NL Mar 22, 2018 •

edited

Loading