[DI] Allow binary values in parameters. #25928

bburnichon · 2018-01-25T13:30:26Z

Q	A
Branch?	master
Bug fix?	yes
New feature?	yes
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets	#25916
License	MIT
Doc PR	none yet

This adds binary type in container xsd definition file

xabbuh · 2018-01-25T13:54:37Z

Doesn't this apply to 2.7 too?

xabbuh · 2018-01-25T13:55:08Z

src/Symfony/Component/DependencyInjection/Dumper/PhpDumper.php

@@ -1775,7 +1775,12 @@ private function export($value)

    private function doExport($value, $resolveEnv = false)
    {
-        if (is_string($value) && false !== strpos($value, "\n")) {
+        if (is_string($value) && preg_match('/[\x00-\x08\x0b-\x1f\x7f-\xff]+/', $value,$matches)) {


Is this really necessary?

Yes, otherwise, PhpDumper generates invalid UTF-8 files. As I was unable to properly add safely required characters

xabbuh · 2018-01-25T13:55:45Z

src/Symfony/Component/DependencyInjection/Dumper/XmlDumper.php

@@ -298,6 +298,10 @@ private function convertParameters(array $parameters, $type, \DOMElement $parent
                $element->setAttribute('type', 'expression');
                $text = $this->document->createTextNode(self::phpToXml((string) $value));
                $element->appendChild($text);
+            } elseif (is_string($value) && preg_match('/[\x00-\x08\x0b-\x1f\x7f-\xff]/', $value)) {


What about !preg_match('//u', $value) instead?

I tried /\p{C}/u but it was not working as expected

I retested it and tried to detect control characters instead except whitespaces

xabbuh · 2018-01-25T13:56:53Z

src/Symfony/Component/DependencyInjection/Loader/XmlFileLoader.php

@@ -511,6 +511,12 @@ private function getArgumentsAsPhp(\DOMElement $node, $name, $file, $lowercase =
                    }
                    $arguments[$key] = new TaggedIteratorArgument($arg->getAttribute('tag'));
                    break;
+                case 'binary':
+                    if (!preg_match('#^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{4}|[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)$#', $arg->nodeValue)) {


Is that necessary? base64_decode() would return false anyway which we could check for to detect errors, can't we?

Yes, could be done this way also. I generally tend to avoid calling methods with invalid parameters.

I would side with @xabbuh here: This class should not care about how a valid base64 string looks like. This is an implementation detail of the base64_decode function.

bburnichon · 2018-01-25T14:09:12Z

I've done this PR against master only as it requires xsd to have a new binary definition. But the actual fix can be applied to all versions I think.

fabbot.io gives an error in a fixture file. Could this be avoided?

derrabus · 2018-01-25T14:50:33Z

src/Symfony/Component/DependencyInjection/Loader/XmlFileLoader.php

@@ -511,6 +511,12 @@ private function getArgumentsAsPhp(\DOMElement $node, $name, $file, $lowercase =
                    }
                    $arguments[$key] = new TaggedIteratorArgument($arg->getAttribute('tag'));
                    break;
+                case 'binary':
+                    if (!preg_match('#^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{4}|[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)$#', $arg->nodeValue)) {


I would side with @xabbuh here: This class should not care about how a valid base64 string looks like. This is an implementation detail of the base64_decode function.

derrabus · 2018-01-25T14:54:53Z

fabbot.io gives an error in a fixture file. Could this be avoided?

You did not cause the error, so you can ignore it, imho.

dunglas · 2018-01-26T14:51:11Z

It deserves a unit test.

I would be more comfortable to merge this change in master instead of in 2.7. It's not that trivial.

bburnichon · 2018-01-26T15:09:42Z

I made PR against master, not 2.7. Also, what kind of unit test need to be added? I already added the use case in Dumpers and Loaders. Do you have another type of parameter to check?

bburnichon · 2018-02-09T17:16:13Z

ping @derrabus @dunglas @xabbuh Can you change target to master instead of 2.7 I do agree that this is a new feature that should go to next version of php. On older version, a bug warning should indicate binary parameters are not well handled.

xabbuh · 2018-03-04T08:34:34Z

The CS complaints in the fixtures file can indeed be ignored. But can you please switch back to using explicit array() instead of the short array notation in the dumper class? :)

xabbuh · 2018-03-04T08:36:31Z

src/Symfony/Component/DependencyInjection/Loader/XmlFileLoader.php

@@ -511,6 +511,12 @@ private function getArgumentsAsPhp(\DOMElement $node, $name, $file, $lowercase =
                    }
                    $arguments[$key] = new TaggedIteratorArgument($arg->getAttribute('tag'));
                    break;
+                case 'binary':
+                    if (false === $value = base64_decode($arg->nodeValue)) {
+                        throw new InvalidArgumentException(sprintf('Tag "<%s>" with type="binary" is not valid base64 encoded string', $name));


[...] is not a valid [...]

Please also terminate the exception message with a dot.

xabbuh · 2018-03-04T08:40:29Z

src/Symfony/Component/DependencyInjection/Dumper/PhpDumper.php

@@ -1775,7 +1775,12 @@ private function export($value)

    private function doExport($value, $resolveEnv = false)
    {
-        if (is_string($value) && false !== strpos($value, "\n")) {
+        if (is_string($value) && \in_array(preg_match('/[^\s\P{Cc}]/u', $value), [false, 1], true)) {


Is the change for the PHP dumper really necessary? Was there anything breaking without this change?

Not really but then generated file could contain non-utf8 characters and opening generated file with an ide leads to warnings about file encoding.

nothing we need to worry about IMO

xabbuh · 2018-03-04T08:41:26Z

src/Symfony/Component/DependencyInjection/Dumper/XmlDumper.php

@@ -298,6 +298,10 @@ private function convertParameters(array $parameters, $type, \DOMElement $parent
                $element->setAttribute('type', 'expression');
                $text = $this->document->createTextNode(self::phpToXml((string) $value));
                $element->appendChild($text);
+            } elseif (is_string($value) && \in_array(preg_match('/[^\s\P{Cc}]/u', $value), [false, 1], true)) {


Why do we need to check for both false and 1? Do we cover both values with the current test suite?

false is returned when value contains non-utf8 characters. 1 is returned when a control character is present in string. You're right a case is missing in current test suite.

bburnichon · 2018-03-05T09:06:01Z

Used traditional array syntax and added test on parameter with a single control char.

nicolas-grekas · 2018-03-19T17:13:26Z

src/Symfony/Component/DependencyInjection/Loader/XmlFileLoader.php

@@ -511,6 +511,12 @@ private function getArgumentsAsPhp(\DOMElement $node, $name, $file, $lowercase =
                    }
                    $arguments[$key] = new TaggedIteratorArgument($arg->getAttribute('tag'));
                    break;
+                case 'binary':


It this really legitimate? I mean: if !!binary is a core yaml feature, it should be transparent. Yaml tagged-values are only for custom tags, not core ones AFAIK.

I looked for a way to indicate in a XML file (which should be a valid UTF-8 file) that a value was base64_encoded(). I though this was the exact same purpose as in yaml file. I did not knew that !!binary was part of the yaml specification.

IMO using the same term here is legitimate. I would keep it as is.

yes, sorry, I misread the code, all good here

nicolas-grekas · 2018-03-19T17:14:27Z

src/Symfony/Component/DependencyInjection/Dumper/PhpDumper.php

@@ -1811,7 +1811,12 @@ private function export($value)

    private function doExport($value, $resolveEnv = false)
    {
-        if (is_string($value) && false !== strpos($value, "\n")) {
+        if (is_string($value) && \in_array(preg_match('/[^\s\P{Cc}]/u', $value), array(false, 1), true)) {


When can preg_match return false here?
What's the reasoning for choosing this regexp? (any reference/inspiration maybe?)

If $value is not an UTF-8 character string, preg_match will return false because of the u modifier.
Regexp was chosen to detects all characters which are control characters \p{Cc} and not a space (which are valid ascii). I mean, putting a bell character in a string is very unlikely done in purpose. I don't know of a way to put this in a file except by copy/paste.

Hum, I see. Then why unicode at all? No need for the u modifier, isn't it?

If I not set u modifier, I can NOT use \p{Cc} to find the allowed characters as its extension for unicode. My first try was just selecting all characters outside standard ASCII but it would filter too much IMO.

no need for unicode, let's list the ASCII Cc explicitly, here they are:
https://en.wikipedia.org/wiki/Unicode_control_characters

Do you want a different regex for XML and PHP serialization? I mean, in XML, I need to check whether string contains only valid UTF-8 characters implying check with the u modifier.
For PHP, I can only check for [\0-\x1f\x7f] but then simple strings like "Foo\tTab" would be base64 encoded whereas "\xf0" would not.

I'd suggest only one way to detect binary, and using this:
[\x00-\x08\x0B\x0E-\x1A\x1C-\x1F\x7F]

nicolas-grekas · 2018-03-22T16:25:29Z

src/Symfony/Component/DependencyInjection/Dumper/PhpDumper.php

@@ -1811,7 +1811,12 @@ private function export($value)

    private function doExport($value, $resolveEnv = false)
    {
-        if (is_string($value) && false !== strpos($value, "\n")) {
+        if (is_string($value) && \in_array(preg_match('/[\x00-\x08\x0B\x0E-\x1A\x1C-\x1F\x7F]/u', $value), array(false, 1), true)) {


no need for the u modifier, thus no need for in_array

nicolas-grekas · 2018-03-22T17:47:18Z

as bugfix on 3.4?

fabpot · 2018-03-27T12:07:26Z

Thank you @bburnichon.

This PR was squashed before being merged into the 4.1-dev branch (closes #25928). Discussion ---------- [DI] Allow binary values in parameters. | Q | A | ------------- | --- | Branch? | master | Bug fix? | yes | New feature? | yes | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #25916 | License | MIT | Doc PR | none yet This adds `binary` type in container xsd definition file Commits ------- cb23134 [DI] Allow binary values in parameters.

stof · 2018-03-27T13:45:57Z

XML Dumper changes are missing their tests

stof · 2018-03-27T14:47:36Z

We should also support it for argument types for consistency

xabbuh · 2018-03-29T07:48:36Z

src/Symfony/Component/DependencyInjection/Dumper/XmlDumper.php

@@ -298,6 +298,10 @@ private function convertParameters(array $parameters, $type, \DOMElement $parent
                $element->setAttribute('type', 'expression');
                $text = $this->document->createTextNode(self::phpToXml((string) $value));
                $element->appendChild($text);
+            } elseif (\is_string($value) && !preg_match('/^[^\x00-\x08\x0B\x0E-\x1A\x1C-\x1F\x7F]*+$/u', $value)) {


Do we still need the u modifier here?

Yes, to ensure string is a valid UTF8 string for XML file.

This PR was merged into the 4.4 branch. Discussion ---------- [DI] fix parsing of argument type=binary in xml | Q | A | ------------- | --- | Branch? | 4.4 | Bug fix? | yes | New feature? | no  | Deprecations? | no  | Tickets | | License | MIT | Doc PR | Parameters and arguments can be of type="binary" since #25928. But the XSD forgot to be updated for arguments. So if you try to run `bin/console debug:container` you'd get an error like ``` In XmlFileLoader.php line 385: Unable to parse file "var/cache/dev/App_KernelDevDebugContainer.xml": [ERROR 1840] Element '{ht tp://symfony.com/schema/dic/services}argument', attribute 'type': [facet 'enumeration'] The value 'binary' is not a n element of the set {'abstract', 'collection', 'service', 'expression', 'string', 'constant', 'iterator', 'service _locator', 'tagged', 'tagged_iterator', 'tagged_locator'}. (in /home/benny/project/ - line 505, column 0) [ERROR 1824] Element '{http://symfony.com/schema/dic/services}argument', attribute 'type': 'binary' is not a valid value of the atomic type '{http://symfony.com/schema/dic/services}argument_type'. (in /home/benny/project/ - line 5 05, column 0)` ``` Commits ------- 70b6a00 [DI] fix parsing of argument type=binary in xml

…ers. (vtsykun) This PR was merged into the 5.4 branch. Discussion ---------- [DependencyInjection] Fix support binary values in parameters. | Q | A | ------------- | --- | Branch? | 5.4 | Bug fix? | yes | New feature? | no | Deprecations? | no | Tickets | Fix #49638 | License | MIT | Doc PR | - This issue related to #25928 ### Step to reproduce 1. Add parameter like this ```yaml parameters: banner_message: "\e[37;44m#StandWith\e[30;43mUkraine\e[0m" ``` 2. Run command `debug:router` Actual result: ![Selection_1343](https://user-images.githubusercontent.com/21358010/224194622-6d814bbc-823d-43db-8e11-43a46f09ad57.png) Commits ------- 8541643 Fix support binary values in parameters.

carsonbot added Status: Needs Review Bug Feature labels Jan 25, 2018

xabbuh added DependencyInjection and removed Feature labels Jan 25, 2018

xabbuh reviewed Jan 25, 2018

View reviewed changes

bburnichon force-pushed the container-with-non-utf8-parameters branch from c43ca2c to e5d9ba8 Compare January 25, 2018 14:00

derrabus requested changes Jan 25, 2018

View reviewed changes

carsonbot added Status: Needs Work and removed Status: Needs Review labels Jan 25, 2018

xabbuh added this to the 2.7 milestone Jan 26, 2018

nicolas-grekas modified the milestones: 2.7, 4.1 Feb 27, 2018

nicolas-grekas changed the title ~~Allow binary values in parameters.~~ [DI] Allow binary values in parameters. Feb 27, 2018

xabbuh reviewed Mar 4, 2018

View reviewed changes

Allow binary values in parameters.

ae1684c

bburnichon force-pushed the container-with-non-utf8-parameters branch from af66815 to ae1684c Compare March 5, 2018 09:14

nicolas-grekas reviewed Mar 19, 2018

View reviewed changes

Change regexp to Invalid Byte selection

60436ac

nicolas-grekas reviewed Mar 22, 2018

View reviewed changes

bburnichon added 2 commits March 22, 2018 18:17

Reverse regexp to avoid in_array()

801e9b3

Revert PhpDumper.php changes

0e46698

carsonbot removed the Status: Needs Work label Mar 22, 2018

nicolas-grekas approved these changes Mar 22, 2018

View reviewed changes

carsonbot added the Status: Reviewed label Mar 22, 2018

fabpot closed this Mar 27, 2018

bburnichon deleted the container-with-non-utf8-parameters branch March 27, 2018 14:06

xabbuh reviewed Mar 29, 2018

View reviewed changes

javiereguiluz mentioned this pull request Apr 20, 2018

[DependencyInjection] Allow binary values in parameters symfony/symfony-docs#9650

Closed

fabpot mentioned this pull request May 7, 2018

Release v4.1.0-BETA1 #27181

Merged

Tobion mentioned this pull request Jun 26, 2020

[DI] fix parsing of argument type=binary in xml #37429

Merged

vtsykun mentioned this pull request Mar 10, 2023

[DependencyInjection] Fix support binary values in parameters. #49651

Merged

[DI] Allow binary values in parameters. #25928

[DI] Allow binary values in parameters. #25928

Conversation

bburnichon commented Jan 25, 2018

xabbuh commented Jan 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bburnichon commented Jan 25, 2018 • edited Loading

Choose a reason for hiding this comment

derrabus commented Jan 25, 2018

dunglas commented Jan 26, 2018

bburnichon commented Jan 26, 2018

bburnichon commented Feb 9, 2018

xabbuh commented Mar 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xabbuh Mar 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bburnichon commented Mar 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolas-grekas Mar 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolas-grekas commented Mar 22, 2018

fabpot commented Mar 27, 2018

stof commented Mar 27, 2018

stof commented Mar 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bburnichon commented Jan 25, 2018 •

edited

Loading

xabbuh Mar 4, 2018 •

edited

Loading

nicolas-grekas Mar 20, 2018 •

edited

Loading