[Process] Allow writing portable "prepared" command lines #24763

Simperfit · 2017-10-31T06:03:58Z

Q	A
Branch?	4.1
Bug fix?	no
New feature?	yes
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets	#23778
License	MIT
Doc PR	symfony/symfony-docs#9295

This give the opportunity to create process commands that allow to changes only the values instead of changing the code.

dunglas · 2017-10-31T09:47:33Z

src/Symfony/Component/Process/Process.php

@@ -278,8 +278,13 @@ public function start(callable $callback = null, array $env = array())

        $options = array('suppress_errors' => true);

+        if ('\\' !== DIRECTORY_SEPARATOR) {


What about:

$windows = '\\' === DIRECTORY_SEPARATOR; $this->replacePlaceholder($commandline, $windows ? array('!', '!') : array('$', '')); if ($windows) {

dunglas · 2017-10-31T09:50:15Z

src/Symfony/Component/Process/Process.php

+
+    private function replacePlaceholder($commandLine, $toReplace)
+    {
+        if (preg_match('/\{([a-zA-Z0-9]*?)\}/', $commandLine)) {


Either this test is useless (because { or } chars are replaced) or the following line is buggy (because { and } are replaced regardless that they match the previous regex).

From what I understand, it's buggy. Only placeholders like {foo} must be replaced. {} should not match, echo '{{foo}' should return echo '{value' too.

Something like this should do the trick:

if (!preg_match_all('/\{([a-zA-Z0-9]+?)\}/', $commandLine, $matches)) { return $commandLine; } $patterns = $replacements = array(); foreach ($matches as $match) { $patterns[] = sprintf('/{%s}/', $match[1]); $replacements[] = sprintf('%s%s%s', $toReplace[0], $match[1], $toReplace[1]); } return preg_replace($patterns, $replacements, $commandLine);

(I've not tested it, but you got the idea)

I understand and will modify according to the exemple

Simperfit · 2017-10-31T14:24:39Z

done @dunglas

Simperfit · 2017-10-31T15:19:48Z

Status: Needs Review

nicolas-grekas · 2017-11-01T17:46:01Z

src/Symfony/Component/Process/Process.php

+
+    private function replacePlaceholder($commandLine, $toReplace)
+    {
+        if (!preg_match_all('/\{([a-zA-Z0-9]+?)\}/', $commandLine, $matches)) {


I think this could be replaced by a single call to preg_replace
the pattern is missing the underscore also

nicolas-grekas · 2017-11-01T17:47:12Z

src/Symfony/Component/Process/Process.php

@@ -278,7 +278,10 @@ public function start(callable $callback = null, array $env = array())

        $options = array('suppress_errors' => true);

-        if ('\\' === DIRECTORY_SEPARATOR) {
+        $windows = '\\' === DIRECTORY_SEPARATOR;
+        $commandline = $this->replacePlaceholder($commandline, $windows ? array('!', '!') : array('$', ''));


the "$toReplace" should be moved inside the method, there is no reason to leak this outside of the method I think

nicolas-grekas · 2017-11-01T17:50:22Z

Someone proposed to fail when a var was used, but not provided when calling. Is that something that would be possible?

Simperfit · 2017-11-01T18:59:24Z

I need to figure out, but I don't see anything that would make it not possible.

Simperfit · 2017-11-07T07:44:03Z

Comments addressed and new throw added.

dunglas · 2017-11-08T20:51:53Z

src/Symfony/Component/Process/Process.php

+            $patterns[] = sprintf('/{%s}/', $match[0]);
+            $replacements[] = sprintf('%s%s%s', $toReplace[0], $match[0], $toReplace[1]);
+            if (isset($env[$match[0]])) {
+                $isInEnv = true;


Shouldn't you store unused placeholders to display a nice error message like The placeholders "foo", "bar" doesn't exist?

I can do that, that's a good idea.

Any update?

xabbuh · 2017-12-07T08:07:48Z

src/Symfony/Component/Process/Process.php

+        }
+
+        if (!empty($env)) {
+            throw new InvalidArgumentException(sprintf('Placeholder%s "%s" doesn\'t exist.', count($env) > 1 ? 's' : '', implode(',', array_keys($env))));


If you make the distinction between singular and plural for "placeholder/s", you should probably do the same for "doesn't/don't" too.

Simperfit · 2018-02-19T14:02:01Z

Doc PR has been added.

dunglas · 2018-02-19T14:07:58Z

Thanks @Simperfit.

…nes (Simperfit) This PR was merged into the 4.1-dev branch. Discussion ---------- [Process] Allow writing portable "prepared" command lines | Q | A | ------------- | --- | Branch? | 4.1 | Bug fix? | no | New feature? | yes | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #23778 | License | MIT | Doc PR | symfony/symfony-docs#9295 This give the opportunity to create process commands that allow to changes only the values instead of changing the code. Commits ------- d1e4f48 [Process] Allow writing portable "prepared" command lines

Tobion · 2018-02-25T17:16:58Z

I don't agree with this implementation. The new syntax is very specific to the Symfony process now and not suited within the a general Process class. This logic should be split out into some sort of ProcessBuilder that returns the final command line that you then pass to the Process.
Why should the process class know about this syntax? It's not its responsiblity.

nicolas-grekas · 2018-02-25T20:39:13Z

@Tobion you can think of this as SQL prepared statements. Note that the Process component doesn't provide any mean to escape shell arguments anymore, exactly like you should not use any for SQL drivers. And also because it is just impossible to properly escape shell arguments (same for SQL btw, when taking portability into account).
Like PHP PDO handles prepared statements directly, it can also be legit to make Process handle "prepared" command lines, to provide portability across OSes for shell placeholders (esp. ${FOO} vs !FOO! on Windows, which you can still use if you really don't want to use the Symfony placeholders btw).

Tobion · 2018-02-25T21:45:53Z

PDO provides different methods for prepared statements to separate concerns. Exactly what I'm asking for.

nicolas-grekas · 2018-02-25T22:18:35Z

I'm really not convinced that a different design would provide anything better sorry.
This change is the last of a series of moves that turn Process into a properly handled abstraction of command lines and escaping of arguments (which should never have been exposed to userland, like sql arguments.)

malarzm · 2018-02-28T15:07:05Z

Late to the game, but isn't this a BC break for folks using something like echo "{{ some_var }} foo bar" >> something.j2 where they expect {{ some_var }} to be echoed to a file?

Tobion · 2018-02-28T15:20:59Z

There are probably bc breaks here. that is also why I don't agree with using the same api for this because you cannot control it anymore.

nicolas-grekas · 2018-02-28T15:25:36Z

Nobody echoes like that (they do file_put_contents() instead)
You need to take escaping into consideration when thinking about this.
We already forced ppl to not escape values on their own (there is no way to do this in 4.0)
So either they do escapeshellargs() on their own, which is totally broken and unsupported.
When done properly, either ppl use the array-way to define command lines, thus cannot use placeholders, either they use env vars already, but the non portable OS-dependent style. - there is no other way for dynamic values.
Last possibility: ppl inject static values that they escaped/wrote themselves (that's your example), which I really don't expect to collide in practice.
All in all, I think this means dynamic values cannot collide in any supported way, and static values are really unlikely to collide.

Tobion · 2018-02-28T15:43:37Z

I would expect that I can run a command with the process component the same way it works on the shell. This is not possible anymore and you have to think and know about Process special behavior now (in addition to correct escaping).

nicolas-grekas · 2018-02-28T15:49:21Z

do you have a real-life example?

malarzm · 2018-02-28T15:50:58Z

Nobody echoes like that (they do file_put_contents() instead)

This is rather bold statement. Also I'm not talking about escaping, in my example the user would want to have plain {{ some_var }} foo bar appended to the something.j2 file ({{ some_var }} is valid syntax for j2). In 4.0 it would work, in 4.1 it'd be an exception.

IMO if you're not willing to revert the feature there should be a feature flag turned off by default in 4.x and switched to on in 5.x.

nicolas-grekas · 2018-02-28T16:00:11Z

@malarzm you were asking about BC break. I explained why I think the collision is more theoretical than practical IMHO.

Tobion · 2018-02-28T16:00:16Z

echo "My template {{ var }}" | bin/console lint:twig kboom

nicolas-grekas · 2018-02-28T16:02:59Z

@Tobion sure, but this a constructed example...
In a real app, you'll do (in 4.0, on Linux):
(new Process('echo $TWIG | bin/console lint:twig'))->run(null, array('TWIG' => 'My template {{ var }}');
as there is no other way to inject a value (and running a command to lint a static string is not something "real", you might agree :) )

Tobion · 2018-02-28T16:07:09Z

It's not about if it's real example or not. It's about expectations. Why am I not allowed to run a process that works with the process component? The job of the process component is it make it easier to run processes from PHP, not harder with edge cases that don't work and behave unexpected.

javiereguiluz · 2018-02-28T16:14:28Z

I think Tobias has a point here. Suddenly, your Symfony app can't run commands that contain {{ ... }} (unless you update your code) Of course it's not super common to include {{ ... }} in your commands but I don't think it's impossible at all.

nicolas-grekas · 2018-02-28T17:04:53Z

OK, see #26344

…red" command lines (Simperfit)" This reverts commit 1364089, reversing changes made to e043478.

…e "prepared" command lines (Simperfit)" (nicolas-grekas) This PR was merged into the 4.1-dev branch. Discussion ---------- Revert "feature #24763 [Process] Allow writing portable "prepared" command lines (Simperfit)" | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | yes | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | - | License | MIT | Doc PR | - This reverts commit 1364089, reversing changes made to e043478. As discussed in #24763 and #26344 This doens't revert the possibility to use prepared command lines. They just won't be *portable* anymore, unless special care is taken by "userland". Ie the placeholders need to be shell-dependent: use eg `echo "$FOO"` on *nix (the double quotes *are* important), and `echo !FOO!` on Windows (no double quotes there). Commits ------- 6a98bfa Revert "feature #24763 [Process] Allow writing portable "prepared" command lines (Simperfit)"

@nicolas-grekas

…nes (Simperfit) This PR was merged into the 4.4 branch. Discussion ---------- [Process] Allow writing portable "prepared" command lines | Q | A | ------------- | --- | Branch? | 4.4 | Bug fix? | no | New feature? | yes  | BC breaks? | no  | Deprecations? | no  | Tests pass? | yes  | Fixed tickets | #23778  | License | MIT | Doc PR | symfony/symfony-docs#11802  Hey here, it's me, again ! I've talked with @nicolas-grekas and he gave me a new way of writing this feature that will not be a problem for people using things like {{ toto }} since we are using the linux style of using envvar, the replaced arguments is only replaced when using windows. This should not break anything and work as expected! see #24763 This makes `"$FOO"` work seamlessly on Linux and on Windows to reference an env var (that can be provided when calling e.g. the "run" method) Commits ------- 3f8354f [Process] Allow writing portable "prepared" command lines

carsonbot added Status: Needs Review Process Feature labels Oct 31, 2017

Simperfit force-pushed the feature/allow-writing-prepared-command-line branch 3 times, most recently from 2db2948 to 2c23e96 Compare October 31, 2017 07:05

chalasr added this to the 4.1 milestone Oct 31, 2017

dunglas requested changes Oct 31, 2017

View reviewed changes

carsonbot added Status: Needs Work and removed Status: Needs Review labels Oct 31, 2017

Simperfit force-pushed the feature/allow-writing-prepared-command-line branch 2 times, most recently from e8973b5 to 0bf646c Compare October 31, 2017 13:12

carsonbot added Status: Needs Review and removed Status: Needs Work labels Oct 31, 2017

Simperfit force-pushed the feature/allow-writing-prepared-command-line branch from 0bf646c to c6c730c Compare November 1, 2017 07:03

nicolas-grekas reviewed Nov 1, 2017

View reviewed changes

Simperfit force-pushed the feature/allow-writing-prepared-command-line branch 2 times, most recently from ca1a267 to 51f6947 Compare November 7, 2017 07:43

Simperfit force-pushed the feature/allow-writing-prepared-command-line branch from 51f6947 to 0dda934 Compare November 7, 2017 07:50

dunglas reviewed Nov 8, 2017

View reviewed changes

Simperfit force-pushed the feature/allow-writing-prepared-command-line branch from 0dda934 to e596524 Compare December 1, 2017 17:49

xabbuh reviewed Dec 7, 2017

View reviewed changes

Simperfit force-pushed the feature/allow-writing-prepared-command-line branch from e596524 to 4bf2bb3 Compare December 7, 2017 09:23

Simperfit force-pushed the feature/allow-writing-prepared-command-line branch from 4bf2bb3 to c49cbb6 Compare December 16, 2017 04:52

Simperfit mentioned this pull request Feb 19, 2018

[Process] allow writing "prepared" command line. symfony/symfony-docs#9295

Closed

dunglas approved these changes Feb 19, 2018

View reviewed changes

dunglas merged commit d1e4f48 into symfony:master Feb 19, 2018

Simperfit deleted the feature/allow-writing-prepared-command-line branch February 26, 2018 08:51

nicolas-grekas mentioned this pull request Feb 28, 2018

[Process] Make prepared command lines opt-in #26344

Closed

nicolas-grekas added a commit to nicolas-grekas/symfony that referenced this pull request Mar 2, 2018

Revert "feature symfony#24763 [Process] Allow writing portable "prepa…

6a98bfa

…red" command lines (Simperfit)" This reverts commit 1364089, reversing changes made to e043478.

nicolas-grekas mentioned this pull request Mar 2, 2018

Revert "feature #24763 [Process] Allow writing portable "prepared" command lines (Simperfit)" #26372

Merged

fabpot mentioned this pull request May 7, 2018

Release v4.1.0-BETA1 #27181

Merged

Simperfit mentioned this pull request Jun 21, 2019

[Process] Allow writing portable "prepared" command lines #32126

Merged

		@@ -278,8 +278,13 @@ public function start(callable $callback = null, array $env = array())

		$options = array('suppress_errors' => true);

		if ('\\' !== DIRECTORY_SEPARATOR) {

[Process] Allow writing portable "prepared" command lines #24763

[Process] Allow writing portable "prepared" command lines #24763

Conversation

Simperfit commented Oct 31, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dunglas Oct 31, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Simperfit commented Oct 31, 2017

Simperfit commented Oct 31, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolas-grekas commented Nov 1, 2017

Simperfit commented Nov 1, 2017

Simperfit commented Nov 7, 2017

dunglas Nov 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Simperfit commented Feb 19, 2018

dunglas commented Feb 19, 2018

Tobion commented Feb 25, 2018 • edited Loading

nicolas-grekas commented Feb 25, 2018 • edited Loading

Tobion commented Feb 25, 2018

nicolas-grekas commented Feb 25, 2018 • edited Loading

malarzm commented Feb 28, 2018

Tobion commented Feb 28, 2018

nicolas-grekas commented Feb 28, 2018 • edited Loading

Tobion commented Feb 28, 2018

nicolas-grekas commented Feb 28, 2018

malarzm commented Feb 28, 2018

nicolas-grekas commented Feb 28, 2018

Tobion commented Feb 28, 2018

nicolas-grekas commented Feb 28, 2018 • edited Loading

Tobion commented Feb 28, 2018

javiereguiluz commented Feb 28, 2018

nicolas-grekas commented Feb 28, 2018

Simperfit commented Oct 31, 2017 •

edited

Loading

dunglas Oct 31, 2017 •

edited

Loading

dunglas Nov 8, 2017 •

edited

Loading

Tobion commented Feb 25, 2018 •

edited

Loading

nicolas-grekas commented Feb 25, 2018 •

edited

Loading

nicolas-grekas commented Feb 25, 2018 •

edited

Loading

nicolas-grekas commented Feb 28, 2018 •

edited

Loading

nicolas-grekas commented Feb 28, 2018 •

edited

Loading