Is there any way to add processes to queue on the fly?

for example I have 10000 users in DB. and for each user I want to call some process (but only 5 in parallel). So it's silly to get all 10000 users and create 10000 processes at once.

Maybe add some process providers, or add events

Cool feature 👍

What about a process manager builder ? Something like this :

namespace Symfony\Component\Process\Manager;

class ManagerBuilder
{
    public function chain(array $processes)
    {
        $manager = new ProcessManager();
        $manager->setMaxParallelProcesses(1);

        foreach ($processes as $name => $process) {
            $manager->add($process, $name);
        }

        return $manager;
    }

    public function parallel(array $processes)
    {
        $manager = new ProcessManager();

        foreach ($processes as $name => $process) {
            $manager->add($process, $name);
        }

        return $manager;
    }
}

usage example :

$manager = $builder->chain(
    $builder->parallel(
        $builder->chain(array($process1, $process2, $process3)),
        $builder->chain(array($process4, $process5))
    ),
    $process6
);
$manager->run();

any ideas ?

@romainneutron Maybe also smth like auto method from node.js async library?
example from library

async.auto({
    get_data: function(callback){
        // async code to get some data
    },
    make_folder: function(callback){
        // async code to create a directory to store a file in
        // this is run at the same time as getting the data
    },
    write_file: ['get_data', 'make_folder', function(callback){
        // once there is some data and the directory exists,
        // write the data to a file in the directory
        callback(null, filename);
    }],
    email_link: ['write_file', function(callback, results){
        // once the file is written let's email a link to it...
        // results.write_file contains the filename returned by write_file.
    }]
});

Hello,

node.js is an event driven environment, and async.js is a tool to deal with async behaviors. Actually, Process is not asynchronous. The Process::start method allows you to run a process in "background" but the polling mechanism should be triggered periodically to fetch output (and results). This polling mechanism is triggered whenever you call a status method (Process::isRunning, Process::checkTimeout, Process::getOutput, etc).

So this kind of behavior could not be ported. Anyway, not in the same fashion. If you're interested in async process, have a look at react project, particularly the child process PR and the https://github.com/reactphp/promise project that provides promise for the react environment. This is out of topic because this environment requires non-blocking calls, what's not at all the case of Symfony.

@romainneutron I mean ability to call processes in some order. But looks like I can achieve this using chain and parallel methods:

for example

$builder->chain(
    $builder->parallel($getDataProcess, $makeFolderProcess)),
    $writeFileProcess,
    $emailLinkProcess
);

Yes it works. The difference is the ability to pass the output between process. This is something that we could implement in the future, see FR #8161

I mean that $writeFileProcess could use the output of $getDataProcess

@fabpot what's the state on this one ? Could it be merge in 2.4 ? If so, I've to finish the ManagerBuilder and work on the doc ASAP.

If it's too late for this, can we merge the interfaces (2 first commits) so the ProcessManager can be used and tested without using a custom symfony repo ?

We are not in a hurry. 2.4 end of dev is at the end of September. I haven't had time to have a look at this PR yet, but I will soon.

👍 Great feature! Would love to see this integrated into 2.4.

I'd really like to see something like a ProcessProvider:

<?php
class ManagerBuilder
{
    public function chain(ProcessProviderInterface $processProvider)
    {
        $manager = ProcessManager::create();
        $manager->setMaxParallelProcesses(1);

        foreach ($processProvider as $name => $process) {
            $manager->add($process, $name);
        }

        return $manager;
    }

    public function parallel(ProcessProviderInterface $processProvider)
    {
        $manager = ProcessManager::create();

        foreach ($processProvider as $name => $process) {
            $manager->add($process, $name);
        }

        return $manager;
    }
}

Usage example:

<?php

// Implemets ProcessProviderInterface
$batchProcessProvider = new BatchProcessProvider();
$batchProcessProvider
    ->setCommandTemplate(
        'exec php -f app/console project:update-users --limit=%limit% --offset=%offset%'
    )
    ->setPlaceholders(array(
        '%limit%'   => 'limit',
        '%offset%' => 'offset'
    ))
    ->setTotalCount(1000000)
    ->setLimit(20000)
;

$q = $em->createQuery('select u from MyProject\Model\User u');

$doctrineBatchProcessProvider = BatchProcessProvider::createFromIterator(
    $q->iterate(),
    'exec php -f app/console project:update-user-attributes --id=%id%',
    array('%id%' => 'id')
);


$manager = $builder->chain(
    $builder->parallel(
        $builder->chain($batchProcessProvider),
        $builder->chain(ProcessProvider::createFromArray(
            array(
                $process4,
                'processName' => $process5
            )
        )),
        $doctrineBatchProcessProvider
    ),
    ProcessProvider::createFromArray(array($process6))
);
$manager->run();

/cc @sirian

@tPl0ch imho - it's silly to add all processes at start point like in ManagerBuilder::parallel example (to decrease memory usage).
My sugesstion - ProcessProvider should implement Iterator interface (http://ru2.php.net/manual/ru/class.iterator.php), and ProcessManager should work correctly with this providers (when free slot for process is available - get next process from process provider)

ProcessProvider is indeed something that would be quite nice. But we need to separate concerns and ProcessProvider should not deal directly with ProcessManager that has the job to run multiple instancied processes.

I think it should be implemented at higher level than ProcessManager. To do so, we need to some informations about its state like ProcessManager::hasSlot or ProcessManager::getQueueLength. What do you think about this ?

ping @fabpot

@fabpot If this PR is not mergeable in 2.4, can we at least think about merging the interfaces (two first commits) ?

I will need more time on this one as I have something equivalent (but not exactly). That's something we will be able to do in 2.5.

Okay then, bring the code at symfonycon hackday ;)

ping @fabpot @romainneutron
Have you move forward on this manager? That's something really interesting and 2.5 is out for long (#8753 (comment))