{"action":"created","issue":{"url":"https://api.github.com/repos/symfony/symfony/issues/27518","repository_url":"https://api.github.com/repos/symfony/symfony","labels_url":"https://api.github.com/repos/symfony/symfony/issues/27518/labels{/name}","comments_url":"https://api.github.com/repos/symfony/symfony/issues/27518/comments","events_url":"https://api.github.com/repos/symfony/symfony/issues/27518/events","html_url":"https://github.com/symfony/symfony/pull/27518","id":329801928,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkyOTYyOTQ2","number":27518,"title":"[Console] Added Parallelization trait","user":{"login":"webmozart","id":176399,"node_id":"MDQ6VXNlcjE3NjM5OQ==","avatar_url":"https://avatars3.githubusercontent.com/u/176399?v=4","gravatar_id":"","url":"https://api.github.com/users/webmozart","html_url":"https://github.com/webmozart","followers_url":"https://api.github.com/users/webmozart/followers","following_url":"https://api.github.com/users/webmozart/following{/other_user}","gists_url":"https://api.github.com/users/webmozart/gists{/gist_id}","starred_url":"https://api.github.com/users/webmozart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/webmozart/subscriptions","organizations_url":"https://api.github.com/users/webmozart/orgs","repos_url":"https://api.github.com/users/webmozart/repos","events_url":"https://api.github.com/users/webmozart/events{/privacy}","received_events_url":"https://api.github.com/users/webmozart/received_events","type":"User","site_admin":false},"labels":[{"id":125388,"node_id":"MDU6TGFiZWwxMjUzODg=","url":"https://api.github.com/repos/symfony/symfony/labels/Console","name":"Console","color":"DDDDDD","default":false},{"id":100378,"node_id":"MDU6TGFiZWwxMDAzNzg=","url":"https://api.github.com/repos/symfony/symfony/labels/Feature","name":"Feature","color":"02e10c","default":false},{"id":238909857,"node_id":"MDU6TGFiZWwyMzg5MDk4NTc=","url":"https://api.github.com/repos/symfony/symfony/labels/Status:%20Needs%20Work","name":"Status: Needs Work","color":"fad8c7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/symfony/symfony/milestones/18","html_url":"https://github.com/symfony/symfony/milestone/18","labels_url":"https://api.github.com/repos/symfony/symfony/milestones/18/labels","id":3234645,"node_id":"MDk6TWlsZXN0b25lMzIzNDY0NQ==","number":18,"title":"next","description":"https://symfony.com/roadmap?version=4.2#checker","creator":{"login":"nicolas-grekas","id":243674,"node_id":"MDQ6VXNlcjI0MzY3NA==","avatar_url":"https://avatars3.githubusercontent.com/u/243674?v=4","gravatar_id":"","url":"https://api.github.com/users/nicolas-grekas","html_url":"https://github.com/nicolas-grekas","followers_url":"https://api.github.com/users/nicolas-grekas/followers","following_url":"https://api.github.com/users/nicolas-grekas/following{/other_user}","gists_url":"https://api.github.com/users/nicolas-grekas/gists{/gist_id}","starred_url":"https://api.github.com/users/nicolas-grekas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicolas-grekas/subscriptions","organizations_url":"https://api.github.com/users/nicolas-grekas/orgs","repos_url":"https://api.github.com/users/nicolas-grekas/repos","events_url":"https://api.github.com/users/nicolas-grekas/events{/privacy}","received_events_url":"https://api.github.com/users/nicolas-grekas/received_events","type":"User","site_admin":false},"open_issues":141,"closed_issues":80,"state":"open","created_at":"2018-04-02T12:03:21Z","updated_at":"2018-06-28T06:43:21Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2018-06-06T10:06:14Z","updated_at":"2018-06-28T07:54:27Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/symfony/symfony/pulls/27518","html_url":"https://github.com/symfony/symfony/pull/27518","diff_url":"https://github.com/symfony/symfony/pull/27518.diff","patch_url":"https://github.com/symfony/symfony/pull/27518.patch"},"body":"| Q             | A\r\n| ------------- | ---\r\n| Branch?       | master\r\n| Bug fix?      | no\r\n| New feature?  | yes\r\n| BC breaks?    | no\r\n| Deprecations? | no\r\n| Tests pass?   | yes\r\n| Fixed tickets | #8454 ?\r\n| License       | MIT\r\n| Doc PR        | TODO\r\n\r\nI propose to add a `Parallelization` trait that we use internally for a while now to the core Console component.\r\n\r\n## What's it about\r\n\r\nThe `Parallelization` trait adds parallelization capabilities to Console commands in a very simple but efficient manner. These commands can then be launched with a `--processes` argument to determine in how many processes the command should be executed.\r\n\r\n## How to use\r\n\r\n1. Add the `Parallelization` trait to your command\r\n2. Implement `configure()` and call `configureParallelization()`\r\n3. Implement `fetchElements()` to return an iterable of strings (e.g. database IDs)\r\n4. Implement `runSingleCommand()` to process the workload for a single element\r\n5. Implement `getElementName()` to return the human readable name of one element\r\n\r\n## Example (simple)\r\n\r\n```php\r\nclass ImportContactsCommand\r\n{\r\n    use Parallelization;\r\n\r\n    protected function configure()\r\n    {\r\n        $this->setName('import:contacts');\r\n\r\n        $this->configureParallelization();\r\n    }\r\n\r\n    protected function fetchElements(InputInterface $input): iterable\r\n    {\r\n        // I am executed in the master process\r\n        $sheet = $this->openExcelSheet();\r\n        \r\n        foreach ($sheet as $row) {\r\n            yield serialize($row);\r\n        }\r\n    }\r\n\r\n    protected function runSingleCommand(string $element, InputInterface $input, OutputInterface $output)\r\n    {\r\n        // I am executed in the child process when using parallelization\r\n        // I am executed in the master process when not using parallelization\r\n        $row = unserialize($row);\r\n        \r\n        $this->importContact($row);\r\n    }\r\n\r\n    protected function getElementName(int $count)\r\n    {\r\n        return 1 === $count ? 'contact' : 'contacts';\r\n    }\r\n\r\n    // ...\r\n}\r\n```\r\n\r\nBy default, this command is executed in a single processes like a regular Symfony command. However, it can be changed to use multiple processes if that speeds up performance:\r\n\r\n```\r\nbin/console import:contacts --processes 4\r\n```\r\n\r\nNow the same workload is distributed among 4 processes.\r\n\r\n## Segments and batches\r\n\r\nWhen distributing a workload among processes, each process receives a *segment* of the processed data. The size of that segment can be configured by overriding `getSegmentSize()`. Depending on the task, different segment sizes may be optimal.\r\n\r\nWithin a process, the work is distributed in *batches*. You can override `runBeforeBatch()` or `runAfterBatch()` in order to prepare or finish the batch, for example in order to flush the database only after a batch has been completed.\r\n\r\nBy default, the batch size is the segment size, hence each process also processes a single batch. For very large segment sizes (e.g. 1000), it might make sense to reduce the batch size (e.g. 100) to improve memory usage and IO throughput. You can do that by overriding `getBatchSize()`.\r\n\r\n## Example (advanced)\r\n\r\n```php\r\nclass ImportContactsCommandWithBatchFlush\r\n{\r\n    use Parallelization;\r\n\r\n    protected function configure()\r\n    {\r\n        $this->setName('import:contacts');\r\n\r\n        $this->configureParallelization();\r\n    }\r\n\r\n    protected function fetchElements(InputInterface $input): iterable\r\n    {\r\n        $sheet = $this->openExcelSheet();\r\n\r\n        foreach ($sheet as $row) {\r\n            yield serialize($row);\r\n        }\r\n    }\r\n\r\n    protected function runSingleCommand(string $element, InputInterface $input, OutputInterface $output)\r\n    {\r\n        $em = $this->getContainer()->get('doctrine')->getManagerForClass(Contact::class);\r\n        \r\n        $row = unserialize($row);\r\n\r\n        $contact = $this->createContact($row);\r\n        \r\n        $em->persist($contact);\r\n    }\r\n    \r\n    protected function runAfterBatch(InputInterface $input, OutputInterface $output)\r\n    {\r\n        $em = $this->getContainer()->get('doctrine')->getManagerForClass(Contact::class);\r\n\r\n        // Persist in a batch\r\n        $em->flush();\r\n        $em->clear();\r\n    }\r\n\r\n    protected function getElementName(int $count)\r\n    {\r\n        return 1 === $count ? 'contact' : 'contacts';\r\n    }\r\n}\r\n```\r\n\r\n## Hooks\r\n\r\n* `runBeforeFirstCommand()` - executed in the master process at the very beginning\r\n* `runAfterLastCommand()` - executed in the master process at the very beginning\r\n* `runBeforeBatch()` - executed in the child process before a batch\r\n* `runAfterBatch()` - executed in the child process after a batch\r\n\r\n## Pending decision\r\n\r\nBefore we proceed with CS details, implementation feedback etc.: Is there any interest to add this to core?"},"comment":{"url":"https://api.github.com/repos/symfony/symfony/issues/comments/400946134","html_url":"https://github.com/symfony/symfony/pull/27518#issuecomment-400946134","issue_url":"https://api.github.com/repos/symfony/symfony/issues/27518","id":400946134,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMDk0NjEzNA==","user":{"login":"theofidry","id":5175937,"node_id":"MDQ6VXNlcjUxNzU5Mzc=","avatar_url":"https://avatars3.githubusercontent.com/u/5175937?v=4","gravatar_id":"","url":"https://api.github.com/users/theofidry","html_url":"https://github.com/theofidry","followers_url":"https://api.github.com/users/theofidry/followers","following_url":"https://api.github.com/users/theofidry/following{/other_user}","gists_url":"https://api.github.com/users/theofidry/gists{/gist_id}","starred_url":"https://api.github.com/users/theofidry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/theofidry/subscriptions","organizations_url":"https://api.github.com/users/theofidry/orgs","repos_url":"https://api.github.com/users/theofidry/repos","events_url":"https://api.github.com/users/theofidry/events{/privacy}","received_events_url":"https://api.github.com/users/theofidry/received_events","type":"User","site_admin":false},"created_at":"2018-06-28T07:54:27Z","updated_at":"2018-06-28T07:54:27Z","author_association":"CONTRIBUTOR","body":"@nicolas-grekas IMO the use case is not that rare, proof is a few people liking the feature and it's also not the first time this feature has been requested. I also know a couple of people that quite like it but never submitted anything because the feature is far from being trivial and they didn't have the confidence to propose a PR.\r\n\r\nThe issues with xargs and parallel:\r\n\r\n- It doesn't work on Windows\r\n- It requires you to have two commands: one to provide the arguments and a second to process it\r\n- You get the bootstrapping process as an overhead for each element since it's no longer a long-living process unless you add more stuff to have the processing command acting as a server\r\n\r\nThe major benefit of it is that the best alternative right now would be to leverage queues, but in a lot of cases it introduces a lot of complexity and you might not have queues at all in your app in the first place.\r\n\r\nBut that's only my opinion about the feature, not so much about the implementation :)"}}
{"actor":{"display_login":"theofidry"}}