{"action":"closed","number":2378,"pull_request":{"url":"https://api.github.com/repos/spotify/luigi/pulls/2378","id":175755766,"html_url":"https://github.com/spotify/luigi/pull/2378","diff_url":"https://github.com/spotify/luigi/pull/2378.diff","patch_url":"https://github.com/spotify/luigi/pull/2378.patch","issue_url":"https://api.github.com/repos/spotify/luigi/issues/2378","number":2378,"state":"closed","locked":false,"title":"Add option for tasks to run on main process","user":{"login":"naor2013","id":5496913,"avatar_url":"https://avatars1.githubusercontent.com/u/5496913?v=4","gravatar_id":"","url":"https://api.github.com/users/naor2013","html_url":"https://github.com/naor2013","followers_url":"https://api.github.com/users/naor2013/followers","following_url":"https://api.github.com/users/naor2013/following{/other_user}","gists_url":"https://api.github.com/users/naor2013/gists{/gist_id}","starred_url":"https://api.github.com/users/naor2013/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/naor2013/subscriptions","organizations_url":"https://api.github.com/users/naor2013/orgs","repos_url":"https://api.github.com/users/naor2013/repos","events_url":"https://api.github.com/users/naor2013/events{/privacy}","received_events_url":"https://api.github.com/users/naor2013/received_events","type":"User","site_admin":false},"body":"<!--- This template is optional. Please use it as a starting point to help guide PRs -->\r\n\r\n<!--- Provide a general summary of your changes in the Title above -->\r\n\r\n## Description\r\n<!--- Describe your changes -->\r\nWhen using multiple workers, the main process is used for the scheduler and the tasks run on the rest of the cores. If you're using a function that utilizes multiple cores, running that function not from the main core can cause some issues. So in some cases, you want your workflow to run on multiple cores except from one task that you want to run on the main core. I added an open to add a property to your task \"run_on_main_process\" which you can set to true (default is false) and using that will force that task to run on the main process.\r\n\r\n## Motivation and Context\r\n<!--- Why is this change required? What problem does it solve? -->\r\n<!--- If it fixes an open issue, please link to the issue here. -->\r\nUsing functions from libraries that use multiple cores (like Scikit-Learn) will have unexpected outcomes when running from a core that's not the main core. Adding that property will solve that issue.\r\n\r\n## Have you tested this? If so, how?\r\n<!--- Valid responses are \"I have included unit tests.\" or --> \r\n<!--- \"I ran my jobs with this code and it works for me.\" -->\r\nI tried it and it worked. Plus, I only changed 1 line of core code with that so I'm not sure how to add unit tests for it (Would love for some help with that if needed).\r\n\r\n<!---\r\nfor more information on how to submit valuable contributions,\r\nsee https://opensource.guide/how-to-contribute/#how-to-submit-a-contribution\r\n-->\r\n","created_at":"2018-03-18T15:06:28Z","updated_at":"2018-05-13T06:42:58Z","closed_at":"2018-05-13T06:42:58Z","merged_at":null,"merge_commit_sha":"f46e482a95456d4de3f30a6e887c35449f3506aa","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/spotify/luigi/pulls/2378/commits","review_comments_url":"https://api.github.com/repos/spotify/luigi/pulls/2378/comments","review_comment_url":"https://api.github.com/repos/spotify/luigi/pulls/comments{/number}","comments_url":"https://api.github.com/repos/spotify/luigi/issues/2378/comments","statuses_url":"https://api.github.com/repos/spotify/luigi/statuses/6acbd61c2583cb535fbaecbb1e4aba91b5954878","head":{"label":"naor2013:master","ref":"master","sha":"6acbd61c2583cb535fbaecbb1e4aba91b5954878","user":{"login":"naor2013","id":5496913,"avatar_url":"https://avatars1.githubusercontent.com/u/5496913?v=4","gravatar_id":"","url":"https://api.github.com/users/naor2013","html_url":"https://github.com/naor2013","followers_url":"https://api.github.com/users/naor2013/followers","following_url":"https://api.github.com/users/naor2013/following{/other_user}","gists_url":"https://api.github.com/users/naor2013/gists{/gist_id}","starred_url":"https://api.github.com/users/naor2013/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/naor2013/subscriptions","organizations_url":"https://api.github.com/users/naor2013/orgs","repos_url":"https://api.github.com/users/naor2013/repos","events_url":"https://api.github.com/users/naor2013/events{/privacy}","received_events_url":"https://api.github.com/users/naor2013/received_events","type":"User","site_admin":false},"repo":{"id":125735817,"name":"luigi","full_name":"naor2013/luigi","owner":{"login":"naor2013","id":5496913,"avatar_url":"https://avatars1.githubusercontent.com/u/5496913?v=4","gravatar_id":"","url":"https://api.github.com/users/naor2013","html_url":"https://github.com/naor2013","followers_url":"https://api.github.com/users/naor2013/followers","following_url":"https://api.github.com/users/naor2013/following{/other_user}","gists_url":"https://api.github.com/users/naor2013/gists{/gist_id}","starred_url":"https://api.github.com/users/naor2013/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/naor2013/subscriptions","organizations_url":"https://api.github.com/users/naor2013/orgs","repos_url":"https://api.github.com/users/naor2013/repos","events_url":"https://api.github.com/users/naor2013/events{/privacy}","received_events_url":"https://api.github.com/users/naor2013/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/naor2013/luigi","description":"Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in. ","fork":true,"url":"https://api.github.com/repos/naor2013/luigi","forks_url":"https://api.github.com/repos/naor2013/luigi/forks","keys_url":"https://api.github.com/repos/naor2013/luigi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/naor2013/luigi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/naor2013/luigi/teams","hooks_url":"https://api.github.com/repos/naor2013/luigi/hooks","issue_events_url":"https://api.github.com/repos/naor2013/luigi/issues/events{/number}","events_url":"https://api.github.com/repos/naor2013/luigi/events","assignees_url":"https://api.github.com/repos/naor2013/luigi/assignees{/user}","branches_url":"https://api.github.com/repos/naor2013/luigi/branches{/branch}","tags_url":"https://api.github.com/repos/naor2013/luigi/tags","blobs_url":"https://api.github.com/repos/naor2013/luigi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/naor2013/luigi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/naor2013/luigi/git/refs{/sha}","trees_url":"https://api.github.com/repos/naor2013/luigi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/naor2013/luigi/statuses/{sha}","languages_url":"https://api.github.com/repos/naor2013/luigi/languages","stargazers_url":"https://api.github.com/repos/naor2013/luigi/stargazers","contributors_url":"https://api.github.com/repos/naor2013/luigi/contributors","subscribers_url":"https://api.github.com/repos/naor2013/luigi/subscribers","subscription_url":"https://api.github.com/repos/naor2013/luigi/subscription","commits_url":"https://api.github.com/repos/naor2013/luigi/commits{/sha}","git_commits_url":"https://api.github.com/repos/naor2013/luigi/git/commits{/sha}","comments_url":"https://api.github.com/repos/naor2013/luigi/comments{/number}","issue_comment_url":"https://api.github.com/repos/naor2013/luigi/issues/comments{/number}","contents_url":"https://api.github.com/repos/naor2013/luigi/contents/{+path}","compare_url":"https://api.github.com/repos/naor2013/luigi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/naor2013/luigi/merges","archive_url":"https://api.github.com/repos/naor2013/luigi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/naor2013/luigi/downloads","issues_url":"https://api.github.com/repos/naor2013/luigi/issues{/number}","pulls_url":"https://api.github.com/repos/naor2013/luigi/pulls{/number}","milestones_url":"https://api.github.com/repos/naor2013/luigi/milestones{/number}","notifications_url":"https://api.github.com/repos/naor2013/luigi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/naor2013/luigi/labels{/name}","releases_url":"https://api.github.com/repos/naor2013/luigi/releases{/id}","deployments_url":"https://api.github.com/repos/naor2013/luigi/deployments","created_at":"2018-03-18T14:55:10Z","updated_at":"2018-03-18T14:58:45Z","pushed_at":"2018-03-18T14:58:41Z","git_url":"git://github.com/naor2013/luigi.git","ssh_url":"git@github.com:naor2013/luigi.git","clone_url":"https://github.com/naor2013/luigi.git","svn_url":"https://github.com/naor2013/luigi","homepage":null,"size":9055,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"spotify:master","ref":"master","sha":"3cf763dc9ad1b1f5e84e9b20428198b23b0861e3","user":{"login":"spotify","id":251374,"avatar_url":"https://avatars2.githubusercontent.com/u/251374?v=4","gravatar_id":"","url":"https://api.github.com/users/spotify","html_url":"https://github.com/spotify","followers_url":"https://api.github.com/users/spotify/followers","following_url":"https://api.github.com/users/spotify/following{/other_user}","gists_url":"https://api.github.com/users/spotify/gists{/gist_id}","starred_url":"https://api.github.com/users/spotify/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spotify/subscriptions","organizations_url":"https://api.github.com/users/spotify/orgs","repos_url":"https://api.github.com/users/spotify/repos","events_url":"https://api.github.com/users/spotify/events{/privacy}","received_events_url":"https://api.github.com/users/spotify/received_events","type":"Organization","site_admin":false},"repo":{"id":5888353,"name":"luigi","full_name":"spotify/luigi","owner":{"login":"spotify","id":251374,"avatar_url":"https://avatars2.githubusercontent.com/u/251374?v=4","gravatar_id":"","url":"https://api.github.com/users/spotify","html_url":"https://github.com/spotify","followers_url":"https://api.github.com/users/spotify/followers","following_url":"https://api.github.com/users/spotify/following{/other_user}","gists_url":"https://api.github.com/users/spotify/gists{/gist_id}","starred_url":"https://api.github.com/users/spotify/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spotify/subscriptions","organizations_url":"https://api.github.com/users/spotify/orgs","repos_url":"https://api.github.com/users/spotify/repos","events_url":"https://api.github.com/users/spotify/events{/privacy}","received_events_url":"https://api.github.com/users/spotify/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/spotify/luigi","description":"Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in. ","fork":false,"url":"https://api.github.com/repos/spotify/luigi","forks_url":"https://api.github.com/repos/spotify/luigi/forks","keys_url":"https://api.github.com/repos/spotify/luigi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/spotify/luigi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/spotify/luigi/teams","hooks_url":"https://api.github.com/repos/spotify/luigi/hooks","issue_events_url":"https://api.github.com/repos/spotify/luigi/issues/events{/number}","events_url":"https://api.github.com/repos/spotify/luigi/events","assignees_url":"https://api.github.com/repos/spotify/luigi/assignees{/user}","branches_url":"https://api.github.com/repos/spotify/luigi/branches{/branch}","tags_url":"https://api.github.com/repos/spotify/luigi/tags","blobs_url":"https://api.github.com/repos/spotify/luigi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/spotify/luigi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/spotify/luigi/git/refs{/sha}","trees_url":"https://api.github.com/repos/spotify/luigi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/spotify/luigi/statuses/{sha}","languages_url":"https://api.github.com/repos/spotify/luigi/languages","stargazers_url":"https://api.github.com/repos/spotify/luigi/stargazers","contributors_url":"https://api.github.com/repos/spotify/luigi/contributors","subscribers_url":"https://api.github.com/repos/spotify/luigi/subscribers","subscription_url":"https://api.github.com/repos/spotify/luigi/subscription","commits_url":"https://api.github.com/repos/spotify/luigi/commits{/sha}","git_commits_url":"https://api.github.com/repos/spotify/luigi/git/commits{/sha}","comments_url":"https://api.github.com/repos/spotify/luigi/comments{/number}","issue_comment_url":"https://api.github.com/repos/spotify/luigi/issues/comments{/number}","contents_url":"https://api.github.com/repos/spotify/luigi/contents/{+path}","compare_url":"https://api.github.com/repos/spotify/luigi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/spotify/luigi/merges","archive_url":"https://api.github.com/repos/spotify/luigi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/spotify/luigi/downloads","issues_url":"https://api.github.com/repos/spotify/luigi/issues{/number}","pulls_url":"https://api.github.com/repos/spotify/luigi/pulls{/number}","milestones_url":"https://api.github.com/repos/spotify/luigi/milestones{/number}","notifications_url":"https://api.github.com/repos/spotify/luigi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/spotify/luigi/labels{/name}","releases_url":"https://api.github.com/repos/spotify/luigi/releases{/id}","deployments_url":"https://api.github.com/repos/spotify/luigi/deployments","created_at":"2012-09-20T15:06:38Z","updated_at":"2018-05-13T03:59:20Z","pushed_at":"2018-05-10T18:39:45Z","git_url":"git://github.com/spotify/luigi.git","ssh_url":"git@github.com:spotify/luigi.git","clone_url":"https://github.com/spotify/luigi.git","svn_url":"https://github.com/spotify/luigi","homepage":null,"size":8410,"stargazers_count":9192,"watchers_count":9192,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1625,"mirror_url":null,"archived":false,"open_issues_count":243,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":1625,"open_issues":243,"watchers":9192,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/spotify/luigi/pulls/2378"},"html":{"href":"https://github.com/spotify/luigi/pull/2378"},"issue":{"href":"https://api.github.com/repos/spotify/luigi/issues/2378"},"comments":{"href":"https://api.github.com/repos/spotify/luigi/issues/2378/comments"},"review_comments":{"href":"https://api.github.com/repos/spotify/luigi/pulls/2378/comments"},"review_comment":{"href":"https://api.github.com/repos/spotify/luigi/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/spotify/luigi/pulls/2378/commits"},"statuses":{"href":"https://api.github.com/repos/spotify/luigi/statuses/6acbd61c2583cb535fbaecbb1e4aba91b5954878"}},"author_association":"NONE","merged":false,"mergeable":true,"rebaseable":true,"mergeable_state":"clean","merged_by":null,"comments":0,"review_comments":1,"maintainer_can_modify":false,"commits":1,"additions":5,"deletions":1,"changed_files":2}}
{"actor":{"display_login":"naor2013"}}