{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/mmlspark/issues/296","repository_url":"https://api.github.com/repos/Azure/mmlspark","labels_url":"https://api.github.com/repos/Azure/mmlspark/issues/296/labels{/name}","comments_url":"https://api.github.com/repos/Azure/mmlspark/issues/296/comments","events_url":"https://api.github.com/repos/Azure/mmlspark/issues/296/events","html_url":"https://github.com/Azure/mmlspark/pull/296","id":318911660,"number":296,"title":"Improving performance of LightGBM for cluster with multiple cores per executor","user":{"login":"imatiach-msft","id":24683184,"avatar_url":"https://avatars0.githubusercontent.com/u/24683184?v=4","gravatar_id":"","url":"https://api.github.com/users/imatiach-msft","html_url":"https://github.com/imatiach-msft","followers_url":"https://api.github.com/users/imatiach-msft/followers","following_url":"https://api.github.com/users/imatiach-msft/following{/other_user}","gists_url":"https://api.github.com/users/imatiach-msft/gists{/gist_id}","starred_url":"https://api.github.com/users/imatiach-msft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/imatiach-msft/subscriptions","organizations_url":"https://api.github.com/users/imatiach-msft/orgs","repos_url":"https://api.github.com/users/imatiach-msft/repos","events_url":"https://api.github.com/users/imatiach-msft/events{/privacy}","received_events_url":"https://api.github.com/users/imatiach-msft/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"imatiach-msft","id":24683184,"avatar_url":"https://avatars0.githubusercontent.com/u/24683184?v=4","gravatar_id":"","url":"https://api.github.com/users/imatiach-msft","html_url":"https://github.com/imatiach-msft","followers_url":"https://api.github.com/users/imatiach-msft/followers","following_url":"https://api.github.com/users/imatiach-msft/following{/other_user}","gists_url":"https://api.github.com/users/imatiach-msft/gists{/gist_id}","starred_url":"https://api.github.com/users/imatiach-msft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/imatiach-msft/subscriptions","organizations_url":"https://api.github.com/users/imatiach-msft/orgs","repos_url":"https://api.github.com/users/imatiach-msft/repos","events_url":"https://api.github.com/users/imatiach-msft/events{/privacy}","received_events_url":"https://api.github.com/users/imatiach-msft/received_events","type":"User","site_admin":false},"assignees":[{"login":"imatiach-msft","id":24683184,"avatar_url":"https://avatars0.githubusercontent.com/u/24683184?v=4","gravatar_id":"","url":"https://api.github.com/users/imatiach-msft","html_url":"https://github.com/imatiach-msft","followers_url":"https://api.github.com/users/imatiach-msft/followers","following_url":"https://api.github.com/users/imatiach-msft/following{/other_user}","gists_url":"https://api.github.com/users/imatiach-msft/gists{/gist_id}","starred_url":"https://api.github.com/users/imatiach-msft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/imatiach-msft/subscriptions","organizations_url":"https://api.github.com/users/imatiach-msft/orgs","repos_url":"https://api.github.com/users/imatiach-msft/repos","events_url":"https://api.github.com/users/imatiach-msft/events{/privacy}","received_events_url":"https://api.github.com/users/imatiach-msft/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2018-04-30T14:32:37Z","updated_at":"2018-05-01T03:59:16Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/Azure/mmlspark/pulls/296","html_url":"https://github.com/Azure/mmlspark/pull/296","diff_url":"https://github.com/Azure/mmlspark/pull/296.diff","patch_url":"https://github.com/Azure/mmlspark/pull/296.patch"},"body":"We now determine the number of CPUs per executor (where 1 executor = 1 JVM).\r\nOn the driver, we start a separate thread prior to calling LightGBM train which waits to receive the (host,port) from (# executors X # cores) workers.\r\nPrior to network init, on each of the workers we allocate a port and send it to the driver thread.\r\nThe driver thread receives all ports, aggregates, and then sends back the result so it can be used for network init.\r\nThis resolves several issues - \r\n1.) we ensure that the ports selected are open (we find only \"open\" ports and bind to them)\r\n2.) we discover the number of CPUs per executor (per JVM) and use that - and all workers now request a port\r\n3.) better logging to detect problems\r\n\r\nThis resolves issue: https://github.com/Azure/mmlspark/issues/292\r\n"},"comment":{"url":"https://api.github.com/repos/Azure/mmlspark/issues/comments/385593821","html_url":"https://github.com/Azure/mmlspark/pull/296#issuecomment-385593821","issue_url":"https://api.github.com/repos/Azure/mmlspark/issues/296","id":385593821,"user":{"login":"mmlspark-bot","id":29117994,"avatar_url":"https://avatars1.githubusercontent.com/u/29117994?v=4","gravatar_id":"","url":"https://api.github.com/users/mmlspark-bot","html_url":"https://github.com/mmlspark-bot","followers_url":"https://api.github.com/users/mmlspark-bot/followers","following_url":"https://api.github.com/users/mmlspark-bot/following{/other_user}","gists_url":"https://api.github.com/users/mmlspark-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/mmlspark-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mmlspark-bot/subscriptions","organizations_url":"https://api.github.com/users/mmlspark-bot/orgs","repos_url":"https://api.github.com/users/mmlspark-bot/repos","events_url":"https://api.github.com/users/mmlspark-bot/events{/privacy}","received_events_url":"https://api.github.com/users/mmlspark-bot/received_events","type":"User","site_admin":false},"created_at":"2018-05-01T03:59:16Z","updated_at":"2018-05-01T03:59:16Z","author_association":"COLLABORATOR","body":"[A build has started.  (5155f289)](https://msdata.visualstudio.com/Vienna/_build/index?buildId=865696&_a=summary)"}}
{"actor":{"display_login":"mmlspark-bot"}}