{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/24817","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/24817/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/24817/comments","events_url":"https://api.github.com/repos/apache/spark/issues/24817/events","html_url":"https://github.com/apache/spark/pull/24817","id":453235922,"node_id":"MDExOlB1bGxSZXF1ZXN0Mjg1OTY3NTUy","number":24817,"title":"[SPARK-27963][core] Allow dynamic allocation without a shuffle service.","user":{"login":"vanzin","id":1694083,"node_id":"MDQ6VXNlcjE2OTQwODM=","avatar_url":"https://avatars0.githubusercontent.com/u/1694083?v=4","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"labels":[{"id":1405801482,"node_id":"MDU6TGFiZWwxNDA1ODAxNDgy","url":"https://api.github.com/repos/apache/spark/labels/SPARK%20CORE","name":"SPARK CORE","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2019-06-06T20:59:33Z","updated_at":"2019-06-28T18:36:46Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/24817","html_url":"https://github.com/apache/spark/pull/24817","diff_url":"https://github.com/apache/spark/pull/24817.diff","patch_url":"https://github.com/apache/spark/pull/24817.patch"},"body":"This change adds a new option that enables dynamic allocation without\r\nthe need for a shuffle service. This mode works by tracking which stages\r\ngenerate shuffle files, and keeping executors that generate data for those\r\nshuffles alive while the jobs that use them are active.\r\n\r\nA separate timeout is also added for shuffle data; so that executors that\r\nhold shuffle data can use a separate timeout before being removed because\r\nof being idle. This allows the shuffle data to be kept around in case it\r\nis needed by some new job, or allow users to be more aggressive in timing\r\nout executors that don't have shuffle data in active use.\r\n\r\nThe code also hooks up to the context cleaner so that shuffles that are\r\ngarbage collected are detected, and the respective executors not held\r\nunnecessarily.\r\n\r\nTesting done with added unit tests, and also with TPC-DS workloads on\r\nYARN without a shuffle service."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/506836767","html_url":"https://github.com/apache/spark/pull/24817#issuecomment-506836767","issue_url":"https://api.github.com/repos/apache/spark/issues/24817","id":506836767,"node_id":"MDEyOklzc3VlQ29tbWVudDUwNjgzNjc2Nw==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2019-06-28T18:36:46Z","updated_at":"2019-06-28T18:36:46Z","author_association":"NONE","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12214/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}