{"action":"created","comment":{"url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/comments/86050224","pull_request_review_id":6728646,"id":86050224,"diff_hunk":"@@ -34,6 +35,12 @@ import (\n \tkube_labels \"k8s.io/kubernetes/pkg/labels\"\n )\n \n+const (\n+\t// MaxPodsPerChunk is the maximum number of pods we can schedule for each\n+\t// chunk in case of failures.\n+\tMaxPodsPerChunk = 3","path":"src/server/pps/server/api_server.go","position":15,"original_position":15,"commit_id":"a39005ac7ee4af21039a369980832cddaae0c6a8","original_commit_id":"a39005ac7ee4af21039a369980832cddaae0c6a8","user":{"login":"derekchiang","id":1535686,"avatar_url":"https://avatars.githubusercontent.com/u/1535686?v=3","gravatar_id":"","url":"https://api.github.com/users/derekchiang","html_url":"https://github.com/derekchiang","followers_url":"https://api.github.com/users/derekchiang/followers","following_url":"https://api.github.com/users/derekchiang/following{/other_user}","gists_url":"https://api.github.com/users/derekchiang/gists{/gist_id}","starred_url":"https://api.github.com/users/derekchiang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekchiang/subscriptions","organizations_url":"https://api.github.com/users/derekchiang/orgs","repos_url":"https://api.github.com/users/derekchiang/repos","events_url":"https://api.github.com/users/derekchiang/events{/privacy}","received_events_url":"https://api.github.com/users/derekchiang/received_events","type":"User","site_admin":false},"body":"Agreed. We can do that in a separate PR.","created_at":"2016-11-01T23:54:44Z","updated_at":"2016-11-01T23:54:44Z","html_url":"https://github.com/pachyderm/pachyderm/pull/1022#discussion_r86050224","pull_request_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/1022","_links":{"self":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/comments/86050224"},"html":{"href":"https://github.com/pachyderm/pachyderm/pull/1022#discussion_r86050224"},"pull_request":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/1022"}}},"pull_request":{"url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/1022","id":91694097,"html_url":"https://github.com/pachyderm/pachyderm/pull/1022","diff_url":"https://github.com/pachyderm/pachyderm/pull/1022.diff","patch_url":"https://github.com/pachyderm/pachyderm/pull/1022.patch","issue_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/1022","number":1022,"state":"open","locked":false,"title":"Improve PPS's fault tolerance","user":{"login":"derekchiang","id":1535686,"avatar_url":"https://avatars.githubusercontent.com/u/1535686?v=3","gravatar_id":"","url":"https://api.github.com/users/derekchiang","html_url":"https://github.com/derekchiang","followers_url":"https://api.github.com/users/derekchiang/followers","following_url":"https://api.github.com/users/derekchiang/following{/other_user}","gists_url":"https://api.github.com/users/derekchiang/gists{/gist_id}","starred_url":"https://api.github.com/users/derekchiang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekchiang/subscriptions","organizations_url":"https://api.github.com/users/derekchiang/orgs","repos_url":"https://api.github.com/users/derekchiang/repos","events_url":"https://api.github.com/users/derekchiang/events{/privacy}","received_events_url":"https://api.github.com/users/derekchiang/received_events","type":"User","site_admin":false},"body":"This PR revamps PPS's job execution model.  The end results are:\r\n\r\n* Jobs can now survive crashes (i.e. failures that are not explicitly reported to PPS), including pod crashes and node crashes.\r\n* Jobs are more resilient to transient failures: when the user container fails, PPS reschedules the container for up to 3 times before giving up.\r\n* We have a good foundation for future work such as 1) having each pod work on multiple chunks of data, 2) straggler detection and mitigation (#442), 3) work recovery from failed jobs (#606) etc.\r\n\r\nThis PR seems to fix #762 and #905, and partially addresses #990.  I might be missing a couple more.","created_at":"2016-10-31T21:28:16Z","updated_at":"2016-11-01T23:54:44Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6a0bd21b6d325268d8a7ae16a0ab152647e10abf","assignee":{"login":"jdoliner","id":43867,"avatar_url":"https://avatars.githubusercontent.com/u/43867?v=3","gravatar_id":"","url":"https://api.github.com/users/jdoliner","html_url":"https://github.com/jdoliner","followers_url":"https://api.github.com/users/jdoliner/followers","following_url":"https://api.github.com/users/jdoliner/following{/other_user}","gists_url":"https://api.github.com/users/jdoliner/gists{/gist_id}","starred_url":"https://api.github.com/users/jdoliner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdoliner/subscriptions","organizations_url":"https://api.github.com/users/jdoliner/orgs","repos_url":"https://api.github.com/users/jdoliner/repos","events_url":"https://api.github.com/users/jdoliner/events{/privacy}","received_events_url":"https://api.github.com/users/jdoliner/received_events","type":"User","site_admin":false},"assignees":[{"login":"jdoliner","id":43867,"avatar_url":"https://avatars.githubusercontent.com/u/43867?v=3","gravatar_id":"","url":"https://api.github.com/users/jdoliner","html_url":"https://github.com/jdoliner","followers_url":"https://api.github.com/users/jdoliner/followers","following_url":"https://api.github.com/users/jdoliner/following{/other_user}","gists_url":"https://api.github.com/users/jdoliner/gists{/gist_id}","starred_url":"https://api.github.com/users/jdoliner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdoliner/subscriptions","organizations_url":"https://api.github.com/users/jdoliner/orgs","repos_url":"https://api.github.com/users/jdoliner/repos","events_url":"https://api.github.com/users/jdoliner/events{/privacy}","received_events_url":"https://api.github.com/users/jdoliner/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/1022/commits","review_comments_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/1022/comments","review_comment_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/comments{/number}","comments_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/1022/comments","statuses_url":"https://api.github.com/repos/pachyderm/pachyderm/statuses/a39005ac7ee4af21039a369980832cddaae0c6a8","head":{"label":"pachyderm:pps-fault-tolerance","ref":"pps-fault-tolerance","sha":"a39005ac7ee4af21039a369980832cddaae0c6a8","user":{"login":"pachyderm","id":10432478,"avatar_url":"https://avatars.githubusercontent.com/u/10432478?v=3","gravatar_id":"","url":"https://api.github.com/users/pachyderm","html_url":"https://github.com/pachyderm","followers_url":"https://api.github.com/users/pachyderm/followers","following_url":"https://api.github.com/users/pachyderm/following{/other_user}","gists_url":"https://api.github.com/users/pachyderm/gists{/gist_id}","starred_url":"https://api.github.com/users/pachyderm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pachyderm/subscriptions","organizations_url":"https://api.github.com/users/pachyderm/orgs","repos_url":"https://api.github.com/users/pachyderm/repos","events_url":"https://api.github.com/users/pachyderm/events{/privacy}","received_events_url":"https://api.github.com/users/pachyderm/received_events","type":"Organization","site_admin":false},"repo":{"id":23653453,"name":"pachyderm","full_name":"pachyderm/pachyderm","owner":{"login":"pachyderm","id":10432478,"avatar_url":"https://avatars.githubusercontent.com/u/10432478?v=3","gravatar_id":"","url":"https://api.github.com/users/pachyderm","html_url":"https://github.com/pachyderm","followers_url":"https://api.github.com/users/pachyderm/followers","following_url":"https://api.github.com/users/pachyderm/following{/other_user}","gists_url":"https://api.github.com/users/pachyderm/gists{/gist_id}","starred_url":"https://api.github.com/users/pachyderm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pachyderm/subscriptions","organizations_url":"https://api.github.com/users/pachyderm/orgs","repos_url":"https://api.github.com/users/pachyderm/repos","events_url":"https://api.github.com/users/pachyderm/events{/privacy}","received_events_url":"https://api.github.com/users/pachyderm/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/pachyderm/pachyderm","description":"Containerized Data Analytics","fork":false,"url":"https://api.github.com/repos/pachyderm/pachyderm","forks_url":"https://api.github.com/repos/pachyderm/pachyderm/forks","keys_url":"https://api.github.com/repos/pachyderm/pachyderm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pachyderm/pachyderm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pachyderm/pachyderm/teams","hooks_url":"https://api.github.com/repos/pachyderm/pachyderm/hooks","issue_events_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/events{/number}","events_url":"https://api.github.com/repos/pachyderm/pachyderm/events","assignees_url":"https://api.github.com/repos/pachyderm/pachyderm/assignees{/user}","branches_url":"https://api.github.com/repos/pachyderm/pachyderm/branches{/branch}","tags_url":"https://api.github.com/repos/pachyderm/pachyderm/tags","blobs_url":"https://api.github.com/repos/pachyderm/pachyderm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pachyderm/pachyderm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pachyderm/pachyderm/git/refs{/sha}","trees_url":"https://api.github.com/repos/pachyderm/pachyderm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pachyderm/pachyderm/statuses/{sha}","languages_url":"https://api.github.com/repos/pachyderm/pachyderm/languages","stargazers_url":"https://api.github.com/repos/pachyderm/pachyderm/stargazers","contributors_url":"https://api.github.com/repos/pachyderm/pachyderm/contributors","subscribers_url":"https://api.github.com/repos/pachyderm/pachyderm/subscribers","subscription_url":"https://api.github.com/repos/pachyderm/pachyderm/subscription","commits_url":"https://api.github.com/repos/pachyderm/pachyderm/commits{/sha}","git_commits_url":"https://api.github.com/repos/pachyderm/pachyderm/git/commits{/sha}","comments_url":"https://api.github.com/repos/pachyderm/pachyderm/comments{/number}","issue_comment_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/comments{/number}","contents_url":"https://api.github.com/repos/pachyderm/pachyderm/contents/{+path}","compare_url":"https://api.github.com/repos/pachyderm/pachyderm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pachyderm/pachyderm/merges","archive_url":"https://api.github.com/repos/pachyderm/pachyderm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pachyderm/pachyderm/downloads","issues_url":"https://api.github.com/repos/pachyderm/pachyderm/issues{/number}","pulls_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls{/number}","milestones_url":"https://api.github.com/repos/pachyderm/pachyderm/milestones{/number}","notifications_url":"https://api.github.com/repos/pachyderm/pachyderm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pachyderm/pachyderm/labels{/name}","releases_url":"https://api.github.com/repos/pachyderm/pachyderm/releases{/id}","deployments_url":"https://api.github.com/repos/pachyderm/pachyderm/deployments","created_at":"2014-09-04T07:50:02Z","updated_at":"2016-10-31T19:35:06Z","pushed_at":"2016-11-01T22:44:19Z","git_url":"git://github.com/pachyderm/pachyderm.git","ssh_url":"git@github.com:pachyderm/pachyderm.git","clone_url":"https://github.com/pachyderm/pachyderm.git","svn_url":"https://github.com/pachyderm/pachyderm","homepage":"http://pachyderm.io","size":37281,"stargazers_count":1343,"watchers_count":1343,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":140,"mirror_url":null,"open_issues_count":197,"forks":140,"open_issues":197,"watchers":1343,"default_branch":"master"}},"base":{"label":"pachyderm:master","ref":"master","sha":"e336feec35765e8fac51ee04c2029717e3c2195b","user":{"login":"pachyderm","id":10432478,"avatar_url":"https://avatars.githubusercontent.com/u/10432478?v=3","gravatar_id":"","url":"https://api.github.com/users/pachyderm","html_url":"https://github.com/pachyderm","followers_url":"https://api.github.com/users/pachyderm/followers","following_url":"https://api.github.com/users/pachyderm/following{/other_user}","gists_url":"https://api.github.com/users/pachyderm/gists{/gist_id}","starred_url":"https://api.github.com/users/pachyderm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pachyderm/subscriptions","organizations_url":"https://api.github.com/users/pachyderm/orgs","repos_url":"https://api.github.com/users/pachyderm/repos","events_url":"https://api.github.com/users/pachyderm/events{/privacy}","received_events_url":"https://api.github.com/users/pachyderm/received_events","type":"Organization","site_admin":false},"repo":{"id":23653453,"name":"pachyderm","full_name":"pachyderm/pachyderm","owner":{"login":"pachyderm","id":10432478,"avatar_url":"https://avatars.githubusercontent.com/u/10432478?v=3","gravatar_id":"","url":"https://api.github.com/users/pachyderm","html_url":"https://github.com/pachyderm","followers_url":"https://api.github.com/users/pachyderm/followers","following_url":"https://api.github.com/users/pachyderm/following{/other_user}","gists_url":"https://api.github.com/users/pachyderm/gists{/gist_id}","starred_url":"https://api.github.com/users/pachyderm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pachyderm/subscriptions","organizations_url":"https://api.github.com/users/pachyderm/orgs","repos_url":"https://api.github.com/users/pachyderm/repos","events_url":"https://api.github.com/users/pachyderm/events{/privacy}","received_events_url":"https://api.github.com/users/pachyderm/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/pachyderm/pachyderm","description":"Containerized Data Analytics","fork":false,"url":"https://api.github.com/repos/pachyderm/pachyderm","forks_url":"https://api.github.com/repos/pachyderm/pachyderm/forks","keys_url":"https://api.github.com/repos/pachyderm/pachyderm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pachyderm/pachyderm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pachyderm/pachyderm/teams","hooks_url":"https://api.github.com/repos/pachyderm/pachyderm/hooks","issue_events_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/events{/number}","events_url":"https://api.github.com/repos/pachyderm/pachyderm/events","assignees_url":"https://api.github.com/repos/pachyderm/pachyderm/assignees{/user}","branches_url":"https://api.github.com/repos/pachyderm/pachyderm/branches{/branch}","tags_url":"https://api.github.com/repos/pachyderm/pachyderm/tags","blobs_url":"https://api.github.com/repos/pachyderm/pachyderm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pachyderm/pachyderm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pachyderm/pachyderm/git/refs{/sha}","trees_url":"https://api.github.com/repos/pachyderm/pachyderm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pachyderm/pachyderm/statuses/{sha}","languages_url":"https://api.github.com/repos/pachyderm/pachyderm/languages","stargazers_url":"https://api.github.com/repos/pachyderm/pachyderm/stargazers","contributors_url":"https://api.github.com/repos/pachyderm/pachyderm/contributors","subscribers_url":"https://api.github.com/repos/pachyderm/pachyderm/subscribers","subscription_url":"https://api.github.com/repos/pachyderm/pachyderm/subscription","commits_url":"https://api.github.com/repos/pachyderm/pachyderm/commits{/sha}","git_commits_url":"https://api.github.com/repos/pachyderm/pachyderm/git/commits{/sha}","comments_url":"https://api.github.com/repos/pachyderm/pachyderm/comments{/number}","issue_comment_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/comments{/number}","contents_url":"https://api.github.com/repos/pachyderm/pachyderm/contents/{+path}","compare_url":"https://api.github.com/repos/pachyderm/pachyderm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pachyderm/pachyderm/merges","archive_url":"https://api.github.com/repos/pachyderm/pachyderm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pachyderm/pachyderm/downloads","issues_url":"https://api.github.com/repos/pachyderm/pachyderm/issues{/number}","pulls_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls{/number}","milestones_url":"https://api.github.com/repos/pachyderm/pachyderm/milestones{/number}","notifications_url":"https://api.github.com/repos/pachyderm/pachyderm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pachyderm/pachyderm/labels{/name}","releases_url":"https://api.github.com/repos/pachyderm/pachyderm/releases{/id}","deployments_url":"https://api.github.com/repos/pachyderm/pachyderm/deployments","created_at":"2014-09-04T07:50:02Z","updated_at":"2016-10-31T19:35:06Z","pushed_at":"2016-11-01T22:44:19Z","git_url":"git://github.com/pachyderm/pachyderm.git","ssh_url":"git@github.com:pachyderm/pachyderm.git","clone_url":"https://github.com/pachyderm/pachyderm.git","svn_url":"https://github.com/pachyderm/pachyderm","homepage":"http://pachyderm.io","size":37281,"stargazers_count":1343,"watchers_count":1343,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":140,"mirror_url":null,"open_issues_count":197,"forks":140,"open_issues":197,"watchers":1343,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/1022"},"html":{"href":"https://github.com/pachyderm/pachyderm/pull/1022"},"issue":{"href":"https://api.github.com/repos/pachyderm/pachyderm/issues/1022"},"comments":{"href":"https://api.github.com/repos/pachyderm/pachyderm/issues/1022/comments"},"review_comments":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/1022/comments"},"review_comment":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/1022/commits"},"statuses":{"href":"https://api.github.com/repos/pachyderm/pachyderm/statuses/a39005ac7ee4af21039a369980832cddaae0c6a8"}}}}
{"actor":{"display_login":"derekchiang"}}