{"action":"created","comment":{"url":"https://api.github.com/repos/mesosphere/marathon/pulls/comments/70640871","id":70640871,"diff_hunk":"@@ -0,0 +1,97 @@\n+package mesosphere.marathon.util.heartbeat","path":"src/main/scala/mesosphere/marathon/util/heartbeat/HeartbeatActor.scala","position":1,"original_position":1,"commit_id":"842019577090529a5687dc1b5d71f0366b35b4b0","original_commit_id":"842019577090529a5687dc1b5d71f0366b35b4b0","user":{"login":"jdef","id":2348332,"avatar_url":"https://avatars.githubusercontent.com/u/2348332?v=3","gravatar_id":"","url":"https://api.github.com/users/jdef","html_url":"https://github.com/jdef","followers_url":"https://api.github.com/users/jdef/followers","following_url":"https://api.github.com/users/jdef/following{/other_user}","gists_url":"https://api.github.com/users/jdef/gists{/gist_id}","starred_url":"https://api.github.com/users/jdef/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdef/subscriptions","organizations_url":"https://api.github.com/users/jdef/orgs","repos_url":"https://api.github.com/users/jdef/repos","events_url":"https://api.github.com/users/jdef/events{/privacy}","received_events_url":"https://api.github.com/users/jdef/received_events","type":"User","site_admin":false},"body":"The heartbeat actor really doesn't have any scheduler integration code, it's completely generic - so that's why it lives where it currently does. `MesosHeartbeatMonitor` is a `Scheduler` implementation and so .. since we don't have a proper `core` package for scheduler stuff yet, I just put it next to the other scheduler stuff.\r\n\r\nYou're the 4th person to suggest that I couple these classes more closely. Since it seems intuitive to the majority ... I'll create `core/heartbeat` to house these.","created_at":"2016-07-13T14:53:37Z","updated_at":"2016-07-13T14:53:37Z","html_url":"https://github.com/mesosphere/marathon/pull/4069#discussion_r70640871","pull_request_url":"https://api.github.com/repos/mesosphere/marathon/pulls/4069","_links":{"self":{"href":"https://api.github.com/repos/mesosphere/marathon/pulls/comments/70640871"},"html":{"href":"https://github.com/mesosphere/marathon/pull/4069#discussion_r70640871"},"pull_request":{"href":"https://api.github.com/repos/mesosphere/marathon/pulls/4069"}}},"pull_request":{"url":"https://api.github.com/repos/mesosphere/marathon/pulls/4069","id":76524174,"html_url":"https://github.com/mesosphere/marathon/pull/4069","diff_url":"https://github.com/mesosphere/marathon/pull/4069.diff","patch_url":"https://github.com/mesosphere/marathon/pull/4069.patch","issue_url":"https://api.github.com/repos/mesosphere/marathon/issues/4069","number":4069,"state":"open","locked":false,"title":"WIP: DO NOT MERGE: Virtual pacemaker using task reconciliation w/ fake task and agent","user":{"login":"jdef","id":2348332,"avatar_url":"https://avatars.githubusercontent.com/u/2348332?v=3","gravatar_id":"","url":"https://api.github.com/users/jdef","html_url":"https://github.com/jdef","followers_url":"https://api.github.com/users/jdef/followers","following_url":"https://api.github.com/users/jdef/following{/other_user}","gists_url":"https://api.github.com/users/jdef/gists{/gist_id}","starred_url":"https://api.github.com/users/jdef/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdef/subscriptions","organizations_url":"https://api.github.com/users/jdef/orgs","repos_url":"https://api.github.com/users/jdef/repos","events_url":"https://api.github.com/users/jdef/events{/privacy}","received_events_url":"https://api.github.com/users/jdef/received_events","type":"User","site_admin":false},"body":"Some context:\r\n* mesos clusters may suffer from periodic network partition conditions\r\n* scheduler.disconnected is not guaranteed to be fired under network partition conditions\r\n* marathon wants to be able to **reliably** detect network partitions (in particular, wants to detect disconnection from the leading mesos master) and enter a disconnected state until the partition is resolved\r\n* mesos v1 http API supports a heartbeat event to mitigate this\r\n  * this is the preferred, long term approach\r\n  * this approach requires a lot of work\r\n* we need a short term solution for this problem, one that is easily backported to prior releases w/ the intent to support customers using older versions\r\n\r\nShort Term Solution (this PR)\r\n* treat most scheduler callbacks as heartbeat \"pulse\" events\r\n* expect that a pulse event should occur within some timeout period/interval\r\n  * missing 1 pulse event is a **skip**\r\n  * missing X subsequent pulse events is a **failure**\r\n* upon **skip** generate a fake explicit task reconciliation request for a bogus task/agent; the leading master should respond immediately with a TASK_LOST. we'll see this as a pulse\r\n* upon **failure** disconnect from the master and await reconnection\r\n* it would be nice if some of this code could be reused for the long term solution (e.g. `HeartbeatActor`)\r\n\r\nCurrent design:\r\n* `MesosHeartbeatMonitor` is a `Scheduler` **decorator**\r\n  - generates `MessagePulse` for most callbacks\r\n  - reacts to **skip** and **failure** conditions\r\n* `HeartbeatActor`\r\n  - generic implementation of a monitor that watches for pulses and reacts when they don't happen as expected (by calling back to a `Reactor` impl)\r\n* `MarathonModules`\r\n  - Guice wiring for instantiating the decorated `Scheduler` impl and `HeartbeatActor`\r\n* `MarathonSchedulerService`\r\n  - deactivates the mesos heartbeat monitor upon `stopLeadership`\r\n\r\nNew command line options:\r\n* `--mesos_heartbeat_interval` :: in the absence of receiving a message from the mesos master during a time window of this duration, attempt to coerce mesos into communicating with marathon\r\n* `--mesos_heartbeat_failure_threshold` :: after missing this number of expected communications from the mesos master, infer that marathon has become disconnected from the master\r\n\r\nTask List\r\n- [x] initial implementation of virtual heartbeat monitor\r\n- [x] implement command line parameters that configure the heartbeat machinery\r\n- [x] command line parameter documentation\r\n- [x] unit testing\r\n- [ ] integration testing\r\n- [ ] ? avoid actually trying to process task status updates for the \"fake\" tasks? (we still want the pulse, but we can filter these from the underlying task update machinery)","created_at":"2016-07-07T00:43:16Z","updated_at":"2016-07-13T14:53:37Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bf7bde205a1700287e2495f6ed6bd34fabc46e5e","assignee":{"login":"jdef","id":2348332,"avatar_url":"https://avatars.githubusercontent.com/u/2348332?v=3","gravatar_id":"","url":"https://api.github.com/users/jdef","html_url":"https://github.com/jdef","followers_url":"https://api.github.com/users/jdef/followers","following_url":"https://api.github.com/users/jdef/following{/other_user}","gists_url":"https://api.github.com/users/jdef/gists{/gist_id}","starred_url":"https://api.github.com/users/jdef/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdef/subscriptions","organizations_url":"https://api.github.com/users/jdef/orgs","repos_url":"https://api.github.com/users/jdef/repos","events_url":"https://api.github.com/users/jdef/events{/privacy}","received_events_url":"https://api.github.com/users/jdef/received_events","type":"User","site_admin":false},"assignees":[{"login":"jdef","id":2348332,"avatar_url":"https://avatars.githubusercontent.com/u/2348332?v=3","gravatar_id":"","url":"https://api.github.com/users/jdef","html_url":"https://github.com/jdef","followers_url":"https://api.github.com/users/jdef/followers","following_url":"https://api.github.com/users/jdef/following{/other_user}","gists_url":"https://api.github.com/users/jdef/gists{/gist_id}","starred_url":"https://api.github.com/users/jdef/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdef/subscriptions","organizations_url":"https://api.github.com/users/jdef/orgs","repos_url":"https://api.github.com/users/jdef/repos","events_url":"https://api.github.com/users/jdef/events{/privacy}","received_events_url":"https://api.github.com/users/jdef/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/mesosphere/marathon/pulls/4069/commits","review_comments_url":"https://api.github.com/repos/mesosphere/marathon/pulls/4069/comments","review_comment_url":"https://api.github.com/repos/mesosphere/marathon/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mesosphere/marathon/issues/4069/comments","statuses_url":"https://api.github.com/repos/mesosphere/marathon/statuses/842019577090529a5687dc1b5d71f0366b35b4b0","head":{"label":"mesosphere:jdef_mesos_master_disconnected","ref":"jdef_mesos_master_disconnected","sha":"842019577090529a5687dc1b5d71f0366b35b4b0","user":{"login":"mesosphere","id":3458585,"avatar_url":"https://avatars.githubusercontent.com/u/3458585?v=3","gravatar_id":"","url":"https://api.github.com/users/mesosphere","html_url":"https://github.com/mesosphere","followers_url":"https://api.github.com/users/mesosphere/followers","following_url":"https://api.github.com/users/mesosphere/following{/other_user}","gists_url":"https://api.github.com/users/mesosphere/gists{/gist_id}","starred_url":"https://api.github.com/users/mesosphere/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mesosphere/subscriptions","organizations_url":"https://api.github.com/users/mesosphere/orgs","repos_url":"https://api.github.com/users/mesosphere/repos","events_url":"https://api.github.com/users/mesosphere/events{/privacy}","received_events_url":"https://api.github.com/users/mesosphere/received_events","type":"Organization","site_admin":false},"repo":{"id":10941409,"name":"marathon","full_name":"mesosphere/marathon","owner":{"login":"mesosphere","id":3458585,"avatar_url":"https://avatars.githubusercontent.com/u/3458585?v=3","gravatar_id":"","url":"https://api.github.com/users/mesosphere","html_url":"https://github.com/mesosphere","followers_url":"https://api.github.com/users/mesosphere/followers","following_url":"https://api.github.com/users/mesosphere/following{/other_user}","gists_url":"https://api.github.com/users/mesosphere/gists{/gist_id}","starred_url":"https://api.github.com/users/mesosphere/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mesosphere/subscriptions","organizations_url":"https://api.github.com/users/mesosphere/orgs","repos_url":"https://api.github.com/users/mesosphere/repos","events_url":"https://api.github.com/users/mesosphere/events{/privacy}","received_events_url":"https://api.github.com/users/mesosphere/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mesosphere/marathon","description":"Deploy and manage containers (including Docker) on top of Apache Mesos at scale.","fork":false,"url":"https://api.github.com/repos/mesosphere/marathon","forks_url":"https://api.github.com/repos/mesosphere/marathon/forks","keys_url":"https://api.github.com/repos/mesosphere/marathon/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mesosphere/marathon/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mesosphere/marathon/teams","hooks_url":"https://api.github.com/repos/mesosphere/marathon/hooks","issue_events_url":"https://api.github.com/repos/mesosphere/marathon/issues/events{/number}","events_url":"https://api.github.com/repos/mesosphere/marathon/events","assignees_url":"https://api.github.com/repos/mesosphere/marathon/assignees{/user}","branches_url":"https://api.github.com/repos/mesosphere/marathon/branches{/branch}","tags_url":"https://api.github.com/repos/mesosphere/marathon/tags","blobs_url":"https://api.github.com/repos/mesosphere/marathon/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mesosphere/marathon/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mesosphere/marathon/git/refs{/sha}","trees_url":"https://api.github.com/repos/mesosphere/marathon/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mesosphere/marathon/statuses/{sha}","languages_url":"https://api.github.com/repos/mesosphere/marathon/languages","stargazers_url":"https://api.github.com/repos/mesosphere/marathon/stargazers","contributors_url":"https://api.github.com/repos/mesosphere/marathon/contributors","subscribers_url":"https://api.github.com/repos/mesosphere/marathon/subscribers","subscription_url":"https://api.github.com/repos/mesosphere/marathon/subscription","commits_url":"https://api.github.com/repos/mesosphere/marathon/commits{/sha}","git_commits_url":"https://api.github.com/repos/mesosphere/marathon/git/commits{/sha}","comments_url":"https://api.github.com/repos/mesosphere/marathon/comments{/number}","issue_comment_url":"https://api.github.com/repos/mesosphere/marathon/issues/comments{/number}","contents_url":"https://api.github.com/repos/mesosphere/marathon/contents/{+path}","compare_url":"https://api.github.com/repos/mesosphere/marathon/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mesosphere/marathon/merges","archive_url":"https://api.github.com/repos/mesosphere/marathon/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mesosphere/marathon/downloads","issues_url":"https://api.github.com/repos/mesosphere/marathon/issues{/number}","pulls_url":"https://api.github.com/repos/mesosphere/marathon/pulls{/number}","milestones_url":"https://api.github.com/repos/mesosphere/marathon/milestones{/number}","notifications_url":"https://api.github.com/repos/mesosphere/marathon/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mesosphere/marathon/labels{/name}","releases_url":"https://api.github.com/repos/mesosphere/marathon/releases{/id}","deployments_url":"https://api.github.com/repos/mesosphere/marathon/deployments","created_at":"2013-06-25T14:56:34Z","updated_at":"2016-07-13T07:25:16Z","pushed_at":"2016-07-13T13:58:43Z","git_url":"git://github.com/mesosphere/marathon.git","ssh_url":"git@github.com:mesosphere/marathon.git","clone_url":"https://github.com/mesosphere/marathon.git","svn_url":"https://github.com/mesosphere/marathon","homepage":"https://mesosphere.github.io/marathon/","size":26971,"stargazers_count":2644,"watchers_count":2644,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":585,"mirror_url":null,"open_issues_count":610,"forks":585,"open_issues":610,"watchers":2644,"default_branch":"master"}},"base":{"label":"mesosphere:master","ref":"master","sha":"55e89fe441ffc6330cc61c04186d987994f495ef","user":{"login":"mesosphere","id":3458585,"avatar_url":"https://avatars.githubusercontent.com/u/3458585?v=3","gravatar_id":"","url":"https://api.github.com/users/mesosphere","html_url":"https://github.com/mesosphere","followers_url":"https://api.github.com/users/mesosphere/followers","following_url":"https://api.github.com/users/mesosphere/following{/other_user}","gists_url":"https://api.github.com/users/mesosphere/gists{/gist_id}","starred_url":"https://api.github.com/users/mesosphere/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mesosphere/subscriptions","organizations_url":"https://api.github.com/users/mesosphere/orgs","repos_url":"https://api.github.com/users/mesosphere/repos","events_url":"https://api.github.com/users/mesosphere/events{/privacy}","received_events_url":"https://api.github.com/users/mesosphere/received_events","type":"Organization","site_admin":false},"repo":{"id":10941409,"name":"marathon","full_name":"mesosphere/marathon","owner":{"login":"mesosphere","id":3458585,"avatar_url":"https://avatars.githubusercontent.com/u/3458585?v=3","gravatar_id":"","url":"https://api.github.com/users/mesosphere","html_url":"https://github.com/mesosphere","followers_url":"https://api.github.com/users/mesosphere/followers","following_url":"https://api.github.com/users/mesosphere/following{/other_user}","gists_url":"https://api.github.com/users/mesosphere/gists{/gist_id}","starred_url":"https://api.github.com/users/mesosphere/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mesosphere/subscriptions","organizations_url":"https://api.github.com/users/mesosphere/orgs","repos_url":"https://api.github.com/users/mesosphere/repos","events_url":"https://api.github.com/users/mesosphere/events{/privacy}","received_events_url":"https://api.github.com/users/mesosphere/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mesosphere/marathon","description":"Deploy and manage containers (including Docker) on top of Apache Mesos at scale.","fork":false,"url":"https://api.github.com/repos/mesosphere/marathon","forks_url":"https://api.github.com/repos/mesosphere/marathon/forks","keys_url":"https://api.github.com/repos/mesosphere/marathon/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mesosphere/marathon/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mesosphere/marathon/teams","hooks_url":"https://api.github.com/repos/mesosphere/marathon/hooks","issue_events_url":"https://api.github.com/repos/mesosphere/marathon/issues/events{/number}","events_url":"https://api.github.com/repos/mesosphere/marathon/events","assignees_url":"https://api.github.com/repos/mesosphere/marathon/assignees{/user}","branches_url":"https://api.github.com/repos/mesosphere/marathon/branches{/branch}","tags_url":"https://api.github.com/repos/mesosphere/marathon/tags","blobs_url":"https://api.github.com/repos/mesosphere/marathon/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mesosphere/marathon/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mesosphere/marathon/git/refs{/sha}","trees_url":"https://api.github.com/repos/mesosphere/marathon/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mesosphere/marathon/statuses/{sha}","languages_url":"https://api.github.com/repos/mesosphere/marathon/languages","stargazers_url":"https://api.github.com/repos/mesosphere/marathon/stargazers","contributors_url":"https://api.github.com/repos/mesosphere/marathon/contributors","subscribers_url":"https://api.github.com/repos/mesosphere/marathon/subscribers","subscription_url":"https://api.github.com/repos/mesosphere/marathon/subscription","commits_url":"https://api.github.com/repos/mesosphere/marathon/commits{/sha}","git_commits_url":"https://api.github.com/repos/mesosphere/marathon/git/commits{/sha}","comments_url":"https://api.github.com/repos/mesosphere/marathon/comments{/number}","issue_comment_url":"https://api.github.com/repos/mesosphere/marathon/issues/comments{/number}","contents_url":"https://api.github.com/repos/mesosphere/marathon/contents/{+path}","compare_url":"https://api.github.com/repos/mesosphere/marathon/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mesosphere/marathon/merges","archive_url":"https://api.github.com/repos/mesosphere/marathon/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mesosphere/marathon/downloads","issues_url":"https://api.github.com/repos/mesosphere/marathon/issues{/number}","pulls_url":"https://api.github.com/repos/mesosphere/marathon/pulls{/number}","milestones_url":"https://api.github.com/repos/mesosphere/marathon/milestones{/number}","notifications_url":"https://api.github.com/repos/mesosphere/marathon/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mesosphere/marathon/labels{/name}","releases_url":"https://api.github.com/repos/mesosphere/marathon/releases{/id}","deployments_url":"https://api.github.com/repos/mesosphere/marathon/deployments","created_at":"2013-06-25T14:56:34Z","updated_at":"2016-07-13T07:25:16Z","pushed_at":"2016-07-13T13:58:43Z","git_url":"git://github.com/mesosphere/marathon.git","ssh_url":"git@github.com:mesosphere/marathon.git","clone_url":"https://github.com/mesosphere/marathon.git","svn_url":"https://github.com/mesosphere/marathon","homepage":"https://mesosphere.github.io/marathon/","size":26971,"stargazers_count":2644,"watchers_count":2644,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":585,"mirror_url":null,"open_issues_count":610,"forks":585,"open_issues":610,"watchers":2644,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mesosphere/marathon/pulls/4069"},"html":{"href":"https://github.com/mesosphere/marathon/pull/4069"},"issue":{"href":"https://api.github.com/repos/mesosphere/marathon/issues/4069"},"comments":{"href":"https://api.github.com/repos/mesosphere/marathon/issues/4069/comments"},"review_comments":{"href":"https://api.github.com/repos/mesosphere/marathon/pulls/4069/comments"},"review_comment":{"href":"https://api.github.com/repos/mesosphere/marathon/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mesosphere/marathon/pulls/4069/commits"},"statuses":{"href":"https://api.github.com/repos/mesosphere/marathon/statuses/842019577090529a5687dc1b5d71f0366b35b4b0"}}}}
{"actor":{"display_login":"jdef"}}