{"action":"created","issue":{"url":"https://api.github.com/repos/apache/flink/issues/2315","repository_url":"https://api.github.com/repos/apache/flink","labels_url":"https://api.github.com/repos/apache/flink/issues/2315/labels{/name}","comments_url":"https://api.github.com/repos/apache/flink/issues/2315/comments","events_url":"https://api.github.com/repos/apache/flink/issues/2315/events","html_url":"https://github.com/apache/flink/pull/2315","id":168567481,"number":2315,"title":"[FLINK-1984] Integrate Flink with Apache Mesos (T1)","user":{"login":"EronWright","id":1775518,"avatar_url":"https://avatars.githubusercontent.com/u/1775518?v=3","gravatar_id":"","url":"https://api.github.com/users/EronWright","html_url":"https://github.com/EronWright","followers_url":"https://api.github.com/users/EronWright/followers","following_url":"https://api.github.com/users/EronWright/following{/other_user}","gists_url":"https://api.github.com/users/EronWright/gists{/gist_id}","starred_url":"https://api.github.com/users/EronWright/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EronWright/subscriptions","organizations_url":"https://api.github.com/users/EronWright/orgs","repos_url":"https://api.github.com/users/EronWright/repos","events_url":"https://api.github.com/users/EronWright/events{/privacy}","received_events_url":"https://api.github.com/users/EronWright/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-08-01T05:08:22Z","updated_at":"2016-08-17T17:16:56Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/flink/pulls/2315","html_url":"https://github.com/apache/flink/pull/2315","diff_url":"https://github.com/apache/flink/pull/2315.diff","patch_url":"https://github.com/apache/flink/pull/2315.patch"},"body":"Part one (\"T1\") of Flink on Mesos.\r\n\r\nImplemented Mesos AppMaster including:\r\n- runners for AppMaster and TaskManager\r\n- MesosFlinkResourceManager as a Mesos framework\r\n- ZK persistent storage for Mesos tasks\r\n- reusable scheduler actors for:\r\n  - offer handling using Netflix Fenzo (LaunchCoordinator)\r\n  - reconciliation (ReconciliationCoordinator)\r\n  - task monitoring (TaskMonitor)\r\n  - connection monitoring (ConnectionMonitor)\r\n- lightweight HTTP server to serve artifacts to the Mesos fetcher (ArtifactServer)\r\n- scenario-based logging for:\r\n  - connectivity issues\r\n  - offer handling (receive, process, decline, rescind, accept)\r\n- incorporated FLINK-4152, FLINK-3904, FLINK-4141, FLINK-3675, FLINK-4166\r\n\r\nDocs aren't yet provided; to be provided when client changes come in.\r\n\r\nSome information on the classes to help reviewers:\r\n\r\n1.  **MesosFlinkResourceManager**.   Few functions:\r\n a. Register as a Mesos scheduler.   Obtain relevant config data (Mesos master URL, creds, etc) from **MesosConfiguration** class (as provided by the runner).\r\n b. Use a **SchedulerProxy** as the callback handler to dispatch Mesos callbacks as Akka messages to the RM.  Route messages accordingly.\r\n c. Handle worker callbacks from base class.   Planning activity occurs synchronously, to a persistent store called **MesosWorkerStore**, and mirrored to the `workersInNew`, `workersInLaunch`, etc that folks will recognize from the YARN implementation.\r\n d. Route offer-related information (new workers, new resource offers) to the **LaunchCoordinator**, who responds with task launch commands contained in **AcceptOffers**.\r\n\r\n2. **LaunchCoordinator**.   The coordinator is an FSM that manages the flow of offers, matching them against a task queue using the Fenzo optimizer.\r\n a. The  `GatheringOffers` state is where the action happens, utilizing the state timeout mechanism to make periodic attempts to launch outstanding tasks.  Note that Fenzo can be tuned to make different tradeoffs about placement quality vs speed (future directions).\r\n b. The launch coordinator is connection-aware, suspending itself when we lose connectivity to Mesos.\r\n c. The launch coordinator is provided a Fenzo `TaskRequest` to describe the needed resources.   Once an offer is matched, the **LaunchableTask** interface facilitates generating the specific `TaskInfo` record to send to Mesos. **LaunchableMesosWorker** is the concrete class that generates the TaskInfo.\r\n\r\n3. **ReconciliationCoordinator**.   Implements the reconciliation protocol described in the Mesos literature.  The **TaskMonitor** actor sends a **Reconcile** message to the coordinator whenever a task’s actual status is unknown (e.g. upon reconnection).\r\n\r\n4. **ConnectionMonitor**.   Provides informative log messages related to the connection state.   Note that the Mesos-provided driver automatically handles reconnection. \r\n\r\n5. **TaskMonitor**.  An FSM that tracks the lifecycle of a Mesos task.  \r\n a. Most state changes are due to plan changes or to actual status changes.\r\n b. Use a state timeout to complain if a task is staging for too long.\r\n \r\n6. **MesosAppMasterRunner**.  The main entrypoint for the AppMaster, which launches the JM, the RM, the artifact server, etc. \r\n a. Prepare a TaskInfo template with common information needed to launch a TM.\r\n b. Otherwise similar to the YARN runner.\r\n\r\n7. **MesosArtifactServer**.  A simple HTTP server for serving artifacts to the Mesos fetcher, to launch the TM.   e.g `flink.jar`, `flink-conf.yaml`, `krb5.keytab` (future).\r\n\r\nNote that the code in the scheduler package is designed to be generic, for reuse by the dispatcher (another Mesos framework) to be developed next.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/flink/issues/comments/240481719","html_url":"https://github.com/apache/flink/pull/2315#issuecomment-240481719","issue_url":"https://api.github.com/repos/apache/flink/issues/2315","id":240481719,"user":{"login":"tillrohrmann","id":5756858,"avatar_url":"https://avatars.githubusercontent.com/u/5756858?v=3","gravatar_id":"","url":"https://api.github.com/users/tillrohrmann","html_url":"https://github.com/tillrohrmann","followers_url":"https://api.github.com/users/tillrohrmann/followers","following_url":"https://api.github.com/users/tillrohrmann/following{/other_user}","gists_url":"https://api.github.com/users/tillrohrmann/gists{/gist_id}","starred_url":"https://api.github.com/users/tillrohrmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tillrohrmann/subscriptions","organizations_url":"https://api.github.com/users/tillrohrmann/orgs","repos_url":"https://api.github.com/users/tillrohrmann/repos","events_url":"https://api.github.com/users/tillrohrmann/events{/privacy}","received_events_url":"https://api.github.com/users/tillrohrmann/received_events","type":"User","site_admin":false},"created_at":"2016-08-17T17:16:55Z","updated_at":"2016-08-17T17:16:55Z","body":"Great work @EronWright. Code quality, testing and extent of this PR is really impressive :-) I think you've nicely modularized the Mesos resource manager with the additional actors. This makes testing much nicer :-)\r\n\r\nI only had some minor comments and questions for my own understanding. Tomorrow I want to try it out on a Mesos cluster to see how it works.\r\n\r\nThere is only one thing I'm wondering. Since we're also currently working on Flip-6 where we try to put a new RPC abstraction in place in order to eventually remove Akka and Scala from flink-runtime, I wanted to ask whether you think that the FSM actors could also be replaced (sometime in the future) by something else? I totally agree that they are a nice abstraction for Mesos and allow to express the logic succinctly."}}
{"actor":{"display_login":"tillrohrmann"}}