{"action":"created","issue":{"url":"https://api.github.com/repos/linkedin/dr-elephant/issues/84","repository_url":"https://api.github.com/repos/linkedin/dr-elephant","labels_url":"https://api.github.com/repos/linkedin/dr-elephant/issues/84/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/dr-elephant/issues/84/comments","events_url":"https://api.github.com/repos/linkedin/dr-elephant/issues/84/events","html_url":"https://github.com/linkedin/dr-elephant/pull/84","id":162164657,"number":84,"title":"LIHADOOP-20318: Make Dr. Elephant analyze running jobs.","user":{"login":"akshayrai","id":4448437,"avatar_url":"https://avatars.githubusercontent.com/u/4448437?v=3","gravatar_id":"","url":"https://api.github.com/users/akshayrai","html_url":"https://github.com/akshayrai","followers_url":"https://api.github.com/users/akshayrai/followers","following_url":"https://api.github.com/users/akshayrai/following{/other_user}","gists_url":"https://api.github.com/users/akshayrai/gists{/gist_id}","starred_url":"https://api.github.com/users/akshayrai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akshayrai/subscriptions","organizations_url":"https://api.github.com/users/akshayrai/orgs","repos_url":"https://api.github.com/users/akshayrai/repos","events_url":"https://api.github.com/users/akshayrai/events{/privacy}","received_events_url":"https://api.github.com/users/akshayrai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-06-24T14:53:26Z","updated_at":"2016-06-30T20:00:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/linkedin/dr-elephant/pulls/84","html_url":"https://github.com/linkedin/dr-elephant/pull/84","diff_url":"https://github.com/linkedin/dr-elephant/pull/84.diff","patch_url":"https://github.com/linkedin/dr-elephant/pull/84.patch"},"body":"This patch will support analysis of Mapreduce jobs in realtime soon after a task completes.\r\n\r\n**Logic flow for better understanding this patch:**\r\n* All the Completed jobs (succeeded and failed) fetched from the RM will go to a completed queue and will be analyzed by a set of threads(say, completedThreads) dedicated to them.\r\n* All the Running jobs(jobs in undefined state) fetched from RM will go to a running queue and will be analyzed by another set of dedicated threads(say, runningThreads) which will compute the live metrics and potentially any new realtime heuristics.\r\n* When a running job completes, it will be added to the completed queue and this will then be processed by a completedThread which will compute the final metrics and run all the heuristics.\r\n\r\n**Code Changes:**\r\n1. You can find the main logic implemented in ```ElephanRunner.java, AnalyticJobGeneratorHadoop2.java and MapReduceFetcherHadoop2.java```\r\n2. Added a GeneralConf.xml to configure the number of executor threads, realtime threads, fetch interval, fetch lag and real time update interval.\r\n3. Added separate evolutions for h2(test) and mysql as they follow different syntax. \r\n\r\n**Schema changes**\r\n1. Added a new level of severity called ```Undefined``` set to 0. We need to shift all the severities one level up as they currently start from 0.\r\n2. Added a new field called status to yarn_app_result which tracks the status of the application. \r\n\r\n**Testing**\r\n1. Added new tests.\r\n2. Tested on local deployment.\r\n3. Tested on a dev cluster.\r\n4. Need to test on a live cluster. (TODO)"},"comment":{"url":"https://api.github.com/repos/linkedin/dr-elephant/issues/comments/229771386","html_url":"https://github.com/linkedin/dr-elephant/pull/84#issuecomment-229771386","issue_url":"https://api.github.com/repos/linkedin/dr-elephant/issues/84","id":229771386,"user":{"login":"paulbramsen","id":4658911,"avatar_url":"https://avatars.githubusercontent.com/u/4658911?v=3","gravatar_id":"","url":"https://api.github.com/users/paulbramsen","html_url":"https://github.com/paulbramsen","followers_url":"https://api.github.com/users/paulbramsen/followers","following_url":"https://api.github.com/users/paulbramsen/following{/other_user}","gists_url":"https://api.github.com/users/paulbramsen/gists{/gist_id}","starred_url":"https://api.github.com/users/paulbramsen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/paulbramsen/subscriptions","organizations_url":"https://api.github.com/users/paulbramsen/orgs","repos_url":"https://api.github.com/users/paulbramsen/repos","events_url":"https://api.github.com/users/paulbramsen/events{/privacy}","received_events_url":"https://api.github.com/users/paulbramsen/received_events","type":"User","site_admin":false},"created_at":"2016-06-30T20:00:42Z","updated_at":"2016-06-30T20:00:42Z","body":"Been testing this out. Pretty cool! One thing to note: the jobtracker URL is broken for running jobs (tested with Hadoop 2.5.0)."}}
{"actor":{"display_login":"paulbramsen"}}