{"action":"created","comment":{"url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/comments/81339107","pull_request_review_id":2326029,"id":81339107,"diff_hunk":"@@ -0,0 +1,324 @@\n+---\n+layout: post\n+title: \"Long running Spark Streaming jobs on YARN cluster\"\n+date: 2016-09-30\n+comments: true\n+categories: [spark, yarn, hdfs]\n+---\n+\n+A long-running Spark Streaming job, once submitted to the YARN cluster should run forever until it is intentionally stopped.\n+Any interruption introduces substantial processing delays and could lead to data loss or duplicates.\n+Neither YARN nor Apache Spark have been designed for running long-running services.\n+But they have been successfully adapted to growing needs of near real-time processing implemented as long-running jobs.\n+Successfully does not necessarily mean without technological challenges.\n+\n+This blog post summarizes my experiences in running mission critical, long-running Spark Streaming jobs on a secured YARN cluster.\n+You will learn how to submit Spark Streaming application to a YARN cluster to avoid sleepless nights during on-call hours.\n+\n+## Fault tolerance\n+\n+In the YARN cluster mode Spark driver runs in the Application Master, the first container allocated by the application.\n+This process is responsible for driving the application and requesting resources (Spark executors) from YARN.\n+What is important, Application Master eliminates need for any another process that run during application lifecycle.\n+Even if an edge Hadoop cluster node where the Spark Streaming job was submitted fails, the application stays unaffected. \n+\n+To run Spark Streaming application in the cluster mode, ensure that the following parameters are given to spark-submit command:\n+\n+    spark-submit --master yarn --deploy-mode cluster\n+\n+Because Spark driver and Application Master share a single JVM, any error in Spark driver stops our long-running job.\n+Fortunately it is possible to configure maximum number of attempts that will be made to re-run the application.\n+It is reasonable to set higher value than default 2 (derived from YARN cluster property ```yarn.resourcemanager.am.max-attempts```).\n+For me 4 works quite well, higher value may cause unnecessary restarts even if the reason of the failure is permanent.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+        --conf spark.yarn.maxAppAttempts=4\n+\n+If the application runs for days or weeks without restart or redeployment on highly utilized cluster, 4 attempts could be exhausted in few hours. \n+To avoid this situation, the attempt counter should be reset on every hour of so.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+        --conf spark.yarn.maxAppAttempts=4 \\\n+        --conf spark.yarn.am.attemptFailuresValidityInterval=1h\n+ \n+Another important setting is a maximum number of executor failures before the application fails. \n+By default it is ```max(2 * num executors, 3)```, well suited for batch jobs but not for long-running jobs.\n+The property comes with corresponding validity interval which also should be set.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+        --conf spark.yarn.maxAppAttempts=4 \\\n+        --conf spark.yarn.am.attemptFailuresValidityInterval=1h \\\n+        --conf spark.yarn.max.executor.failures={8 * num_executors} \\\n+        --conf spark.yarn.executor.failuresValidityInterval=1h\n+        \n+For long-running jobs you could also consider to boost maximum number of task failures before giving up the job.\n+By default tasks will be retried 4 times and then job fails.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+        --conf spark.yarn.maxAppAttempts=4 \\\n+        --conf spark.yarn.am.attemptFailuresValidityInterval=1h \\\n+        --conf spark.yarn.max.executor.failures={8 * num_executors} \\\n+        --conf spark.yarn.executor.failuresValidityInterval=1h \\\n+        --conf spark.task.maxFailures=8\n+ \n+## Performance\n+\n+When Spark Streaming application is submitted to the cluster, YARN queue where the job runs must be defined.\n+I strongly recommend using YARN Capacity Scheduler and submitting long-running jobs to separate queue.\n+Without a separate YARN queue your long-running job will be preempted by a massive Hive query sooner or later.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+        --conf spark.yarn.maxAppAttempts=4 \\\n+        --conf spark.yarn.am.attemptFailuresValidityInterval=1h \\\n+        --conf spark.yarn.max.executor.failures={8 * num_executors} \\\n+        --conf spark.yarn.executor.failuresValidityInterval=1h \\\n+        --conf spark.task.maxFailures=8 \\\n+        --queue realtime_queue\n+\n+\n+Another important issue for Spark Streaming job is keeping processing time stable and highly predictable.\n+Processing time should stay below batch duration to avoid delays. \n+I've found that Spark speculative execution helps a lot, especially on busy cluster. ","path":"source/_posts/2016-09-30-spark-streaming-on-yarn.markdown","position":81,"original_position":81,"commit_id":"06ef748b1459a994bd5cb7b8c9d25f5033e91f50","original_commit_id":"d6e894db051b6518eb37d73f63190523b35652bf","user":{"login":"arciuchm","id":10066133,"avatar_url":"https://avatars.githubusercontent.com/u/10066133?v=3","gravatar_id":"","url":"https://api.github.com/users/arciuchm","html_url":"https://github.com/arciuchm","followers_url":"https://api.github.com/users/arciuchm/followers","following_url":"https://api.github.com/users/arciuchm/following{/other_user}","gists_url":"https://api.github.com/users/arciuchm/gists{/gist_id}","starred_url":"https://api.github.com/users/arciuchm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arciuchm/subscriptions","organizations_url":"https://api.github.com/users/arciuchm/orgs","repos_url":"https://api.github.com/users/arciuchm/repos","events_url":"https://api.github.com/users/arciuchm/events{/privacy}","received_events_url":"https://api.github.com/users/arciuchm/received_events","type":"User","site_admin":false},"body":"on a busy cluster","created_at":"2016-09-30T13:33:43Z","updated_at":"2016-09-30T13:37:23Z","html_url":"https://github.com/mkuthan/mkuthan.github.io/pull/1#discussion_r81339107","pull_request_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1","_links":{"self":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/comments/81339107"},"html":{"href":"https://github.com/mkuthan/mkuthan.github.io/pull/1#discussion_r81339107"},"pull_request":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1"}}},"pull_request":{"url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1","id":87456305,"html_url":"https://github.com/mkuthan/mkuthan.github.io/pull/1","diff_url":"https://github.com/mkuthan/mkuthan.github.io/pull/1.diff","patch_url":"https://github.com/mkuthan/mkuthan.github.io/pull/1.patch","issue_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/1","number":1,"state":"open","locked":false,"title":"Spark Streaming on Yarn","user":{"login":"mkuthan","id":1560526,"avatar_url":"https://avatars.githubusercontent.com/u/1560526?v=3","gravatar_id":"","url":"https://api.github.com/users/mkuthan","html_url":"https://github.com/mkuthan","followers_url":"https://api.github.com/users/mkuthan/followers","following_url":"https://api.github.com/users/mkuthan/following{/other_user}","gists_url":"https://api.github.com/users/mkuthan/gists{/gist_id}","starred_url":"https://api.github.com/users/mkuthan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkuthan/subscriptions","organizations_url":"https://api.github.com/users/mkuthan/orgs","repos_url":"https://api.github.com/users/mkuthan/repos","events_url":"https://api.github.com/users/mkuthan/events{/privacy}","received_events_url":"https://api.github.com/users/mkuthan/received_events","type":"User","site_admin":false},"body":"","created_at":"2016-09-30T08:51:00Z","updated_at":"2016-09-30T13:37:23Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6cdef63ea396c37a1fe959ceb1b8c0611dff529e","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1/commits","review_comments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1/comments","review_comment_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/1/comments","statuses_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/statuses/06ef748b1459a994bd5cb7b8c9d25f5033e91f50","head":{"label":"mkuthan:ss_long_running_job","ref":"ss_long_running_job","sha":"06ef748b1459a994bd5cb7b8c9d25f5033e91f50","user":{"login":"mkuthan","id":1560526,"avatar_url":"https://avatars.githubusercontent.com/u/1560526?v=3","gravatar_id":"","url":"https://api.github.com/users/mkuthan","html_url":"https://github.com/mkuthan","followers_url":"https://api.github.com/users/mkuthan/followers","following_url":"https://api.github.com/users/mkuthan/following{/other_user}","gists_url":"https://api.github.com/users/mkuthan/gists{/gist_id}","starred_url":"https://api.github.com/users/mkuthan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkuthan/subscriptions","organizations_url":"https://api.github.com/users/mkuthan/orgs","repos_url":"https://api.github.com/users/mkuthan/repos","events_url":"https://api.github.com/users/mkuthan/events{/privacy}","received_events_url":"https://api.github.com/users/mkuthan/received_events","type":"User","site_admin":false},"repo":{"id":19867233,"name":"mkuthan.github.io","full_name":"mkuthan/mkuthan.github.io","owner":{"login":"mkuthan","id":1560526,"avatar_url":"https://avatars.githubusercontent.com/u/1560526?v=3","gravatar_id":"","url":"https://api.github.com/users/mkuthan","html_url":"https://github.com/mkuthan","followers_url":"https://api.github.com/users/mkuthan/followers","following_url":"https://api.github.com/users/mkuthan/following{/other_user}","gists_url":"https://api.github.com/users/mkuthan/gists{/gist_id}","starred_url":"https://api.github.com/users/mkuthan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkuthan/subscriptions","organizations_url":"https://api.github.com/users/mkuthan/orgs","repos_url":"https://api.github.com/users/mkuthan/repos","events_url":"https://api.github.com/users/mkuthan/events{/privacy}","received_events_url":"https://api.github.com/users/mkuthan/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mkuthan/mkuthan.github.io","description":"Blog, presentations and other public content published with Jekyll and Octopress.","fork":false,"url":"https://api.github.com/repos/mkuthan/mkuthan.github.io","forks_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/forks","keys_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/teams","hooks_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/hooks","issue_events_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/events{/number}","events_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/events","assignees_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/assignees{/user}","branches_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/branches{/branch}","tags_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/tags","blobs_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/refs{/sha}","trees_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/statuses/{sha}","languages_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/languages","stargazers_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/stargazers","contributors_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/contributors","subscribers_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/subscribers","subscription_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/subscription","commits_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/commits{/sha}","git_commits_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/commits{/sha}","comments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/comments{/number}","issue_comment_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/comments{/number}","contents_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/contents/{+path}","compare_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/merges","archive_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/downloads","issues_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues{/number}","pulls_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls{/number}","milestones_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/milestones{/number}","notifications_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/labels{/name}","releases_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/releases{/id}","deployments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/deployments","created_at":"2014-05-16T18:50:19Z","updated_at":"2016-01-29T19:10:25Z","pushed_at":"2016-09-30T13:36:12Z","git_url":"git://github.com/mkuthan/mkuthan.github.io.git","ssh_url":"git@github.com:mkuthan/mkuthan.github.io.git","clone_url":"https://github.com/mkuthan/mkuthan.github.io.git","svn_url":"https://github.com/mkuthan/mkuthan.github.io","homepage":"","size":4815,"stargazers_count":0,"watchers_count":0,"language":"HTML","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"mkuthan:source","ref":"source","sha":"16f1c98088dcb8a183847c896f611b38a09e8d0f","user":{"login":"mkuthan","id":1560526,"avatar_url":"https://avatars.githubusercontent.com/u/1560526?v=3","gravatar_id":"","url":"https://api.github.com/users/mkuthan","html_url":"https://github.com/mkuthan","followers_url":"https://api.github.com/users/mkuthan/followers","following_url":"https://api.github.com/users/mkuthan/following{/other_user}","gists_url":"https://api.github.com/users/mkuthan/gists{/gist_id}","starred_url":"https://api.github.com/users/mkuthan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkuthan/subscriptions","organizations_url":"https://api.github.com/users/mkuthan/orgs","repos_url":"https://api.github.com/users/mkuthan/repos","events_url":"https://api.github.com/users/mkuthan/events{/privacy}","received_events_url":"https://api.github.com/users/mkuthan/received_events","type":"User","site_admin":false},"repo":{"id":19867233,"name":"mkuthan.github.io","full_name":"mkuthan/mkuthan.github.io","owner":{"login":"mkuthan","id":1560526,"avatar_url":"https://avatars.githubusercontent.com/u/1560526?v=3","gravatar_id":"","url":"https://api.github.com/users/mkuthan","html_url":"https://github.com/mkuthan","followers_url":"https://api.github.com/users/mkuthan/followers","following_url":"https://api.github.com/users/mkuthan/following{/other_user}","gists_url":"https://api.github.com/users/mkuthan/gists{/gist_id}","starred_url":"https://api.github.com/users/mkuthan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkuthan/subscriptions","organizations_url":"https://api.github.com/users/mkuthan/orgs","repos_url":"https://api.github.com/users/mkuthan/repos","events_url":"https://api.github.com/users/mkuthan/events{/privacy}","received_events_url":"https://api.github.com/users/mkuthan/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mkuthan/mkuthan.github.io","description":"Blog, presentations and other public content published with Jekyll and Octopress.","fork":false,"url":"https://api.github.com/repos/mkuthan/mkuthan.github.io","forks_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/forks","keys_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/teams","hooks_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/hooks","issue_events_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/events{/number}","events_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/events","assignees_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/assignees{/user}","branches_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/branches{/branch}","tags_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/tags","blobs_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/refs{/sha}","trees_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/statuses/{sha}","languages_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/languages","stargazers_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/stargazers","contributors_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/contributors","subscribers_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/subscribers","subscription_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/subscription","commits_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/commits{/sha}","git_commits_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/commits{/sha}","comments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/comments{/number}","issue_comment_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/comments{/number}","contents_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/contents/{+path}","compare_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/merges","archive_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/downloads","issues_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues{/number}","pulls_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls{/number}","milestones_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/milestones{/number}","notifications_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/labels{/name}","releases_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/releases{/id}","deployments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/deployments","created_at":"2014-05-16T18:50:19Z","updated_at":"2016-01-29T19:10:25Z","pushed_at":"2016-09-30T13:36:12Z","git_url":"git://github.com/mkuthan/mkuthan.github.io.git","ssh_url":"git@github.com:mkuthan/mkuthan.github.io.git","clone_url":"https://github.com/mkuthan/mkuthan.github.io.git","svn_url":"https://github.com/mkuthan/mkuthan.github.io","homepage":"","size":4815,"stargazers_count":0,"watchers_count":0,"language":"HTML","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1"},"html":{"href":"https://github.com/mkuthan/mkuthan.github.io/pull/1"},"issue":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/1"},"comments":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/1/comments"},"review_comments":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1/comments"},"review_comment":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1/commits"},"statuses":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/statuses/06ef748b1459a994bd5cb7b8c9d25f5033e91f50"}}}}
{"actor":{"display_login":"arciuchm"}}