{"action":"created","comment":{"url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/comments/81333597","pull_request_review_id":2319243,"id":81333597,"diff_hunk":"@@ -0,0 +1,324 @@\n+---\n+layout: post\n+title: \"Long running Spark Streaming jobs on YARN cluster\"\n+date: 2016-09-30\n+comments: true\n+categories: [spark, yarn, hdfs]\n+---\n+\n+A long-running Spark Streaming job, once submitted to the YARN cluster should run forever until it is intentionally stopped.\n+Any interruption introduces substantial processing delays and could lead to data loss or duplicates.\n+Neither YARN nor Apache Spark have been designed for running long-running services.\n+But they have been successfully adapted to growing needs of near real-time processing implemented as long-running jobs.\n+Successfully does not necessarily mean without technological challenges.\n+\n+This blog post summarizes my experiences in running mission critical, long-running Spark Streaming jobs on a secured YARN cluster.\n+You will learn how to submit Spark Streaming application to a YARN cluster to avoid sleepless nights during on-call hours.\n+\n+## Fault tolerance\n+\n+In the YARN cluster mode Spark driver runs in the Application Master, the first container allocated by the application.\n+This process is responsible for driving the application and requesting resources (Spark executors) from YARN.\n+What is important, Application Master eliminates need for any another process that run during application lifecycle.\n+Even if an edge Hadoop cluster node where the Spark Streaming job was submitted fails, the application stays unaffected. \n+\n+To run Spark Streaming application in the cluster mode, ensure that the following parameters are given to spark-submit command:\n+\n+    spark-submit --master yarn --deploy-mode cluster\n+\n+Because Spark driver and Application Master share a single JVM, any error in Spark driver stops our long-running job.\n+Fortunately it is possible to configure maximum number of attempts that will be made to re-run the application.\n+It is reasonable to set higher value than default 2 (derived from YARN cluster property ```yarn.resourcemanager.am.max-attempts```).\n+For me 4 works quite well, higher value may cause unnecessary restarts even if the reason of the failure is permanent.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+        --conf spark.yarn.maxAppAttempts=4\n+\n+If the application runs for days or weeks without restart or redeployment on highly utilized cluster, 4 attempts could be exhausted in few hours. \n+To avoid this situation, the attempt counter should be reset on every hour of so.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+        --conf spark.yarn.maxAppAttempts=4 \\\n+        --conf spark.yarn.am.attemptFailuresValidityInterval=1h\n+ \n+Another important setting is a maximum number of executor failures before the application fails. \n+By default it is ```max(2 * num executors, 3)```, well suited for batch jobs but not for long-running jobs.\n+The property comes with corresponding validity interval which also should be set.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+        --conf spark.yarn.maxAppAttempts=4 \\\n+        --conf spark.yarn.am.attemptFailuresValidityInterval=1h \\\n+        --conf spark.yarn.max.executor.failures={8 * num_executors} \\\n+        --conf spark.yarn.executor.failuresValidityInterval=1h\n+        \n+For long-running jobs you could also consider to boost maximum number of task failures before giving up the job.\n+By default tasks will be retried 4 times and then job fails.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+        --conf spark.yarn.maxAppAttempts=4 \\\n+        --conf spark.yarn.am.attemptFailuresValidityInterval=1h \\\n+        --conf spark.yarn.max.executor.failures={8 * num_executors} \\\n+        --conf spark.yarn.executor.failuresValidityInterval=1h \\\n+        --conf spark.task.maxFailures=8\n+ \n+## Performance\n+\n+When Spark Streaming application is submitted to the cluster, YARN queue where the job runs must be defined.\n+I strongly recommend to use YARN Capacity Scheduler and separate queue for long-running jobs.\n+Without a separate YARN queue your long-running job will be preempted by a massive Hive query sooner or later.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+        --conf spark.yarn.maxAppAttempts=4 \\\n+        --conf spark.yarn.am.attemptFailuresValidityInterval=1h \\\n+        --conf spark.yarn.max.executor.failures={8 * num_executors} \\\n+        --conf spark.yarn.executor.failuresValidityInterval=1h \\\n+        --conf spark.task.maxFailures=8 \\\n+        --queue realtime_queue\n+\n+\n+Another important performance factor for Spark Streaming job is processing time predictability. \n+Processing time should stay below batch time to avoid delays. \n+I've found that Spark speculative execution helps a lot, especially on busy cluster. \n+Batch processing times are much more stable than when speculative execution is disabled.\n+Unfortunately speculative mode can be enabled only if Spark actions are idempotent.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+        --conf spark.yarn.maxAppAttempts=4 \\\n+        --conf spark.yarn.am.attemptFailuresValidityInterval=1h \\\n+        --conf spark.yarn.max.executor.failures={8 * num_executors} \\\n+        --conf spark.yarn.executor.failuresValidityInterval=1h \\\n+        --conf spark.task.maxFailures=8 \\\n+        --queue realtime_queue \\\n+        --conf spark.speculation=true\n+\n+\n+## Security\n+\n+On a secured HDFS cluster, long-running Spark Streaming jobs fails due to Kerberos ticket expiration.\n+Without additional settings, Kerberos ticket is issued when Spark Streaming job is submitted to the cluster.\n+When ticket expires Spark Streaming job is not able to write or read data from HDFS anymore.\n+ \n+In theory (based on documentation) it should be enough to pass Kerberos principal and keytab as spark-submit command:\n+ \n+    spark-submit --master yarn --deploy-mode cluster \\\n+         --conf spark.yarn.maxAppAttempts=4 \\\n+         --conf spark.yarn.am.attemptFailuresValidityInterval=1h \\\n+         --conf spark.yarn.max.executor.failures={8 * num_executors} \\\n+         --conf spark.yarn.executor.failuresValidityInterval=1h \\\n+         --conf spark.task.maxFailures=8 \\\n+         --queue realtime_queue \\\n+         --conf spark.speculation=true \\\n+         --principal user/hostname@domain \\\n+         --keytab /path/to/foo.keytab\n+\n+In practice, due to several bugs ([HDFS-9276](https://issues.apache.org/jira/browse/HDFS-9276), [SPARK-11182](https://issues.apache.org/jira/browse/SPARK-11182))\n+HDFS cache must be disabled. If not, Spark will not be able to read updated token from file on HDFS.\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+         --conf spark.yarn.maxAppAttempts=4 \\\n+         --conf spark.yarn.am.attemptFailuresValidityInterval=1h \\\n+         --conf spark.yarn.max.executor.failures={8 * num_executors} \\\n+         --conf spark.yarn.executor.failuresValidityInterval=1h \\\n+         --conf spark.task.maxFailures=8 \\\n+         --queue realtime_queue \\\n+         --conf spark.speculation=true \\\n+         --principal user/hostname@domain \\\n+         --keytab /path/to/foo.keytab \\\n+         --conf spark.hadoop.fs.hdfs.impl.disable.cache=true\n+\n+\n+## Logging\n+\n+The easiest way for Spark jobs to access application logs is to configure Log4j console appender, \n+wait for application termination and use ```yarn logs -applicationId [applicationId]``` command.\n+Unfortunately it is not feasible to terminate long-running Spark Streaming jobs to access the logs.\n+\n+I recommend to install and configure Elastic, Logstash and Kibana (ELK stack).\n+ELK installation and configuration is out of this blog post scope,\n+but remember to log the following context fields:\n+\n+* YARN application id\n+* YARN container hostname\n+* Executor id (Spark driver is always 000001, Spark executors start from 000002)\n+* YARN attempt (to check how many times Spark driver has been restarted, attempts are decreased during application lifecycle accordingly to ```spark.yarn.am.attemptFailuresValidityInterval```)\n+\n+Log4j configuration with Logstash specific appender and layout definition should be passed to spark-submit command:\n+ \n+    spark-submit --master yarn --deploy-mode cluster \\\n+         --conf spark.yarn.maxAppAttempts=4 \\\n+         --conf spark.yarn.am.attemptFailuresValidityInterval=1h \\\n+         --conf spark.yarn.max.executor.failures={8 * num_executors} \\\n+         --conf spark.yarn.executor.failuresValidityInterval=1h \\\n+         --conf spark.task.maxFailures=8 \\\n+         --queue realtime_queue \\\n+         --conf spark.speculation=true \\\n+         --principal user/hostname@domain \\\n+         --keytab /path/to/foo.keytab \\\n+         --conf spark.hadoop.fs.hdfs.impl.disable.cache=true \\\n+         --conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties \\\n+         --conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties \\\n+         --files /path/to/log4j.properties\n+\n+Finally Kibana dashboard for Spark Job might look like:\n+\n+{% img /images/blog/spark_job_logging.png [spark job logging] %}\n+\n+## Monitoring\n+\n+Long running job runs 24/7 so it is important to have an insight into historical metrics. \n+Again, external tools are needed. \n+I recommend to install Graphite for collecting metrics and Grafana for building dashboards.\n+\n+First, Spark needs to be configured to report metrics into Graphite, prepare the ```metrics.properties``` file:\n+\n+    *.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink\n+    *.sink.graphite.host=[hostname]\n+    *.sink.graphite.port=[port]\n+    *.sink.graphite.prefix=stats.analytics // this prefix will be used later on\n+    \n+    driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource\n+    executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource\n+\n+And configure spark-submit command:\n+\n+    spark-submit --master yarn --deploy-mode cluster \\\n+         --conf spark.yarn.maxAppAttempts=4 \\\n+         --conf spark.yarn.am.attemptFailuresValidityInterval=1h \\\n+         --conf spark.yarn.max.executor.failures={8 * num_executors} \\\n+         --conf spark.yarn.executor.failuresValidityInterval=1h \\\n+         --conf spark.task.maxFailures=8 \\\n+         --queue realtime_queue \\\n+         --conf spark.speculation=true \\\n+         --principal user/hostname@domain \\\n+         --keytab /path/to/foo.keytab \\\n+         --conf spark.hadoop.fs.hdfs.impl.disable.cache=true \\\n+         --conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties \\\n+         --conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties \\\n+         --files /path/to/log4j.properties:/path/to/metrics.properties\n+\n+Spark publishes tons of metrics from driver and executors.\n+If I have to choose the most important one, it would be the last received batch records.\n+When ```StreamingMetrics.streaming.lastReceivedBatch_records == 0``` it probably means that Spark Streaming job has been stopped or failed.\n+\n+Other important metrics are listed below:\n+\n+When total delay is greater than batch interval, latency of the processing pipeline increases.\n+\n+    driver.StreamingMetrics.streaming.lastCompletedBatch_totalDelay\n+\n+When number of active tasks is lower than ```number of executors * number of cores```, allocated resources are not fully utilized.\n+\n+    executor.threadpool.activeTasks\n+\n+How much RAM is used for RDD cache.\n+\n+    driver.BlockManager.memory.memUsed_MB\n+\n+When there is not enough RAM for RDD cache, how much data has been spilled to disk. \n+You should increase executor memory or change ```spark.memory.fraction``` Spark property to avoid performance degradation. \n+\n+    driver.BlockManager.disk.diskSpaceUsed_MB\n+\n+What is JVM memory utilization on Spark driver.\n+\n+    driver.jvm.heap.used\n+    driver.jvm.non-heap.used\n+    driver.jvm.pools.G1-Old-Gen.used\n+    driver.jvm.pools.G1-Eden-Space.used\n+    driver.jvm.pools.G1-Survivor-Space.used\n+\n+How much time is spent on GC on Spark driver.\n+\n+    driver.jvm.G1-Old-Generation.time\n+    driver.jvm.G1-Young-Generation.time\n+\n+What is JMV memory utilization on Spark executors.\n+\n+    [0-9]*.jvm.heap.used\n+    [0-9]*.jvm.non-heap.used\n+    [0-9]*.jvm.pools.G1-Old-Gen.used\n+    [0-9]*.jvm.pools.G1-Survivor-Space.used\n+    [0-9]*.jvm.pools.G1-Eden-Space.used\n+\n+How much time is spent on GC on Spark executors.\n+\n+    [0-9]*.jvm.G1-Old-Generation.time\n+    [0-9]*.jvm.G1-Young-Generation.time\n+\n+\n+When you configure first Grafana dashboard for Spark job, perhaps the first question will emerge: \n+\n+> How to configure Graphite query when metrics for every Spark application run are reported under its own application id?\n+\n+For driver metrics use wildcard ```.*(application_[0-9]+).*``` and ```aliasSub``` Graphite function to present 'application id' as graph legend:\n+\n+    aliasSub(stats.analytics.$job_name.*.prod.$dc.*.driver.jvm.heap.used, \".*(application_[0-9]+).*\", \"heap: \\1\")\n+    \n+For executor metrics again use wildcard ```.*(application_[0-9]+).*```, ```groupByNode``` Graphite function to sum metrics from all Spark executors and ```aliasSub``` Graphite function to present 'application id' as graph legend:\n+\n+    aliasSub(groupByNode(stats.analytics.$job_name.*.prod.$dc.*.[0-9]*.jvm.heap.used, 6, \"sumSeries\"), \"(.*)\", \"heap: \\1\")\n+\n+Finally Grafana dashboard for Spark Job might look like:\n+\n+{% img /images/blog/spark_job_monitoring.png [spark job monitoring] %}\n+\n+If Spark application is restarted frequently, metrics for old, already finished runs should be deleted from Graphite.\n+Because Graphite does not compact inactive metrics, old metrics slow down Graphite itself and Grafana queries.\n+\n+## Graceful stop\n+\n+The last puzzle element is how to stop Spark Streaming application deployed on YARN in a graceful way.\n+The standard method for stopping (or rather killing) YARN application is using a command ```yarn application -kill [applicationId]```.\n+And this command stops the Spark Streaming application but the application might be killed in the middle of the batch.\n+So if the job reads data from Kafka, save processing results on HDFS and finally commit Kafka offsets\n+you should expect duplicated data on HDFS when job was stopped just before committing offsets.\n+\n+The first attempt to solve graceful shutdown issue was to call Spark streaming context stop method in shutdown hook.\n+\n+``` scala\n+sys.addShutdownHook {\n+    streamingContext.stop(stopSparkContext = true, stopGracefully = true)\n+}\n+```\n+\n+Disappointingly a shutdown hook is called too late to finish started batch and Spark application is killed almost immediately.\n+Moreover there is no guarantee that a shutdown hook will be called by JVM at all.\n+\n+At the time of writing this blog post the only confirmed way to shutdown gracefully Spark Streaming application on YARN\n+is to notify somehow the application about planned shutdown, and then stop streaming context programmatically (but not from shutdown hook).\n+Command ```yarn application -kill``` should be used only as a last resort if notified application did not stop after defined timeout.","path":"source/_posts/2016-09-30-spark-streaming-on-yarn.markdown","position":289,"original_position":289,"commit_id":"aa17921025c14bb15664473e20e37840bab684cd","original_commit_id":"aa17921025c14bb15664473e20e37840bab684cd","user":{"login":"pguzik","id":14015095,"avatar_url":"https://avatars.githubusercontent.com/u/14015095?v=3","gravatar_id":"","url":"https://api.github.com/users/pguzik","html_url":"https://github.com/pguzik","followers_url":"https://api.github.com/users/pguzik/followers","following_url":"https://api.github.com/users/pguzik/following{/other_user}","gists_url":"https://api.github.com/users/pguzik/gists{/gist_id}","starred_url":"https://api.github.com/users/pguzik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pguzik/subscriptions","organizations_url":"https://api.github.com/users/pguzik/orgs","repos_url":"https://api.github.com/users/pguzik/repos","events_url":"https://api.github.com/users/pguzik/events{/privacy}","received_events_url":"https://api.github.com/users/pguzik/received_events","type":"User","site_admin":false},"body":"is notifying the application","created_at":"2016-09-30T13:00:33Z","updated_at":"2016-09-30T13:02:31Z","html_url":"https://github.com/mkuthan/mkuthan.github.io/pull/1#discussion_r81333597","pull_request_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1","_links":{"self":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/comments/81333597"},"html":{"href":"https://github.com/mkuthan/mkuthan.github.io/pull/1#discussion_r81333597"},"pull_request":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1"}}},"pull_request":{"url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1","id":87456305,"html_url":"https://github.com/mkuthan/mkuthan.github.io/pull/1","diff_url":"https://github.com/mkuthan/mkuthan.github.io/pull/1.diff","patch_url":"https://github.com/mkuthan/mkuthan.github.io/pull/1.patch","issue_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/1","number":1,"state":"open","locked":false,"title":"Spark Streaming on Yarn","user":{"login":"mkuthan","id":1560526,"avatar_url":"https://avatars.githubusercontent.com/u/1560526?v=3","gravatar_id":"","url":"https://api.github.com/users/mkuthan","html_url":"https://github.com/mkuthan","followers_url":"https://api.github.com/users/mkuthan/followers","following_url":"https://api.github.com/users/mkuthan/following{/other_user}","gists_url":"https://api.github.com/users/mkuthan/gists{/gist_id}","starred_url":"https://api.github.com/users/mkuthan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkuthan/subscriptions","organizations_url":"https://api.github.com/users/mkuthan/orgs","repos_url":"https://api.github.com/users/mkuthan/repos","events_url":"https://api.github.com/users/mkuthan/events{/privacy}","received_events_url":"https://api.github.com/users/mkuthan/received_events","type":"User","site_admin":false},"body":"","created_at":"2016-09-30T08:51:00Z","updated_at":"2016-09-30T13:02:31Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7327d92d5baa8a651fa734413f5ecdc835d87a15","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1/commits","review_comments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1/comments","review_comment_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/1/comments","statuses_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/statuses/aa17921025c14bb15664473e20e37840bab684cd","head":{"label":"mkuthan:ss_long_running_job","ref":"ss_long_running_job","sha":"aa17921025c14bb15664473e20e37840bab684cd","user":{"login":"mkuthan","id":1560526,"avatar_url":"https://avatars.githubusercontent.com/u/1560526?v=3","gravatar_id":"","url":"https://api.github.com/users/mkuthan","html_url":"https://github.com/mkuthan","followers_url":"https://api.github.com/users/mkuthan/followers","following_url":"https://api.github.com/users/mkuthan/following{/other_user}","gists_url":"https://api.github.com/users/mkuthan/gists{/gist_id}","starred_url":"https://api.github.com/users/mkuthan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkuthan/subscriptions","organizations_url":"https://api.github.com/users/mkuthan/orgs","repos_url":"https://api.github.com/users/mkuthan/repos","events_url":"https://api.github.com/users/mkuthan/events{/privacy}","received_events_url":"https://api.github.com/users/mkuthan/received_events","type":"User","site_admin":false},"repo":{"id":19867233,"name":"mkuthan.github.io","full_name":"mkuthan/mkuthan.github.io","owner":{"login":"mkuthan","id":1560526,"avatar_url":"https://avatars.githubusercontent.com/u/1560526?v=3","gravatar_id":"","url":"https://api.github.com/users/mkuthan","html_url":"https://github.com/mkuthan","followers_url":"https://api.github.com/users/mkuthan/followers","following_url":"https://api.github.com/users/mkuthan/following{/other_user}","gists_url":"https://api.github.com/users/mkuthan/gists{/gist_id}","starred_url":"https://api.github.com/users/mkuthan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkuthan/subscriptions","organizations_url":"https://api.github.com/users/mkuthan/orgs","repos_url":"https://api.github.com/users/mkuthan/repos","events_url":"https://api.github.com/users/mkuthan/events{/privacy}","received_events_url":"https://api.github.com/users/mkuthan/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mkuthan/mkuthan.github.io","description":"Blog, presentations and other public content published with Jekyll and Octopress.","fork":false,"url":"https://api.github.com/repos/mkuthan/mkuthan.github.io","forks_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/forks","keys_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/teams","hooks_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/hooks","issue_events_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/events{/number}","events_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/events","assignees_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/assignees{/user}","branches_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/branches{/branch}","tags_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/tags","blobs_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/refs{/sha}","trees_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/statuses/{sha}","languages_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/languages","stargazers_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/stargazers","contributors_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/contributors","subscribers_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/subscribers","subscription_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/subscription","commits_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/commits{/sha}","git_commits_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/commits{/sha}","comments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/comments{/number}","issue_comment_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/comments{/number}","contents_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/contents/{+path}","compare_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/merges","archive_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/downloads","issues_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues{/number}","pulls_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls{/number}","milestones_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/milestones{/number}","notifications_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/labels{/name}","releases_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/releases{/id}","deployments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/deployments","created_at":"2014-05-16T18:50:19Z","updated_at":"2016-01-29T19:10:25Z","pushed_at":"2016-09-30T10:57:02Z","git_url":"git://github.com/mkuthan/mkuthan.github.io.git","ssh_url":"git@github.com:mkuthan/mkuthan.github.io.git","clone_url":"https://github.com/mkuthan/mkuthan.github.io.git","svn_url":"https://github.com/mkuthan/mkuthan.github.io","homepage":"","size":4815,"stargazers_count":0,"watchers_count":0,"language":"HTML","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"mkuthan:source","ref":"source","sha":"16f1c98088dcb8a183847c896f611b38a09e8d0f","user":{"login":"mkuthan","id":1560526,"avatar_url":"https://avatars.githubusercontent.com/u/1560526?v=3","gravatar_id":"","url":"https://api.github.com/users/mkuthan","html_url":"https://github.com/mkuthan","followers_url":"https://api.github.com/users/mkuthan/followers","following_url":"https://api.github.com/users/mkuthan/following{/other_user}","gists_url":"https://api.github.com/users/mkuthan/gists{/gist_id}","starred_url":"https://api.github.com/users/mkuthan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkuthan/subscriptions","organizations_url":"https://api.github.com/users/mkuthan/orgs","repos_url":"https://api.github.com/users/mkuthan/repos","events_url":"https://api.github.com/users/mkuthan/events{/privacy}","received_events_url":"https://api.github.com/users/mkuthan/received_events","type":"User","site_admin":false},"repo":{"id":19867233,"name":"mkuthan.github.io","full_name":"mkuthan/mkuthan.github.io","owner":{"login":"mkuthan","id":1560526,"avatar_url":"https://avatars.githubusercontent.com/u/1560526?v=3","gravatar_id":"","url":"https://api.github.com/users/mkuthan","html_url":"https://github.com/mkuthan","followers_url":"https://api.github.com/users/mkuthan/followers","following_url":"https://api.github.com/users/mkuthan/following{/other_user}","gists_url":"https://api.github.com/users/mkuthan/gists{/gist_id}","starred_url":"https://api.github.com/users/mkuthan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkuthan/subscriptions","organizations_url":"https://api.github.com/users/mkuthan/orgs","repos_url":"https://api.github.com/users/mkuthan/repos","events_url":"https://api.github.com/users/mkuthan/events{/privacy}","received_events_url":"https://api.github.com/users/mkuthan/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mkuthan/mkuthan.github.io","description":"Blog, presentations and other public content published with Jekyll and Octopress.","fork":false,"url":"https://api.github.com/repos/mkuthan/mkuthan.github.io","forks_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/forks","keys_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/teams","hooks_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/hooks","issue_events_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/events{/number}","events_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/events","assignees_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/assignees{/user}","branches_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/branches{/branch}","tags_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/tags","blobs_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/refs{/sha}","trees_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/statuses/{sha}","languages_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/languages","stargazers_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/stargazers","contributors_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/contributors","subscribers_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/subscribers","subscription_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/subscription","commits_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/commits{/sha}","git_commits_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/git/commits{/sha}","comments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/comments{/number}","issue_comment_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/comments{/number}","contents_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/contents/{+path}","compare_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/merges","archive_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/downloads","issues_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues{/number}","pulls_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls{/number}","milestones_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/milestones{/number}","notifications_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/labels{/name}","releases_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/releases{/id}","deployments_url":"https://api.github.com/repos/mkuthan/mkuthan.github.io/deployments","created_at":"2014-05-16T18:50:19Z","updated_at":"2016-01-29T19:10:25Z","pushed_at":"2016-09-30T10:57:02Z","git_url":"git://github.com/mkuthan/mkuthan.github.io.git","ssh_url":"git@github.com:mkuthan/mkuthan.github.io.git","clone_url":"https://github.com/mkuthan/mkuthan.github.io.git","svn_url":"https://github.com/mkuthan/mkuthan.github.io","homepage":"","size":4815,"stargazers_count":0,"watchers_count":0,"language":"HTML","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1"},"html":{"href":"https://github.com/mkuthan/mkuthan.github.io/pull/1"},"issue":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/1"},"comments":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/issues/1/comments"},"review_comments":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1/comments"},"review_comment":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/pulls/1/commits"},"statuses":{"href":"https://api.github.com/repos/mkuthan/mkuthan.github.io/statuses/aa17921025c14bb15664473e20e37840bab684cd"}}}}
{"actor":{"display_login":"pguzik"}}