{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12388","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12388/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12388/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12388/events","html_url":"https://github.com/apache/spark/pull/12388","id":148281292,"number":12388,"title":"[SPARK-14628][WIP] Simplify task metrics by always tracking read/write metrics","user":{"login":"rxin","id":323388,"avatar_url":"https://avatars.githubusercontent.com/u/323388?v=3","gravatar_id":"","url":"https://api.github.com/users/rxin","html_url":"https://github.com/rxin","followers_url":"https://api.github.com/users/rxin/followers","following_url":"https://api.github.com/users/rxin/following{/other_user}","gists_url":"https://api.github.com/users/rxin/gists{/gist_id}","starred_url":"https://api.github.com/users/rxin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxin/subscriptions","organizations_url":"https://api.github.com/users/rxin/orgs","repos_url":"https://api.github.com/users/rxin/repos","events_url":"https://api.github.com/users/rxin/events{/privacy}","received_events_url":"https://api.github.com/users/rxin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2016-04-14T07:52:24Z","updated_at":"2016-04-15T01:24:15Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12388","html_url":"https://github.com/apache/spark/pull/12388","diff_url":"https://github.com/apache/spark/pull/12388.diff","patch_url":"https://github.com/apache/spark/pull/12388.patch"},"body":"## What changes were proposed in this pull request?\r\nPart of the reason why TaskMetrics and its callers are complicated are due to the optional metrics we collect, including input, output, shuffle read, and shuffle write. I think we can always track them and just assign 0 as the initial values. It is usually very obvious whether a task is supposed to read any data or not. By always tracking them, we can remove a lot of map, foreach, flatMap, getOrElse(0L) calls throughout Spark.\r\n\r\nThis patch also changes a few behaviors.\r\n\r\n1. Removed the distinction of data read/write methods (e.g. Hadoop, Memory, Network, etc).\r\n2. Accumulate all data reads and writes, rather than only the first method. (Fixes SPARK-5225)\r\n\r\n## How was this patch tested?\r\nExisting unit tests."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/210236476","html_url":"https://github.com/apache/spark/pull/12388#issuecomment-210236476","issue_url":"https://api.github.com/repos/apache/spark/issues/12388","id":210236476,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-04-15T01:24:15Z","updated_at":"2016-04-15T01:24:15Z","body":"**[Test build #55884 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55884/consoleFull)** for PR 12388 at commit [`a6a3604`](https://github.com/apache/spark/commit/a6a36047f2c5da5a569b899efdf6c0e118f7122a).\n * This patch **fails Scala style tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}