{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12217","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12217/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12217/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12217/events","html_url":"https://github.com/apache/spark/pull/12217","id":146432740,"number":12217,"title":"[WIP][SPARK-14408][CORE] Changed RDD.treeAggregate to use fold instead of reduce","user":{"login":"jkbradley","id":5084283,"avatar_url":"https://avatars.githubusercontent.com/u/5084283?v=3","gravatar_id":"","url":"https://api.github.com/users/jkbradley","html_url":"https://github.com/jkbradley","followers_url":"https://api.github.com/users/jkbradley/followers","following_url":"https://api.github.com/users/jkbradley/following{/other_user}","gists_url":"https://api.github.com/users/jkbradley/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbradley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbradley/subscriptions","organizations_url":"https://api.github.com/users/jkbradley/orgs","repos_url":"https://api.github.com/users/jkbradley/repos","events_url":"https://api.github.com/users/jkbradley/events{/privacy}","received_events_url":"https://api.github.com/users/jkbradley/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2016-04-06T20:57:14Z","updated_at":"2016-04-08T12:23:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12217","html_url":"https://github.com/apache/spark/pull/12217","diff_url":"https://github.com/apache/spark/pull/12217.diff","patch_url":"https://github.com/apache/spark/pull/12217.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nPreviously, RDD.treeAggregate used reduceByKey and reduce in its implementation, neither of which technically allows the seq/combOps to modify and return their first arguments.\r\n\r\nThis PR uses foldByKey and fold instead and notes that aggregate and treeAggregate are semantically identical in the Scala doc.\r\n\r\n## How was this patch tested?\r\n\r\nExisting unit tests"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/207410404","html_url":"https://github.com/apache/spark/pull/12217#issuecomment-207410404","issue_url":"https://api.github.com/repos/apache/spark/issues/12217","id":207410404,"user":{"login":"srowen","id":822522,"avatar_url":"https://avatars.githubusercontent.com/u/822522?v=3","gravatar_id":"","url":"https://api.github.com/users/srowen","html_url":"https://github.com/srowen","followers_url":"https://api.github.com/users/srowen/followers","following_url":"https://api.github.com/users/srowen/following{/other_user}","gists_url":"https://api.github.com/users/srowen/gists{/gist_id}","starred_url":"https://api.github.com/users/srowen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/srowen/subscriptions","organizations_url":"https://api.github.com/users/srowen/orgs","repos_url":"https://api.github.com/users/srowen/repos","events_url":"https://api.github.com/users/srowen/events{/privacy}","received_events_url":"https://api.github.com/users/srowen/received_events","type":"User","site_admin":false},"created_at":"2016-04-08T12:23:18Z","updated_at":"2016-04-08T12:23:18Z","body":"@jkbradley I see what you mean, but, thinking about it, can this work? the reduce function is applied directly to RDD elements, so modifying one of the arguments and returning it means you're mutating the elements of the RDD in memory, which may have some undefined consequences. For fold, it's fine because the left argument is always actually the zero-value object. Right? Or am I not thinking about it correctly. It might happen to be fine to use reduce in some cases where the RDD values are not used again."}}