{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13537","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13537/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13537/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13537/events","html_url":"https://github.com/apache/spark/pull/13537","id":158812906,"number":13537,"title":"[SPARK-15794] Should truncate toString() of very wide plans","user":{"login":"ericl","id":14922,"avatar_url":"https://avatars.githubusercontent.com/u/14922?v=3","gravatar_id":"","url":"https://api.github.com/users/ericl","html_url":"https://github.com/ericl","followers_url":"https://api.github.com/users/ericl/followers","following_url":"https://api.github.com/users/ericl/following{/other_user}","gists_url":"https://api.github.com/users/ericl/gists{/gist_id}","starred_url":"https://api.github.com/users/ericl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericl/subscriptions","organizations_url":"https://api.github.com/users/ericl/orgs","repos_url":"https://api.github.com/users/ericl/repos","events_url":"https://api.github.com/users/ericl/events{/privacy}","received_events_url":"https://api.github.com/users/ericl/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-06-07T01:51:38Z","updated_at":"2016-06-08T06:12:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13537","html_url":"https://github.com/apache/spark/pull/13537","diff_url":"https://github.com/apache/spark/pull/13537.diff","patch_url":"https://github.com/apache/spark/pull/13537.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nWith very wide tables, e.g. thousands of fields, the plan output is unreadable and often causes OOMs due to inefficient string processing. This truncates all struct and operator field lists to a user configurable threshold to limit performance impact.\r\n\r\nIt would also be nice to optimize string generation to avoid these sort of O(n^2) slowdowns entirely (i.e. use StringBuilder everywhere including expressions), but this is probably too large of a change for 2.0 at this point, and truncation has other benefits for usability.\r\n\r\n## How was this patch tested?\r\n\r\nAdded a microbenchmark that covers this case particularly well. I also ran the microbenchmark while varying the truncation threshold.\r\n\r\n```\r\nnumFields = 5\r\nwide shallowly nested struct field r/w:  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\n2000 wide x 50 rows (write in-mem)            2336 / 2558          0.0       23364.4       0.1X\r\n\r\nnumFields = 25\r\nwide shallowly nested struct field r/w:  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\n2000 wide x 50 rows (write in-mem)            4237 / 4465          0.0       42367.9       0.1X\r\n\r\nnumFields = 100\r\nwide shallowly nested struct field r/w:  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\n2000 wide x 50 rows (write in-mem)          10458 / 11223          0.0      104582.0       0.0X\r\n\r\nnumFields = Infinity\r\nwide shallowly nested struct field r/w:  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\n[info]   java.lang.OutOfMemoryError: Java heap space\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/224498313","html_url":"https://github.com/apache/spark/pull/13537#issuecomment-224498313","issue_url":"https://api.github.com/repos/apache/spark/issues/13537","id":224498313,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-06-08T06:12:30Z","updated_at":"2016-06-08T06:12:30Z","body":"**[Test build #60154 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60154/consoleFull)** for PR 13537 at commit [`1d5574a`](https://github.com/apache/spark/commit/1d5574a96a10824e3a75d935d662d43c0454ae36)."}}
{"actor":{"display_login":"SparkQA"}}