{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22219","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22219/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22219/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22219/events","html_url":"https://github.com/apache/spark/pull/22219","id":353762969,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEwNzI1NzM3","number":22219,"title":"[SPARK-25224][SQL] Improvement of Spark SQL ThriftServer memory management","user":{"login":"Dooyoung-Hwang","id":42426020,"node_id":"MDQ6VXNlcjQyNDI2MDIw","avatar_url":"https://avatars2.githubusercontent.com/u/42426020?v=4","gravatar_id":"","url":"https://api.github.com/users/Dooyoung-Hwang","html_url":"https://github.com/Dooyoung-Hwang","followers_url":"https://api.github.com/users/Dooyoung-Hwang/followers","following_url":"https://api.github.com/users/Dooyoung-Hwang/following{/other_user}","gists_url":"https://api.github.com/users/Dooyoung-Hwang/gists{/gist_id}","starred_url":"https://api.github.com/users/Dooyoung-Hwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dooyoung-Hwang/subscriptions","organizations_url":"https://api.github.com/users/Dooyoung-Hwang/orgs","repos_url":"https://api.github.com/users/Dooyoung-Hwang/repos","events_url":"https://api.github.com/users/Dooyoung-Hwang/events{/privacy}","received_events_url":"https://api.github.com/users/Dooyoung-Hwang/received_events","type":"User","site_admin":false},"labels":[{"id":1405794576,"node_id":"MDU6TGFiZWwxNDA1Nzk0NTc2","url":"https://api.github.com/repos/apache/spark/labels/SQL","name":"SQL","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":31,"created_at":"2018-08-24T12:25:47Z","updated_at":"2019-10-25T22:53:33Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22219","html_url":"https://github.com/apache/spark/pull/22219","diff_url":"https://github.com/apache/spark/pull/22219.diff","patch_url":"https://github.com/apache/spark/pull/22219.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nSpark SQL only have two options for managing thriftserver memory - enable spark.sql.thriftServer.incrementalCollect or not\r\n\r\n### The case of enabling spark.sql.thriftServer.incrementalCollects\r\n   - Pros\r\n        - thriftserver can handle large output without OOM.\r\n   - Cons\r\n        - Performance degradation because of executing task partition by partition.\r\n        - Handle queries with count-limit inefficiently because of executing all partitions.\r\n        - Does not cache result for FETCH_FIRST\r\n\r\n### The case of disabling spark.sql.thriftServer.incrementalCollects\r\n   - Pros\r\n        - Good performance for small output\r\n   - Cons\r\n        - Memory peak usage is too large because allocating decompressed & deserialized rows in \"batch\" manner, and OOM could occur for large output.\r\n        - It is difficult to measure memory peak usage of Query, so configuring spark.driver.maxResultSize is very difficult.\r\n        - If decompressed & deserialized rows fills up eden area of JVM Heap, they moves to old Gen and could increase possibility of \"Full GC\" that stops the world.\r\n \r\n### The improvement idea of solving these problems is below.\r\n   - DataSet does not decompress & deserialize result, and just return total row count & iterator to SQL-Executor. By doing that, only compressed data reside in memory, so that the memory usage is not only much lower than before but can be controlled with spark.driver.maxResultSize config.\r\n\r\n   - After SQL-Executor get total row count & iterator from DataSet, SQL-Executor could decide whether deserializing them collectively or iteratively with considering returned row count.\r\n\r\n## How was this patch tested?\r\nAdd test cases.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/546537855","html_url":"https://github.com/apache/spark/pull/22219#issuecomment-546537855","issue_url":"https://api.github.com/repos/apache/spark/issues/22219","id":546537855,"node_id":"MDEyOklzc3VlQ29tbWVudDU0NjUzNzg1NQ==","user":{"login":"kaushikCanada","id":21248009,"node_id":"MDQ6VXNlcjIxMjQ4MDA5","avatar_url":"https://avatars2.githubusercontent.com/u/21248009?v=4","gravatar_id":"","url":"https://api.github.com/users/kaushikCanada","html_url":"https://github.com/kaushikCanada","followers_url":"https://api.github.com/users/kaushikCanada/followers","following_url":"https://api.github.com/users/kaushikCanada/following{/other_user}","gists_url":"https://api.github.com/users/kaushikCanada/gists{/gist_id}","starred_url":"https://api.github.com/users/kaushikCanada/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaushikCanada/subscriptions","organizations_url":"https://api.github.com/users/kaushikCanada/orgs","repos_url":"https://api.github.com/users/kaushikCanada/repos","events_url":"https://api.github.com/users/kaushikCanada/events{/privacy}","received_events_url":"https://api.github.com/users/kaushikCanada/received_events","type":"User","site_admin":false},"created_at":"2019-10-25T22:53:33Z","updated_at":"2019-10-25T22:53:33Z","author_association":"NONE","body":"has this patch been introduced into spark? can i use this on the new spark 2.4?"}}
{"actor":{"display_login":"kaushikCanada"}}