{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21252","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21252/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21252/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21252/events","html_url":"https://github.com/apache/spark/pull/21252","id":320600346,"number":21252,"title":"[SPARK-24193] Sort by disk when number of limit is big in TakeOrderedAndProjectExec","user":{"login":"jinxing64","id":4058918,"avatar_url":"https://avatars0.githubusercontent.com/u/4058918?v=4","gravatar_id":"","url":"https://api.github.com/users/jinxing64","html_url":"https://github.com/jinxing64","followers_url":"https://api.github.com/users/jinxing64/followers","following_url":"https://api.github.com/users/jinxing64/following{/other_user}","gists_url":"https://api.github.com/users/jinxing64/gists{/gist_id}","starred_url":"https://api.github.com/users/jinxing64/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jinxing64/subscriptions","organizations_url":"https://api.github.com/users/jinxing64/orgs","repos_url":"https://api.github.com/users/jinxing64/repos","events_url":"https://api.github.com/users/jinxing64/events{/privacy}","received_events_url":"https://api.github.com/users/jinxing64/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2018-05-06T15:06:01Z","updated_at":"2018-05-16T18:17:50Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21252","html_url":"https://github.com/apache/spark/pull/21252","diff_url":"https://github.com/apache/spark/pull/21252.diff","patch_url":"https://github.com/apache/spark/pull/21252.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nPhysical plan of `select colA from t order by colB limit M` is `TakeOrderedAndProject`;\r\nCurrently `TakeOrderedAndProject` sorts data in memory, see https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala#L158 \r\nWe can add a config – if the number of limit (M) is too big, we can sort by disk. Thus memory issue can be resolved.\r\n\r\n## How was this patch tested?\r\n\r\nTest added\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/389616365","html_url":"https://github.com/apache/spark/pull/21252#issuecomment-389616365","issue_url":"https://api.github.com/repos/apache/spark/issues/21252","id":389616365,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-05-16T18:17:50Z","updated_at":"2018-05-16T18:17:50Z","author_association":"NONE","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}