{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22754","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22754/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22754/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22754/events","html_url":"https://github.com/apache/spark/pull/22754","id":370928963,"node_id":"MDExOlB1bGxSZXF1ZXN0MjIzNDkxMjU1","number":22754,"title":"[SPARK-25776][CORE]The disk write buffer size must be greater than 12","user":{"login":"10110346","id":24688163,"node_id":"MDQ6VXNlcjI0Njg4MTYz","avatar_url":"https://avatars2.githubusercontent.com/u/24688163?v=4","gravatar_id":"","url":"https://api.github.com/users/10110346","html_url":"https://github.com/10110346","followers_url":"https://api.github.com/users/10110346/followers","following_url":"https://api.github.com/users/10110346/following{/other_user}","gists_url":"https://api.github.com/users/10110346/gists{/gist_id}","starred_url":"https://api.github.com/users/10110346/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/10110346/subscriptions","organizations_url":"https://api.github.com/users/10110346/orgs","repos_url":"https://api.github.com/users/10110346/repos","events_url":"https://api.github.com/users/10110346/events{/privacy}","received_events_url":"https://api.github.com/users/10110346/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":28,"created_at":"2018-10-17T07:11:36Z","updated_at":"2018-11-04T06:28:22Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22754","html_url":"https://github.com/apache/spark/pull/22754","diff_url":"https://github.com/apache/spark/pull/22754.diff","patch_url":"https://github.com/apache/spark/pull/22754.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n In `UnsafeSorterSpillWriter.java`, when we write a record to a spill file wtih ` void write(Object baseObject, long baseOffset,  int recordLength, long keyPrefix)`, `recordLength` and `keyPrefix`  will be  written  the disk write buffer  first, and these will take 12 bytes, so the disk write buffer size must be greater than 12.\r\n\r\n If `diskWriteBufferSize` is  10, it will print this exception info:\r\n\r\n_java.lang.ArrayIndexOutOfBoundsException: 10 \r\n   at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillWriter.writeLongToBuffer (UnsafeSorterSpillWriter.java:91)\r\n\tat org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillWriter.write(UnsafeSorterSpillWriter.java:123)\r\n\tat org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spillIterator(UnsafeExternalSorter.java:498)\r\n\tat org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill(UnsafeExternalSorter.java:222)\r\n\tat org.apache.spark.memory.MemoryConsumer.spill(MemoryConsumer.java:65)_\r\n\r\n## How was this patch tested?\r\nExisting UT in `UnsafeExternalSorterSuite`\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/435645737","html_url":"https://github.com/apache/spark/pull/22754#issuecomment-435645737","issue_url":"https://api.github.com/repos/apache/spark/issues/22754","id":435645737,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNTY0NTczNw==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-11-04T06:28:22Z","updated_at":"2018-11-04T06:28:22Z","author_association":"NONE","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98434/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}