{"push_id":1471978304,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"07fcbea516cda66498b9346467a34733f14e8605","before":"3cff8161578b65139c9740fed694d4b3c81fa74a","commits":[{"sha":"07fcbea516cda66498b9346467a34733f14e8605","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-18800][SQL] Correct the assert in UnsafeKVExternalSorter which ensures array size\n\n## What changes were proposed in this pull request?\n\n`UnsafeKVExternalSorter` uses `UnsafeInMemorySorter` to sort the records of `BytesToBytesMap` if it is given a map.\n\nCurrently we use the number of keys in `BytesToBytesMap` to determine if the array used for sort is enough or not. We has an assert that ensures the size of the array is enough: `map.numKeys() <= map.getArray().size() / 2`.\n\nHowever, each record in the map takes two entries in the array, one is record pointer, another is key prefix. So the correct assert should be `map.numKeys() * 2 <= map.getArray().size() / 2`.\n\n## How was this patch tested?\n\nN/A\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #16232 from viirya/SPARK-18800-fix-UnsafeKVExternalSorter.","distinct":true,"url":"https://api.github.com/repos/wangyum/spark/commits/07fcbea516cda66498b9346467a34733f14e8605"}]}
{"actor":{"display_login":"wangyum"}}