{"push_id":1258357528,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"cf0cce90364d17afe780ff9a5426dfcefa298535","before":"071eaaf9d2b63589f2e66e5279a16a5a484de6f5","commits":[{"sha":"cf0cce90364d17afe780ff9a5426dfcefa298535","author":{"name":"Sital Kedia","email":"e4343f82e25883a7c046475fe9553138fac5aa32@fb.com"},"message":"[SPARK-17113] [SHUFFLE] Job failure due to Executor OOM in offheap mode\n\n## What changes were proposed in this pull request?\n\nThis PR fixes executor OOM in offheap mode due to bug in Cooperative Memory Management for UnsafeExternSorter.  UnsafeExternalSorter was checking if memory page is being used by upstream by comparing the base object address of the current page with the base object address of upstream. However, in case of offheap memory allocation, the base object addresses are always null, so there was no spilling happening and eventually the operator would OOM.\n\nFollowing is the stack trace this issue addresses -\njava.lang.OutOfMemoryError: Unable to acquire 1220 bytes of memory, got 0\n\tat org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:120)\n\tat org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.acquireNewPageIfNecessary(UnsafeExternalSorter.java:341)\n\tat org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.insertRecord(UnsafeExternalSorter.java:362)\n\tat org.apache.spark.sql.execution.UnsafeExternalRowSorter.insertRow(UnsafeExternalRowSorter.java:93)\n\tat org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:170)\n\n## How was this patch tested?\n\nTested by running the failing job.\n\nAuthor: Sital Kedia <skedia@fb.com>\n\nCloses #14693 from sitalkedia/fix_offheap_oom.","distinct":true,"url":"https://api.github.com/repos/apache/spark/commits/cf0cce90364d17afe780ff9a5426dfcefa298535"}]}
{"actor":{"display_login":"asfgit"}}