{"action":"created","issue":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/367","labels_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/367/labels{/name}","comments_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/367/comments","events_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/367/events","html_url":"https://github.com/spark-jobserver/spark-jobserver/issues/367","id":127206211,"number":367,"title":"OutOfMemoryError: Direct buffer memory","user":{"login":"LHaferkamp","id":5586401,"avatar_url":"https://avatars.githubusercontent.com/u/5586401?v=3","gravatar_id":"","url":"https://api.github.com/users/LHaferkamp","html_url":"https://github.com/LHaferkamp","followers_url":"https://api.github.com/users/LHaferkamp/followers","following_url":"https://api.github.com/users/LHaferkamp/following{/other_user}","gists_url":"https://api.github.com/users/LHaferkamp/gists{/gist_id}","starred_url":"https://api.github.com/users/LHaferkamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LHaferkamp/subscriptions","organizations_url":"https://api.github.com/users/LHaferkamp/orgs","repos_url":"https://api.github.com/users/LHaferkamp/repos","events_url":"https://api.github.com/users/LHaferkamp/events{/privacy}","received_events_url":"https://api.github.com/users/LHaferkamp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-01-18T11:19:40Z","updated_at":"2016-01-22T12:19:34Z","closed_at":null,"body":"Currently running jobs via Spark jobserver yields OutOfMemory errors as shown below, but submitting the same job via spark-submit runs smoothly. \r\nSpark Monitoring tells me:\r\n\"Job aborted due to stage failure: Exception while getting task result: org.apache.spark.storage.BlockFetchException: Failed to fetch block from 1 locations.\"\r\nin org.apache.spark.api.java.AbstractJavaRDDLike.mapToPair(JavaRDDLike.scala:46)\r\nThe logs show an OutOfMemory as reason:\r\n```\r\n16/01/18 07:53:16 WARN server.TransportChannelHandler: Exception in connection from XXXXX.org/XXX.XX.XX.XXX:47133\r\njava.lang.OutOfMemoryError: Direct buffer memory\r\n\tat java.nio.Bits.reserveMemory(Bits.java:658)\r\n\tat java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)\r\n\tat java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)\r\n\tat io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:651)\r\n\tat io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:237)\r\n\tat io.netty.buffer.PoolArena.allocate(PoolArena.java:215)\r\n\tat io.netty.buffer.PoolArena.allocate(PoolArena.java:132)\r\n\tat io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:271)\r\n\tat io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)\r\n\tat io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146)\r\n\tat io.netty.buffer.CompositeByteBuf.allocBuffer(CompositeByteBuf.java:1345)\r\n\tat io.netty.buffer.CompositeByteBuf.consolidateIfNeeded(CompositeByteBuf.java:276)\r\n\tat io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:116)\r\n\tat org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)\r\n\tat org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:82)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)\r\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)\r\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n16/01/18 07:53:17 ERROR scheduler.TaskSchedulerImpl: Lost executor 0 on XXXXX.org: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.\r\n```\r\nI had this problem running Spark 1.4 and upgraded then to Spark 1.6 due to some hints to memory leaks in Netty (https://issues.apache.org/jira/browse/SPARK-8101), but the same problem still exists. \r\nPlayed also around with various memory settings and partitioning which didn't help. \r\nCurious why its happening only when using Spark jobserver and not at spark-submit."},"comment":{"url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/comments/173902697","html_url":"https://github.com/spark-jobserver/spark-jobserver/issues/367#issuecomment-173902697","issue_url":"https://api.github.com/repos/spark-jobserver/spark-jobserver/issues/367","id":173902697,"user":{"login":"LHaferkamp","id":5586401,"avatar_url":"https://avatars.githubusercontent.com/u/5586401?v=3","gravatar_id":"","url":"https://api.github.com/users/LHaferkamp","html_url":"https://github.com/LHaferkamp","followers_url":"https://api.github.com/users/LHaferkamp/followers","following_url":"https://api.github.com/users/LHaferkamp/following{/other_user}","gists_url":"https://api.github.com/users/LHaferkamp/gists{/gist_id}","starred_url":"https://api.github.com/users/LHaferkamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LHaferkamp/subscriptions","organizations_url":"https://api.github.com/users/LHaferkamp/orgs","repos_url":"https://api.github.com/users/LHaferkamp/repos","events_url":"https://api.github.com/users/LHaferkamp/events{/privacy}","received_events_url":"https://api.github.com/users/LHaferkamp/received_events","type":"User","site_admin":false},"created_at":"2016-01-22T12:19:34Z","updated_at":"2016-01-22T12:19:34Z","body":"After carefully reading http://spark.apache.org/docs/latest/configuration.html and reducing total size of serialized results to `spark.driver.maxResultSize = \"1G\"` in `spark.context-settings` of job configuration it seems to be working. I'm currently checking if it remains stable."}}