{"action":"created","issue":{"url":"https://api.github.com/repos/netty/netty/issues/3559","repository_url":"https://api.github.com/repos/netty/netty","labels_url":"https://api.github.com/repos/netty/netty/issues/3559/labels{/name}","comments_url":"https://api.github.com/repos/netty/netty/issues/3559/comments","events_url":"https://api.github.com/repos/netty/netty/issues/3559/events","html_url":"https://github.com/netty/netty/issues/3559","id":65662943,"number":3559,"title":"OutOfMemory error while doing a file upload of size 7GB through Netty","user":{"login":"karuturi","id":186833,"avatar_url":"https://avatars.githubusercontent.com/u/186833?v=3","gravatar_id":"","url":"https://api.github.com/users/karuturi","html_url":"https://github.com/karuturi","followers_url":"https://api.github.com/users/karuturi/followers","following_url":"https://api.github.com/users/karuturi/following{/other_user}","gists_url":"https://api.github.com/users/karuturi/gists{/gist_id}","starred_url":"https://api.github.com/users/karuturi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karuturi/subscriptions","organizations_url":"https://api.github.com/users/karuturi/orgs","repos_url":"https://api.github.com/users/karuturi/repos","events_url":"https://api.github.com/users/karuturi/events{/privacy}","received_events_url":"https://api.github.com/users/karuturi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-04-01T11:24:11Z","updated_at":"2016-06-07T17:23:08Z","closed_at":null,"body":"This is the exception I see\r\n\r\n    java.lang.OutOfMemoryError: Java heap space\r\n    \tat io.netty.buffer.UnpooledHeapByteBuf.capacity(UnpooledHeapByteBuf.java:114)\r\n    \tat io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251)\r\n    \tat io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849)\r\n    \tat io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841)\r\n    \tat io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831)\r\n    \tat io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.offer(HttpPostMultipartRequestDecoder.java:337)\r\n    \tat io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.offer(HttpPostMultipartRequestDecoder.java:48)\r\n    \tat io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.offer(HttpPostRequestDecoder.java:236)\r\n    \tat org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:200)\r\n    \tat org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)\r\n    \tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)\r\n    \tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)\r\n    \tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)\r\n    \tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n    \tat io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)\r\n    \tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)\r\n    \tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)\r\n    \tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:182)\r\n    \tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)\r\n    \tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)\r\n    \tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)\r\n    \tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)\r\n    \tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)\r\n    \tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)\r\n    \tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)\r\n    \tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)\r\n    \tat io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)\r\n    \tat io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)\r\n    \tat java.lang.Thread.run(Thread.java:745)\r\n\r\nI see that its trying to allocate bytes for new capacity. \r\n\r\n    byte[] newArray = new byte[newCapacity];\r\n\r\nBut, why is this happening only for big files? Is it not releasing the existing memory?\r\nIs there any max file size limit?"},"comment":{"url":"https://api.github.com/repos/netty/netty/issues/comments/224351930","html_url":"https://github.com/netty/netty/issues/3559#issuecomment-224351930","issue_url":"https://api.github.com/repos/netty/netty/issues/3559","id":224351930,"user":{"login":"rfielding","id":89911,"avatar_url":"https://avatars.githubusercontent.com/u/89911?v=3","gravatar_id":"","url":"https://api.github.com/users/rfielding","html_url":"https://github.com/rfielding","followers_url":"https://api.github.com/users/rfielding/followers","following_url":"https://api.github.com/users/rfielding/following{/other_user}","gists_url":"https://api.github.com/users/rfielding/gists{/gist_id}","starred_url":"https://api.github.com/users/rfielding/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rfielding/subscriptions","organizations_url":"https://api.github.com/users/rfielding/orgs","repos_url":"https://api.github.com/users/rfielding/repos","events_url":"https://api.github.com/users/rfielding/events{/privacy}","received_events_url":"https://api.github.com/users/rfielding/received_events","type":"User","site_admin":false},"created_at":"2016-06-07T17:23:08Z","updated_at":"2016-06-07T17:23:08Z","body":"@jknack I had run into not only this problem last year, but all the dependents on this code that *also* make this assumption (netty -> Finagle/Finatra -> user apps).  I gave up on the framework eventually.  This issue was closed, so you can assume that it will never be fixed.\r\n\r\nhttp4s downloaded a 47GB file - though I didn't try uploading such a large file.  Or you can use a raw http connection and implement your own multipart handling (it's not that hard if you get rid of the layers of abstraction).  Actually, I eventually moved on to Go, where I could control the details of the http connection completely.  The problem is that you only need buffers large enough to absorb incoming traffic without creating delays, which is really only a few kB, and stream them the way that http was designed.  "}}
{"actor":{"display_login":"rfielding"}}