{"action":"created","issue":{"url":"https://api.github.com/repos/pravega/pravega/issues/3957","repository_url":"https://api.github.com/repos/pravega/pravega","labels_url":"https://api.github.com/repos/pravega/pravega/issues/3957/labels{/name}","comments_url":"https://api.github.com/repos/pravega/pravega/issues/3957/comments","events_url":"https://api.github.com/repos/pravega/pravega/issues/3957/events","html_url":"https://github.com/pravega/pravega/issues/3957","id":460368087,"node_id":"MDU6SXNzdWU0NjAzNjgwODc=","number":3957,"title":"Readers failed with \"java.lang.OutOfMemoryError:\" on scaling up segments from 1000 to 5000","user":{"login":"SomeshJoshi19","id":33201406,"node_id":"MDQ6VXNlcjMzMjAxNDA2","avatar_url":"https://avatars1.githubusercontent.com/u/33201406?v=4","gravatar_id":"","url":"https://api.github.com/users/SomeshJoshi19","html_url":"https://github.com/SomeshJoshi19","followers_url":"https://api.github.com/users/SomeshJoshi19/followers","following_url":"https://api.github.com/users/SomeshJoshi19/following{/other_user}","gists_url":"https://api.github.com/users/SomeshJoshi19/gists{/gist_id}","starred_url":"https://api.github.com/users/SomeshJoshi19/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SomeshJoshi19/subscriptions","organizations_url":"https://api.github.com/users/SomeshJoshi19/orgs","repos_url":"https://api.github.com/users/SomeshJoshi19/repos","events_url":"https://api.github.com/users/SomeshJoshi19/events{/privacy}","received_events_url":"https://api.github.com/users/SomeshJoshi19/received_events","type":"User","site_admin":false},"labels":[{"id":406474440,"node_id":"MDU6TGFiZWw0MDY0NzQ0NDA=","url":"https://api.github.com/repos/pravega/pravega/labels/kind/bug","name":"kind/bug","color":"b60205","default":false},{"id":520512604,"node_id":"MDU6TGFiZWw1MjA1MTI2MDQ=","url":"https://api.github.com/repos/pravega/pravega/labels/priority/P1","name":"priority/P1","color":"f9d0c4","default":false},{"id":1222704862,"node_id":"MDU6TGFiZWwxMjIyNzA0ODYy","url":"https://api.github.com/repos/pravega/pravega/labels/version/0.5.0","name":"version/0.5.0","color":"5319e7","default":false},{"id":1222707119,"node_id":"MDU6TGFiZWwxMjIyNzA3MTE5","url":"https://api.github.com/repos/pravega/pravega/labels/version/0.6.0","name":"version/0.6.0","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":{"login":"shrids","id":11114165,"node_id":"MDQ6VXNlcjExMTE0MTY1","avatar_url":"https://avatars1.githubusercontent.com/u/11114165?v=4","gravatar_id":"","url":"https://api.github.com/users/shrids","html_url":"https://github.com/shrids","followers_url":"https://api.github.com/users/shrids/followers","following_url":"https://api.github.com/users/shrids/following{/other_user}","gists_url":"https://api.github.com/users/shrids/gists{/gist_id}","starred_url":"https://api.github.com/users/shrids/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shrids/subscriptions","organizations_url":"https://api.github.com/users/shrids/orgs","repos_url":"https://api.github.com/users/shrids/repos","events_url":"https://api.github.com/users/shrids/events{/privacy}","received_events_url":"https://api.github.com/users/shrids/received_events","type":"User","site_admin":false},"assignees":[{"login":"shrids","id":11114165,"node_id":"MDQ6VXNlcjExMTE0MTY1","avatar_url":"https://avatars1.githubusercontent.com/u/11114165?v=4","gravatar_id":"","url":"https://api.github.com/users/shrids","html_url":"https://github.com/shrids","followers_url":"https://api.github.com/users/shrids/followers","following_url":"https://api.github.com/users/shrids/following{/other_user}","gists_url":"https://api.github.com/users/shrids/gists{/gist_id}","starred_url":"https://api.github.com/users/shrids/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shrids/subscriptions","organizations_url":"https://api.github.com/users/shrids/orgs","repos_url":"https://api.github.com/users/shrids/repos","events_url":"https://api.github.com/users/shrids/events{/privacy}","received_events_url":"https://api.github.com/users/shrids/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2019-06-25T11:29:07Z","updated_at":"2019-06-28T15:30:45Z","closed_at":null,"author_association":"NONE","body":"**Problem description**\r\n\r\nOn scaling up number of segments from 1000 to 5000, observed `java.lang.OutOfMemoryError: Java heap space`  error and `SegmentTruncatedException`  warning messages.\r\n\r\n**6 parallel Pravega streams** from 6 separate client pods were running during the scaling operation with IO workload supplied by each stream is: - 1 worker ( 1 reader & 1 writer ), 100 events/sec, 100 bytes data and run time as 600 secs.\r\n\r\n1) Writing operation passed for all the 6  streams running parallelly, matching the expected results: -\r\n```\r\n59112 records Writing, 95.269 records/sec, 100 bytes record size, 0.01 MB/sec, 33656.9 ms avg latency, 183810.0 ms max latency, 3083 ms 50th, 18136 ms 75th, 39423 ms 95th, 44866 ms 99th, 46285 ms 99.9th, 46453 ms 99.99th.\r\n```\r\n2) Reading operation failed for all the 6 streams running together with OOM error: -\r\n```\r\n[grpc-default-executor-0] INFO io.pravega.client.netty.impl.ConnectionPoolImpl - Reusing connection: Connection(uri=PravegaNodeUri(endpoint=172.24.24.16, port=12345), flowHandler=io.pravega.client.netty.impl.FlowHandler@3a86bc1e, connected=java.util.concurrent.CompletableFuture@7608718d[Completed normally])\r\n[grpc-default-executor-0] INFO io.pravega.client.netty.impl.FlowHandler - Creating Flow 2174 for endpoint 172.24.24.16. The current Channel is [id: 0x77e74122, L:/172.24.24.17:50908 - R:172.24.24.16/172.24.24.16:12345].\r\njava.lang.OutOfMemoryError: Java heap space\r\nDumping heap to java_pid1161.hprof ...\r\nHeap dump file created [1262610900 bytes in 1.261 secs]\r\n[grpc-default-executor-1] INFO io.pravega.client.netty.impl.ConnectionPoolImpl - Reusing connection: Connection(uri=PravegaNodeUri(endpoint=172.24.24.15, port=12345), flowHandler=io.pravega.client.netty.impl.FlowHandler@6ae82baa, connected=java.util.concurrent.CompletableFuture@5b00f831[Completed normally])\r\n[grpc-default-executor-1] INFO io.pravega.client.netty.impl.FlowHandler - Creating Flow 2175 for endpoint 172.24.24.15. The current Channel is [id: 0x587b2178, L:/172.24.24.17:59724 - R:172.24.24.15/172.24.24.15:12345].\r\n```\r\nand SegmentTruncationException: -\r\n\r\n```\r\n[epollEventLoopGroup-4-2] INFO io.pravega.client.netty.impl.FlowHandler - Connection established with endpoint 172.24.24.15 on ChannelId [id: 0x5d4f6c92]\r\n[epollEventLoopGroup-4-1] INFO io.pravega.client.segment.impl.AsyncSegmentInputStreamImpl - Received segmentIsTruncated WireCommands.SegmentIsTruncated(type=SEGMENT_IS_TRUNCATED, requestId=4294967296, segment=Scope/_RGitr2Test1/0.#epoch.0, startOffset=571922, serverStackTrace=, offset=0)\r\n[epollEventLoopGroup-4-1] WARN io.pravega.client.segment.impl.AsyncSegmentInputStreamImpl - Exception while reading from Segment : Scope/_RGitr2Test1/0.#epoch.0\r\nio.pravega.client.segment.impl.SegmentTruncatedException\r\n        at io.pravega.client.segment.impl.AsyncSegmentInputStreamImpl$ResponseProcessor.segmentIsTruncated(AsyncSegmentInputStreamImpl.java:87)\r\n        at io.pravega.shared.protocol.netty.WireCommands$SegmentIsTruncated.process(WireCommands.java:202)\r\n        at io.pravega.shared.protocol.netty.ReplyProcessor.process(ReplyProcessor.java:20)\r\n        at io.pravega.client.netty.impl.FlowHandler.lambda$channelRead$2(FlowHandler.java:209)\r\n        at java.util.Optional.ifPresent(Optional.java:159)\r\n        at io.pravega.client.netty.impl.FlowHandler.channelRead(FlowHandler.java:207)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\r\n        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\r\n        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)\r\n        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\r\n        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\r\n        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)\r\n        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\r\n        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\r\n        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)\r\n        at io.pravega.shared.protocol.netty.ExceptionLoggingHandler.channelRead(ExceptionLoggingHandler.java:37)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\r\n        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\r\n        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\r\n        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)\r\n        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)\r\n        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)\r\n        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)\r\n        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)\r\n        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\r\n        at java.lang.Thread.run(Thread.java:748)\r\n```\r\n3) The heap dump analysis suspects instances from `io.pravega.client.stream.impl.EventStreamReaderImpl` class for memory leakage problem.\r\nClass Histogram report: -\r\n```\r\n\r\nClass Name                                                                  | Objects | Shallow Heap |    Retained Heap\r\n------------------------------------------------------------------------------------------------------------------------\r\nio.pravega.common.util.Retry$RetryWithBackoff                               |   1,175 |       47,000 |        >= 47,000\r\nio.pravega.client.netty.impl.ClientConnectionImpl                           |   1,170 |       37,440 |        >= 57,232\r\nio.pravega.client.segment.impl.AsyncSegmentInputStreamImpl                  |   1,167 |       74,688 |       >= 598,640\r\nio.pravega.client.segment.impl.SegmentInputStreamImpl                       |   1,167 |       65,352 | >= 1,223,365,864\r\nio.pravega.common.util.CircularBuffer                                       |   1,167 |       28,008 | >= 1,222,798,424\r\nio.pravega.client.segment.impl.AsyncSegmentInputStreamImpl$ResponseProcessor|   1,167 |       18,672 |        >= 19,080\r\nio.pravega.client.segment.impl.EventSegmentReaderImpl                       |   1,166 |       27,984 | >= 1,223,494,704\r\nio.pravega.shared.protocol.netty.WireCommands$SegmentRead                   |     966 |       46,368 |       >= 451,872\r\nio.pravega.client.segment.impl.Segment                                      |   6,004 |      192,128 |       >= 912,136\r\nio.pravega.common.util.Retry$RetryAndThrowBase$$Lambda$24             \t    |     201 |        8,040 |        >= 13,208\r\nio.pravega.common.concurrent.Futures$Loop                                   |     201 |        6,432 |        >= 28,544\r\nio.pravega.common.util.Retry$RetryAndThrowBase$$Lambda$23                   |     201 |        3,216 |         >= 3,216\r\nio.pravega.shared.protocol.netty.WireCommands$ReadSegment                   |     200 |        9,600 |        >= 30,456\r\nio.pravega.common.util.Retry$RetryAndThrowBase$$Lambda$35                   |     200 |        6,400 |         >= 6,400\r\nio.pravega.client.segment.impl.AsyncSegmentInputStreamImpl$$Lambda$148      |     200 |        4,800 |         >= 4,952\r\nio.pravega.common.util.Retry$RetryAndThrowConditionally                     |     200 |        4,800 |         >= 8,000\r\n------------------------------------------------------------------------------------------------------------------------\r\n```\r\n4) None of the client pod were evicted due to lack of resources. The resource allocation for all the 6 pods: -\r\n```\r\n        Started:      Mon, 24 Jun 2019 10:53:09 +0530\r\n    Ready:          True\r\n    Restart Count:  0\r\n    Limits:\r\n      cpu:     4\r\n      memory:  6Gi\r\n    Requests:\r\n      cpu:        2\r\n      memory:     4Gi\r\n```\r\n\r\nPravega-Operator deployment with 2 controllers, 3 Zookeepers, 4 Bookies and 3 Segment stores.\r\nEnvironment details: PKS / K8 with medium cluster:\r\n```\r\n1 master nodes @ large.cpu (4 CPU, 4 GB Ram, 16 GB Disk)\r\n3 worker nodes @ xlarge.cpu(4 cpu, 16 GB Ram, 32 GB Disk)\r\nTier-1 storage is from VSAN datastore\r\nTier-2 storage curved on NFS Client provisioner using Isilon as backend\r\n```\r\nPravega details:\r\n```\r\nPravega version : 0.5.0-2269.6f8a820\r\nZookeeper Operator : pravega/zookeeper-operator:0.2.1\r\nPravega Operator: pravega/pravega-operator:0.3.2\r\n```"},"comment":{"url":"https://api.github.com/repos/pravega/pravega/issues/comments/506776727","html_url":"https://github.com/pravega/pravega/issues/3957#issuecomment-506776727","issue_url":"https://api.github.com/repos/pravega/pravega/issues/3957","id":506776727,"node_id":"MDEyOklzc3VlQ29tbWVudDUwNjc3NjcyNw==","user":{"login":"SomeshJoshi19","id":33201406,"node_id":"MDQ6VXNlcjMzMjAxNDA2","avatar_url":"https://avatars1.githubusercontent.com/u/33201406?v=4","gravatar_id":"","url":"https://api.github.com/users/SomeshJoshi19","html_url":"https://github.com/SomeshJoshi19","followers_url":"https://api.github.com/users/SomeshJoshi19/followers","following_url":"https://api.github.com/users/SomeshJoshi19/following{/other_user}","gists_url":"https://api.github.com/users/SomeshJoshi19/gists{/gist_id}","starred_url":"https://api.github.com/users/SomeshJoshi19/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SomeshJoshi19/subscriptions","organizations_url":"https://api.github.com/users/SomeshJoshi19/orgs","repos_url":"https://api.github.com/users/SomeshJoshi19/repos","events_url":"https://api.github.com/users/SomeshJoshi19/events{/privacy}","received_events_url":"https://api.github.com/users/SomeshJoshi19/received_events","type":"User","site_admin":false},"created_at":"2019-06-28T15:30:45Z","updated_at":"2019-06-28T15:30:45Z","author_association":"NONE","body":"@tkaitchuck minimum heap size allocation was 512 mb (xms) but maximum limit (xmx) value was not given while performing this stream run"}}
{"actor":{"display_login":"SomeshJoshi19"}}