{"action":"created","issue":{"url":"https://api.github.com/repos/twitter/heron/issues/2803","repository_url":"https://api.github.com/repos/twitter/heron","labels_url":"https://api.github.com/repos/twitter/heron/issues/2803/labels{/name}","comments_url":"https://api.github.com/repos/twitter/heron/issues/2803/comments","events_url":"https://api.github.com/repos/twitter/heron/issues/2803/events","html_url":"https://github.com/twitter/heron/issues/2803","id":307826570,"number":2803,"title":"Memory allocation issue with setComponentRam() assigned between 192 - 256 MB","user":{"login":"aahmed-se","id":8009085,"avatar_url":"https://avatars0.githubusercontent.com/u/8009085?v=4","gravatar_id":"","url":"https://api.github.com/users/aahmed-se","html_url":"https://github.com/aahmed-se","followers_url":"https://api.github.com/users/aahmed-se/followers","following_url":"https://api.github.com/users/aahmed-se/following{/other_user}","gists_url":"https://api.github.com/users/aahmed-se/gists{/gist_id}","starred_url":"https://api.github.com/users/aahmed-se/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aahmed-se/subscriptions","organizations_url":"https://api.github.com/users/aahmed-se/orgs","repos_url":"https://api.github.com/users/aahmed-se/repos","events_url":"https://api.github.com/users/aahmed-se/events{/privacy}","received_events_url":"https://api.github.com/users/aahmed-se/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-03-22T21:46:50Z","updated_at":"2018-03-28T22:10:10Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"setting up a topology results in an error within heron\r\n```\r\ncom.twitter.heron.api.Config.setComponentRam(conf, \"sentence\", ByteAmount.fromMegabytes(192));\r\n```\r\n\r\nthe problem only seems to be with memory range 192-256MB .\r\n\r\n```\r\nheron_internals.yaml --override_config_file=./heron-conf/override.yaml --ckptmgr_port=58984 --ckptmgr_id=ckptmgr-1\r\n[2018-03-22 14:42:05 -0700] [INFO]: container_1_split_2 stdout: Invalid maximum heap size: -Xmx0M\r\n[2018-03-22 14:42:05 -0700] [INFO]: container_1_split_2 stdout: Error: Could not create the Java Virtual Machine.\r\n[2018-03-22 14:42:05 -0700] [INFO]: container_1_split_2 stdout: Error: A fatal exception has occurred. Program will exit.\r\n[2018-03-22 14:42:05 -0700] [INFO]: container_1_split_2 stdout: \r\n[2018-03-22 14:42:05 -0700] [INFO]: Logging pid 94266 to file stmgr-1.pi\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/twitter/heron/issues/comments/377054530","html_url":"https://github.com/twitter/heron/issues/2803#issuecomment-377054530","issue_url":"https://api.github.com/repos/twitter/heron/issues/2803","id":377054530,"user":{"login":"ashvina","id":10034060,"avatar_url":"https://avatars1.githubusercontent.com/u/10034060?v=4","gravatar_id":"","url":"https://api.github.com/users/ashvina","html_url":"https://github.com/ashvina","followers_url":"https://api.github.com/users/ashvina/followers","following_url":"https://api.github.com/users/ashvina/following{/other_user}","gists_url":"https://api.github.com/users/ashvina/gists{/gist_id}","starred_url":"https://api.github.com/users/ashvina/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ashvina/subscriptions","organizations_url":"https://api.github.com/users/ashvina/orgs","repos_url":"https://api.github.com/users/ashvina/repos","events_url":"https://api.github.com/users/ashvina/events{/privacy}","received_events_url":"https://api.github.com/users/ashvina/received_events","type":"User","site_admin":false},"created_at":"2018-03-28T22:10:10Z","updated_at":"2018-03-28T22:10:10Z","author_association":"COLLABORATOR","body":"This issue is not caused by RCRR packing.\r\n\r\nI launched a topology with `192` ram allocated to a bolt `counter`. I found the following in the executor logs.\r\n```\r\n[INFO]: component name: counter, ram request: 201326592, total jvm size: 192M, cache size: 64M, metaspace size: 128M\r\n```\r\nThe executor correctly received ram size. Which means packing worked as expected.\r\n\r\nHowever the issue still exists, because the process fails to launch:\r\n```\r\n[INFO]: container_1_counter_3 stdout: Invalid maximum heap size: -Xmx0M\r\n```\r\n\r\nThe culprit is https://github.com/twitter/heron/blob/master/heron/executor/src/python/heron_executor.py#L562\r\n\r\n```\r\n    # TO DO (Karthik) to be moved into keys and defaults files\r\n    code_cache_size_mb = 64\r\n    java_metasize_mb = 128\r\n...\r\n      total_jvm_size = int(self.component_ram_map[component_name] / (1024 * 1024))\r\n      heap_size_mb = total_jvm_size - code_cache_size_mb - java_metasize_mb\r\n      Log.info(\"component name: %s, ram request: %d, total jvm size: %dM, \"\r\n               \"cache size: %dM, metaspace size: %dM\"\r\n               % (component_name, self.component_ram_map[component_name],\r\n                  total_jvm_size, code_cache_size_mb, java_metasize_mb))\r\n```\r\n\r\n@kramasamy may know more about the reason.\r\n\r\n@jerrypeng @nwangtw, you may want look at this issue before adding any new configs to correctly estimate jvm size."}}
{"actor":{"display_login":"ashvina"}}