{"action":"created","issue":{"url":"https://api.github.com/repos/akka/akka/issues/17547","labels_url":"https://api.github.com/repos/akka/akka/issues/17547/labels{/name}","comments_url":"https://api.github.com/repos/akka/akka/issues/17547/comments","events_url":"https://api.github.com/repos/akka/akka/issues/17547/events","html_url":"https://github.com/akka/akka/issues/17547","id":79073223,"number":17547,"title":"AbstractNodeQueue potential GC leak - many objects get tenured if we have \"bursts\" of messages","user":{"login":"dom54","id":12540594,"avatar_url":"https://avatars.githubusercontent.com/u/12540594?v=3","gravatar_id":"","url":"https://api.github.com/users/dom54","html_url":"https://github.com/dom54","followers_url":"https://api.github.com/users/dom54/followers","following_url":"https://api.github.com/users/dom54/following{/other_user}","gists_url":"https://api.github.com/users/dom54/gists{/gist_id}","starred_url":"https://api.github.com/users/dom54/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dom54/subscriptions","organizations_url":"https://api.github.com/users/dom54/orgs","repos_url":"https://api.github.com/users/dom54/repos","events_url":"https://api.github.com/users/dom54/events{/privacy}","received_events_url":"https://api.github.com/users/dom54/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/akka/akka/labels/1%20-%20triaged","name":"1 - triaged","color":"5319e7"},{"url":"https://api.github.com/repos/akka/akka/labels/t:core","name":"t:core","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/akka/akka/milestones/55","html_url":"https://github.com/akka/akka/milestones/2.4.x","labels_url":"https://api.github.com/repos/akka/akka/milestones/55/labels","id":1257017,"number":55,"title":"2.4.x","description":"","creator":{"login":"patriknw","id":336161,"avatar_url":"https://avatars.githubusercontent.com/u/336161?v=3","gravatar_id":"","url":"https://api.github.com/users/patriknw","html_url":"https://github.com/patriknw","followers_url":"https://api.github.com/users/patriknw/followers","following_url":"https://api.github.com/users/patriknw/following{/other_user}","gists_url":"https://api.github.com/users/patriknw/gists{/gist_id}","starred_url":"https://api.github.com/users/patriknw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/patriknw/subscriptions","organizations_url":"https://api.github.com/users/patriknw/orgs","repos_url":"https://api.github.com/users/patriknw/repos","events_url":"https://api.github.com/users/patriknw/events{/privacy}","received_events_url":"https://api.github.com/users/patriknw/received_events","type":"User","site_admin":false},"open_issues":223,"closed_issues":8,"state":"open","created_at":"2015-08-16T17:29:51Z","updated_at":"2016-01-02T20:55:07Z","due_on":null,"closed_at":null},"comments":28,"created_at":"2015-05-21T16:36:45Z","updated_at":"2016-01-04T08:48:51Z","closed_at":null,"body":"Akka: 2.3.11\r\nScala: 2.11.4\r\n\r\nI have noticed that when using the SingleConsumerOnlyUnboundedMailbox there are excessive amounts of garbage created given a certain frequency of messages.\r\n\r\nI have traced the issue to the AbstractNodeQueue - if you have many actors (say, 500,000) and therefore many queues, **the last node created for the last message sent to each actor will become the tail, and therefore if no new messages are received for a period of a few minutes (or a few minor GCs), will be copied and eventually tenured - for 500,000 actors thats 500,000 node objects**\r\n\r\n```\r\npublic final T poll() {\r\n        final Node<T> next = peekNode();\r\n        if (next == null) return null;\r\n        else {\r\n            final T ret = next.value;\r\n            next.value = null;\r\n            Unsafe.instance.putOrderedObject(this, tailOffset, next); //next will survive GC's if quiet\r\n            return ret;\r\n        }\r\n    }\r\n```\r\n\r\nThis can cause a lot of objects to be copied between survivor spaces (causing it to be very slow) and eventually will leak into tenured space.\r\n\r\nThe problem can be resolved by setting the tail to be a preallocated instance if the queue is empty. However, this means a few additional compareAndSet / Get and therefore has an impact on the performance of the queue. As it seems like its an edge case perhaps its acceptable in most cases (not in mine!)\r\n\r\nCorresponding test to highlight the issue (you can see the tenured heap increasing massively) - use JVM options -Xms256m -Xmx256m -verbose:gc -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=4 and use a config that uses the SingleConsumerOnlyUnboundedMailbox\r\n\r\n```\r\n@Test\r\n\tpublic void testGCLeak() throws InterruptedException {\r\n\t\tActorSystem akka = ActorSystem.create(\"test\");\r\n\t\tActorRef[] refs = new ActorRef[50000];\r\n\t\tfor (int i = 0;i<refs.length;i++) {\r\n\t\t\trefs[i] = akka.actorOf(Props.create(DummyActor.class).withMailbox(\"default-mailbox\")); //DummyActor does nothing\r\n\t\t}\r\n\t\tfor (int i = 0;i<5;i++) {\r\n\t\t\tforceGC(); //force a minor GC 5 times\r\n\t\t}\r\n\t\tSystem.out.println(\"start\");\r\n\t\tfor (int y = 0;y<1000000;y++) {\r\n\t\t\tfor (int i = 0;i<refs.length;i++) {\r\n\t\t\t\t\trefs[i].tell(new Object(), ActorRef.noSender());\r\n\t\t\t}\r\n\t\t\tfor (int i = 0;i<refs.length;i++) {\r\n\t\t\t\trefs[i].tell(new Object(), ActorRef.noSender());\r\n\t\t\t}\r\n\t\t\tfor (int i = 0;i<refs.length;i++) {\r\n\t\t\t\trefs[i].tell(new Object(), ActorRef.noSender());\r\n\t\t\t}\r\n\t\t\tThread.sleep(2000);\r\n\t\t\tfor (int i = 0;i<5;i++) {\r\n\t\t\t\tforceGC(); //force a minor GC 5 times\r\n\t\t\t}\r\n\t\t\tSystem.out.println(\"Iteration \"+y);\r\n\t\t}\r\n\t\t\r\n\t\tSystem.out.println(\"done\");\r\n\t\t\r\n\t}\r\n\tprivate void forceGC() throws InterruptedException {\r\n\t\tfor (int i = 0;i<100000;i++) {\r\n\t\t\tnew Object[1024].toString(); //encourage minor GC\r\n\t\t}\r\n\t}\r\n```\r\n\r\nI am looking for feedback on the correctness and possible optimizations on the fix I have applied to the class... As I have had to remove the lazySet of the node.next in favour of a CAS. And the consumer will be slower when the queue is empty/when consuming the last element of a queue.\r\n\r\nCode changed in AbstractNodeQueue:\r\n```\r\nprivate final Node<T> tailInstance = new Node<T>(null);\r\n\r\nprotected GCFriendlyAbstractNodeQueue() {\r\n\t\t_tailDoNotCallMeDirectly = tailInstance;\r\n\t\tset(tailInstance);\r\n}\r\n\r\n@SuppressWarnings(\"restriction\")\r\n\tpublic final T poll() {\r\n\t\t\r\n\t\tNode<T> tail = null;\r\n\t\tNode<T> current = null;\r\n\t\tNode<T> next = null;\r\n\t\tfor(;;) {\r\n\t\t\t\r\n\t\t\ttail = ((Node<T>)Unsafe.instance.getObjectVolatile(this, tailOffset));\r\n \t\t\tnext = tail.next();\r\n\t\t\tcurrent = get();\r\n\t\t\tif (next != null || current == tail)\r\n\t\t\t\tbreak;\r\n\t\t}\r\n\r\n\t\tif (next == null) \r\n\t\t\treturn null;\r\n\t\telse { \r\n\t\t\tfinal T ret = next.value;\r\n\t\t\tnext.value = null;\r\n\r\n\t\t\tif (current==next) { //no more elements\r\n\t\t\t\tNode<T> tailInstanceNext = tailInstance.next();\r\n\t\t\t\tif (this.compareAndSet(current,tailInstance)) { //if this passes, the current node is the root empty tail\r\n\t\t\t\t\ttailInstance.compareAndSetNext(tailInstanceNext, null); //this may fail but thats OK - either a producer has cleared the previous reference, or we are setting it to null\r\n\t\t\t\t\tUnsafe.instance.putOrderedObject(this, tailOffset, tailInstance); //ready for the next poll, we set the tail to the instance\r\n\t\t\t\t} else { //if it doesnt, a producer has added an element while consuming\r\n\t\t\t\t\tUnsafe.instance.putOrderedObject(this, tailOffset, next); //otherwise, a producer has already added to the linked list (the next node is the tail)\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t\telse {\r\n\t\t\t\tUnsafe.instance.putOrderedObject(this, tailOffset, next);\r\n\t\t\t}\r\n\t\t\treturn ret;\r\n\t\t}\r\n\t}\r\n\r\npublic static class Node<T> {\r\n\t\tpublic T value;\r\n\t\t@SuppressWarnings(\"unused\")\r\n\t\tprivate final AtomicReference<Node<T>> _next = new AtomicReference<GCFriendlyAbstractNodeQueue.Node<T>>(null);\r\n\r\n\t\tpublic Node() {\r\n\t\t\tthis(null);\r\n\t\t}\r\n\r\n\t\tpublic Node(final T value) {\r\n\t\t\tthis.value = value;\r\n\t\t}\r\n\r\n\t\t@SuppressWarnings(\"unchecked\")\r\n\t\tpublic final Node<T> next() {\r\n\t\t\treturn _next.get();\r\n\t\t}\r\n\r\n\t\tprotected final void setNext(final Node<T> newNext) {\r\n\t\t\t_next.set(newNext);\r\n\t\t}\r\n\r\n\t\tprotected boolean compareAndSetNext(final Node<T> oldNode,final Node<T> newNode) {\r\n\t\t\treturn _next.compareAndSet(oldNode,newNode);\r\n\t\t}\r\n\r\n\t\t@Override\r\n\t\tpublic String toString() {\r\n\t\t\treturn super.toString()+\"_next=\" + _next + \"]\";\r\n\t\t}\r\n\t\t\r\n\t\t\r\n\t}\r\n\r\n\r\n```\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/akka/akka/issues/comments/168613971","html_url":"https://github.com/akka/akka/issues/17547#issuecomment-168613971","issue_url":"https://api.github.com/repos/akka/akka/issues/17547","id":168613971,"user":{"login":"huafengw","id":5011399,"avatar_url":"https://avatars.githubusercontent.com/u/5011399?v=3","gravatar_id":"","url":"https://api.github.com/users/huafengw","html_url":"https://github.com/huafengw","followers_url":"https://api.github.com/users/huafengw/followers","following_url":"https://api.github.com/users/huafengw/following{/other_user}","gists_url":"https://api.github.com/users/huafengw/gists{/gist_id}","starred_url":"https://api.github.com/users/huafengw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/huafengw/subscriptions","organizations_url":"https://api.github.com/users/huafengw/orgs","repos_url":"https://api.github.com/users/huafengw/repos","events_url":"https://api.github.com/users/huafengw/events{/privacy}","received_events_url":"https://api.github.com/users/huafengw/received_events","type":"User","site_admin":false},"created_at":"2016-01-04T08:48:51Z","updated_at":"2016-01-04T08:48:51Z","body":"Hi all,\r\n  We built a real-time streaming engine using Akka and when we were doing performance tuning, we met the same problem. Here is the related issue: https://github.com/gearpump/gearpump/issues/1816. Basically, our benchmark will launch about 100 Actors across 4 nodes and continually shuffles messages, the message size is 100 bytes. \r\n\r\n  We adopt the fix and built a new jar file to run our benchmark, and here is the result:\r\n  **Before the fix:** The throughout of our benchmark is **14,600,000** msg/second\r\n  **After the fix:** The throughput increase to **18,000,000** msg/second, also the receive latency reduced 1x.\r\n  So we consider this fix to be very important to heavy-load Akka applications.\r\n  \r\n  "}}