{"action":"created","issue":{"url":"https://api.github.com/repos/Tribler/tribler/issues/2062","repository_url":"https://api.github.com/repos/Tribler/tribler","labels_url":"https://api.github.com/repos/Tribler/tribler/issues/2062/labels{/name}","comments_url":"https://api.github.com/repos/Tribler/tribler/issues/2062/comments","events_url":"https://api.github.com/repos/Tribler/tribler/issues/2062/events","html_url":"https://github.com/Tribler/tribler/pull/2062","id":144994302,"number":2062,"title":"Fixed a memory issue when running the tunnel helpers","user":{"login":"devos50","id":1707075,"avatar_url":"https://avatars.githubusercontent.com/u/1707075?v=3","gravatar_id":"","url":"https://api.github.com/users/devos50","html_url":"https://github.com/devos50","followers_url":"https://api.github.com/users/devos50/followers","following_url":"https://api.github.com/users/devos50/following{/other_user}","gists_url":"https://api.github.com/users/devos50/gists{/gist_id}","starred_url":"https://api.github.com/users/devos50/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/devos50/subscriptions","organizations_url":"https://api.github.com/users/devos50/orgs","repos_url":"https://api.github.com/users/devos50/repos","events_url":"https://api.github.com/users/devos50/events{/privacy}","received_events_url":"https://api.github.com/users/devos50/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Tribler/tribler/labels/needs%20discussion","name":"needs discussion","color":"006b75"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-03-31T19:26:20Z","updated_at":"2016-04-11T15:55:01Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Tribler/tribler/pulls/2062","html_url":"https://github.com/Tribler/tribler/pull/2062","diff_url":"https://github.com/Tribler/tribler/pull/2062.diff","patch_url":"https://github.com/Tribler/tribler/pull/2062.patch"},"body":"After a day of memory investigation, I found the root cause of the memory usage problem in the tunnel helpers when they are running for a long time.\r\n\r\nThe problem is that the Twisted log observer is intercepting lower-level Twisted events. When Twisted starts, a `LimitedHistoryLogObserver` is created, however, the name is very misleading since the observed history is not limited at all! An entry is written to the log, every time something interesting happens in the Tunnel community (for instance, the reactor starts or stops listening on a specific port or the `TunnelExitSocket` protocol starts).\r\n\r\nSearching on the internet, brought me to this issue: [here](http://twistedmatrix.com/trac/ticket/7841). A workaround has been implemented [here](https://github.com/plq/neurons/blob/417836a729706f2d1449109ccbb7d4a510227050/neurons/daemon/config.py#L623). I basically took their approach with some minor changes. For instance, I don't replay the logs in `LimitedHistoryLogObserver`.\r\n\r\nThe fact that the memory is increasing faster when the community is running as exit node can easily be explained since we are opening more tunnels thus writing more log entries.\r\n\r\nIf you’re interested in how I found this exactly, I’ve written a blog post about it [here](http://blog.code-up.nl/finding-python-memory-peaks-using-meliae/).\r\n\r\nFixes #2000 "},"comment":{"url":"https://api.github.com/repos/Tribler/tribler/issues/comments/208417968","html_url":"https://github.com/Tribler/tribler/pull/2062#issuecomment-208417968","issue_url":"https://api.github.com/repos/Tribler/tribler/issues/2062","id":208417968,"user":{"login":"whirm","id":529494,"avatar_url":"https://avatars.githubusercontent.com/u/529494?v=3","gravatar_id":"","url":"https://api.github.com/users/whirm","html_url":"https://github.com/whirm","followers_url":"https://api.github.com/users/whirm/followers","following_url":"https://api.github.com/users/whirm/following{/other_user}","gists_url":"https://api.github.com/users/whirm/gists{/gist_id}","starred_url":"https://api.github.com/users/whirm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whirm/subscriptions","organizations_url":"https://api.github.com/users/whirm/orgs","repos_url":"https://api.github.com/users/whirm/repos","events_url":"https://api.github.com/users/whirm/events{/privacy}","received_events_url":"https://api.github.com/users/whirm/received_events","type":"User","site_admin":false},"created_at":"2016-04-11T15:55:01Z","updated_at":"2016-04-11T15:55:01Z","body":"hm, you forgot to move the test to dispersy"}}