{"action":"created","comment":{"url":"https://api.github.com/repos/Tribler/tribler/pulls/comments/59237303","id":59237303,"diff_hunk":"@@ -1 +1 @@\n-Subproject commit d056b388836cec227f6752683045534325b5b5e4\n+Subproject commit 67c22ed23f0e079d775cbe6df05d907b1282fc24","path":"Tribler/dispersy","position":2,"original_position":2,"commit_id":"64c77443510d08d49e5bb0b349bb2ba8d51e545b","original_commit_id":"64c77443510d08d49e5bb0b349bb2ba8d51e545b","user":{"login":"whirm","id":529494,"avatar_url":"https://avatars.githubusercontent.com/u/529494?v=3","gravatar_id":"","url":"https://api.github.com/users/whirm","html_url":"https://github.com/whirm","followers_url":"https://api.github.com/users/whirm/followers","following_url":"https://api.github.com/users/whirm/following{/other_user}","gists_url":"https://api.github.com/users/whirm/gists{/gist_id}","starred_url":"https://api.github.com/users/whirm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whirm/subscriptions","organizations_url":"https://api.github.com/users/whirm/orgs","repos_url":"https://api.github.com/users/whirm/repos","events_url":"https://api.github.com/users/whirm/events{/privacy}","received_events_url":"https://api.github.com/users/whirm/received_events","type":"User","site_admin":false},"body":"I merged the test, so that should be 57ef0d210d98f2d9e41fca01fb48ef355efacf3a","created_at":"2016-04-11T16:38:55Z","updated_at":"2016-04-11T16:38:55Z","html_url":"https://github.com/Tribler/tribler/pull/2062#discussion_r59237303","pull_request_url":"https://api.github.com/repos/Tribler/tribler/pulls/2062","_links":{"self":{"href":"https://api.github.com/repos/Tribler/tribler/pulls/comments/59237303"},"html":{"href":"https://github.com/Tribler/tribler/pull/2062#discussion_r59237303"},"pull_request":{"href":"https://api.github.com/repos/Tribler/tribler/pulls/2062"}}},"pull_request":{"url":"https://api.github.com/repos/Tribler/tribler/pulls/2062","id":64844608,"html_url":"https://github.com/Tribler/tribler/pull/2062","diff_url":"https://github.com/Tribler/tribler/pull/2062.diff","patch_url":"https://github.com/Tribler/tribler/pull/2062.patch","issue_url":"https://api.github.com/repos/Tribler/tribler/issues/2062","number":2062,"state":"open","locked":false,"title":"Fixed a memory issue when running the tunnel helpers","user":{"login":"devos50","id":1707075,"avatar_url":"https://avatars.githubusercontent.com/u/1707075?v=3","gravatar_id":"","url":"https://api.github.com/users/devos50","html_url":"https://github.com/devos50","followers_url":"https://api.github.com/users/devos50/followers","following_url":"https://api.github.com/users/devos50/following{/other_user}","gists_url":"https://api.github.com/users/devos50/gists{/gist_id}","starred_url":"https://api.github.com/users/devos50/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/devos50/subscriptions","organizations_url":"https://api.github.com/users/devos50/orgs","repos_url":"https://api.github.com/users/devos50/repos","events_url":"https://api.github.com/users/devos50/events{/privacy}","received_events_url":"https://api.github.com/users/devos50/received_events","type":"User","site_admin":false},"body":"After a day of memory investigation, I found the root cause of the memory usage problem in the tunnel helpers when they are running for a long time.\r\n\r\nThe problem is that the Twisted log observer is intercepting lower-level Twisted events. When Twisted starts, a `LimitedHistoryLogObserver` is created, however, the name is very misleading since the observed history is not limited at all! An entry is written to the log, every time something interesting happens in the Tunnel community (for instance, the reactor starts or stops listening on a specific port or the `TunnelExitSocket` protocol starts).\r\n\r\nSearching on the internet, brought me to this issue: [here](http://twistedmatrix.com/trac/ticket/7841). A workaround has been implemented [here](https://github.com/plq/neurons/blob/417836a729706f2d1449109ccbb7d4a510227050/neurons/daemon/config.py#L623). I basically took their approach with some minor changes. For instance, I don't replay the logs in `LimitedHistoryLogObserver`.\r\n\r\nThe fact that the memory is increasing faster when the community is running as exit node can easily be explained since we are opening more tunnels thus writing more log entries.\r\n\r\nIf you’re interested in how I found this exactly, I’ve written a blog post about it [here](http://blog.code-up.nl/finding-python-memory-peaks-using-meliae/).\r\n\r\nFixes #2000 ","created_at":"2016-03-31T19:26:20Z","updated_at":"2016-04-11T16:38:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"515883581a9f8ff6c8ea469b00bb3c74be612379","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/Tribler/tribler/pulls/2062/commits","review_comments_url":"https://api.github.com/repos/Tribler/tribler/pulls/2062/comments","review_comment_url":"https://api.github.com/repos/Tribler/tribler/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Tribler/tribler/issues/2062/comments","statuses_url":"https://api.github.com/repos/Tribler/tribler/statuses/64c77443510d08d49e5bb0b349bb2ba8d51e545b","head":{"label":"devos50:tunnel_helpers_memory_fix","ref":"tunnel_helpers_memory_fix","sha":"64c77443510d08d49e5bb0b349bb2ba8d51e545b","user":{"login":"devos50","id":1707075,"avatar_url":"https://avatars.githubusercontent.com/u/1707075?v=3","gravatar_id":"","url":"https://api.github.com/users/devos50","html_url":"https://github.com/devos50","followers_url":"https://api.github.com/users/devos50/followers","following_url":"https://api.github.com/users/devos50/following{/other_user}","gists_url":"https://api.github.com/users/devos50/gists{/gist_id}","starred_url":"https://api.github.com/users/devos50/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/devos50/subscriptions","organizations_url":"https://api.github.com/users/devos50/orgs","repos_url":"https://api.github.com/users/devos50/repos","events_url":"https://api.github.com/users/devos50/events{/privacy}","received_events_url":"https://api.github.com/users/devos50/received_events","type":"User","site_admin":false},"repo":{"id":45194074,"name":"tribler","full_name":"devos50/tribler","owner":{"login":"devos50","id":1707075,"avatar_url":"https://avatars.githubusercontent.com/u/1707075?v=3","gravatar_id":"","url":"https://api.github.com/users/devos50","html_url":"https://github.com/devos50","followers_url":"https://api.github.com/users/devos50/followers","following_url":"https://api.github.com/users/devos50/following{/other_user}","gists_url":"https://api.github.com/users/devos50/gists{/gist_id}","starred_url":"https://api.github.com/users/devos50/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/devos50/subscriptions","organizations_url":"https://api.github.com/users/devos50/orgs","repos_url":"https://api.github.com/users/devos50/repos","events_url":"https://api.github.com/users/devos50/events{/privacy}","received_events_url":"https://api.github.com/users/devos50/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/devos50/tribler","description":"Privacy with BitTorrent and resilient to shut down","fork":true,"url":"https://api.github.com/repos/devos50/tribler","forks_url":"https://api.github.com/repos/devos50/tribler/forks","keys_url":"https://api.github.com/repos/devos50/tribler/keys{/key_id}","collaborators_url":"https://api.github.com/repos/devos50/tribler/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/devos50/tribler/teams","hooks_url":"https://api.github.com/repos/devos50/tribler/hooks","issue_events_url":"https://api.github.com/repos/devos50/tribler/issues/events{/number}","events_url":"https://api.github.com/repos/devos50/tribler/events","assignees_url":"https://api.github.com/repos/devos50/tribler/assignees{/user}","branches_url":"https://api.github.com/repos/devos50/tribler/branches{/branch}","tags_url":"https://api.github.com/repos/devos50/tribler/tags","blobs_url":"https://api.github.com/repos/devos50/tribler/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/devos50/tribler/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/devos50/tribler/git/refs{/sha}","trees_url":"https://api.github.com/repos/devos50/tribler/git/trees{/sha}","statuses_url":"https://api.github.com/repos/devos50/tribler/statuses/{sha}","languages_url":"https://api.github.com/repos/devos50/tribler/languages","stargazers_url":"https://api.github.com/repos/devos50/tribler/stargazers","contributors_url":"https://api.github.com/repos/devos50/tribler/contributors","subscribers_url":"https://api.github.com/repos/devos50/tribler/subscribers","subscription_url":"https://api.github.com/repos/devos50/tribler/subscription","commits_url":"https://api.github.com/repos/devos50/tribler/commits{/sha}","git_commits_url":"https://api.github.com/repos/devos50/tribler/git/commits{/sha}","comments_url":"https://api.github.com/repos/devos50/tribler/comments{/number}","issue_comment_url":"https://api.github.com/repos/devos50/tribler/issues/comments{/number}","contents_url":"https://api.github.com/repos/devos50/tribler/contents/{+path}","compare_url":"https://api.github.com/repos/devos50/tribler/compare/{base}...{head}","merges_url":"https://api.github.com/repos/devos50/tribler/merges","archive_url":"https://api.github.com/repos/devos50/tribler/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/devos50/tribler/downloads","issues_url":"https://api.github.com/repos/devos50/tribler/issues{/number}","pulls_url":"https://api.github.com/repos/devos50/tribler/pulls{/number}","milestones_url":"https://api.github.com/repos/devos50/tribler/milestones{/number}","notifications_url":"https://api.github.com/repos/devos50/tribler/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/devos50/tribler/labels{/name}","releases_url":"https://api.github.com/repos/devos50/tribler/releases{/id}","deployments_url":"https://api.github.com/repos/devos50/tribler/deployments","created_at":"2015-10-29T15:57:50Z","updated_at":"2015-11-20T16:00:03Z","pushed_at":"2016-04-11T16:30:38Z","git_url":"git://github.com/devos50/tribler.git","ssh_url":"git@github.com:devos50/tribler.git","clone_url":"https://github.com/devos50/tribler.git","svn_url":"https://github.com/devos50/tribler","homepage":"http://www.tribler.org","size":113501,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"devel"}},"base":{"label":"Tribler:devel","ref":"devel","sha":"a2f1238bda6f2ecad51f386cfc0e6cfec3b84b93","user":{"login":"Tribler","id":2913489,"avatar_url":"https://avatars.githubusercontent.com/u/2913489?v=3","gravatar_id":"","url":"https://api.github.com/users/Tribler","html_url":"https://github.com/Tribler","followers_url":"https://api.github.com/users/Tribler/followers","following_url":"https://api.github.com/users/Tribler/following{/other_user}","gists_url":"https://api.github.com/users/Tribler/gists{/gist_id}","starred_url":"https://api.github.com/users/Tribler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Tribler/subscriptions","organizations_url":"https://api.github.com/users/Tribler/orgs","repos_url":"https://api.github.com/users/Tribler/repos","events_url":"https://api.github.com/users/Tribler/events{/privacy}","received_events_url":"https://api.github.com/users/Tribler/received_events","type":"Organization","site_admin":false},"repo":{"id":8411137,"name":"tribler","full_name":"Tribler/tribler","owner":{"login":"Tribler","id":2913489,"avatar_url":"https://avatars.githubusercontent.com/u/2913489?v=3","gravatar_id":"","url":"https://api.github.com/users/Tribler","html_url":"https://github.com/Tribler","followers_url":"https://api.github.com/users/Tribler/followers","following_url":"https://api.github.com/users/Tribler/following{/other_user}","gists_url":"https://api.github.com/users/Tribler/gists{/gist_id}","starred_url":"https://api.github.com/users/Tribler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Tribler/subscriptions","organizations_url":"https://api.github.com/users/Tribler/orgs","repos_url":"https://api.github.com/users/Tribler/repos","events_url":"https://api.github.com/users/Tribler/events{/privacy}","received_events_url":"https://api.github.com/users/Tribler/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Tribler/tribler","description":"Privacy with BitTorrent and resilient to shut down","fork":false,"url":"https://api.github.com/repos/Tribler/tribler","forks_url":"https://api.github.com/repos/Tribler/tribler/forks","keys_url":"https://api.github.com/repos/Tribler/tribler/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Tribler/tribler/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Tribler/tribler/teams","hooks_url":"https://api.github.com/repos/Tribler/tribler/hooks","issue_events_url":"https://api.github.com/repos/Tribler/tribler/issues/events{/number}","events_url":"https://api.github.com/repos/Tribler/tribler/events","assignees_url":"https://api.github.com/repos/Tribler/tribler/assignees{/user}","branches_url":"https://api.github.com/repos/Tribler/tribler/branches{/branch}","tags_url":"https://api.github.com/repos/Tribler/tribler/tags","blobs_url":"https://api.github.com/repos/Tribler/tribler/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Tribler/tribler/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Tribler/tribler/git/refs{/sha}","trees_url":"https://api.github.com/repos/Tribler/tribler/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Tribler/tribler/statuses/{sha}","languages_url":"https://api.github.com/repos/Tribler/tribler/languages","stargazers_url":"https://api.github.com/repos/Tribler/tribler/stargazers","contributors_url":"https://api.github.com/repos/Tribler/tribler/contributors","subscribers_url":"https://api.github.com/repos/Tribler/tribler/subscribers","subscription_url":"https://api.github.com/repos/Tribler/tribler/subscription","commits_url":"https://api.github.com/repos/Tribler/tribler/commits{/sha}","git_commits_url":"https://api.github.com/repos/Tribler/tribler/git/commits{/sha}","comments_url":"https://api.github.com/repos/Tribler/tribler/comments{/number}","issue_comment_url":"https://api.github.com/repos/Tribler/tribler/issues/comments{/number}","contents_url":"https://api.github.com/repos/Tribler/tribler/contents/{+path}","compare_url":"https://api.github.com/repos/Tribler/tribler/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Tribler/tribler/merges","archive_url":"https://api.github.com/repos/Tribler/tribler/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Tribler/tribler/downloads","issues_url":"https://api.github.com/repos/Tribler/tribler/issues{/number}","pulls_url":"https://api.github.com/repos/Tribler/tribler/pulls{/number}","milestones_url":"https://api.github.com/repos/Tribler/tribler/milestones{/number}","notifications_url":"https://api.github.com/repos/Tribler/tribler/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Tribler/tribler/labels{/name}","releases_url":"https://api.github.com/repos/Tribler/tribler/releases{/id}","deployments_url":"https://api.github.com/repos/Tribler/tribler/deployments","created_at":"2013-02-25T14:51:58Z","updated_at":"2016-04-09T00:22:26Z","pushed_at":"2016-04-11T16:31:50Z","git_url":"git://github.com/Tribler/tribler.git","ssh_url":"git@github.com:Tribler/tribler.git","clone_url":"https://github.com/Tribler/tribler.git","svn_url":"https://github.com/Tribler/tribler","homepage":"http://www.tribler.org","size":126180,"stargazers_count":1851,"watchers_count":1851,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":197,"mirror_url":null,"open_issues_count":240,"forks":197,"open_issues":240,"watchers":1851,"default_branch":"devel"}},"_links":{"self":{"href":"https://api.github.com/repos/Tribler/tribler/pulls/2062"},"html":{"href":"https://github.com/Tribler/tribler/pull/2062"},"issue":{"href":"https://api.github.com/repos/Tribler/tribler/issues/2062"},"comments":{"href":"https://api.github.com/repos/Tribler/tribler/issues/2062/comments"},"review_comments":{"href":"https://api.github.com/repos/Tribler/tribler/pulls/2062/comments"},"review_comment":{"href":"https://api.github.com/repos/Tribler/tribler/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Tribler/tribler/pulls/2062/commits"},"statuses":{"href":"https://api.github.com/repos/Tribler/tribler/statuses/64c77443510d08d49e5bb0b349bb2ba8d51e545b"}}}}