{"action":"created","comment":{"url":"https://api.github.com/repos/root-project/root/pulls/comments/209126392","pull_request_review_id":145082370,"id":209126392,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIwOTEyNjM5Mg==","diff_hunk":"@@ -725,20 +740,48 @@ void TBasket::Reset()\n          }\n       }\n    }\n-   /*\n-      Philippe has asked us to keep this turned off until we finish memory fragmentation studies.\n-   // If fBufferRef grew since we last saw it, shrink it to 105% of the occupied size\n-   if (curSize > fLastWriteBufferSize) {\n-      if (newSize == -1) {\n-         newSize = Int_t(1.05*Float_t(fBufferRef->Length()));\n+   // If fBufferRef grew since we last saw it, shrink it to \"target memory ratio\" of the occupied size\n+   // This discourages us from having poorly-occupied buffers on branches with little variability.\n+   //\n+   // Does not help protect against a burst in event sizes, but does help in the cases where the basket\n+   // size jumps from 4MB to 8MB while filling the basket, but we only end up utilizing 4.1MB.\n+   //\n+   // The above code block is meant to protect against extremely large events.\n+\n+   Float_t target_mem_ratio = fBranch->GetTree()->GetTargetMemoryRatio();\n+   Int_t max_size = std::max(fLastWriteBufferSize[0], std::max(fLastWriteBufferSize[1], fLastWriteBufferSize[2]));","path":"tree/tree/src/TBasket.cxx","position":59,"original_position":59,"commit_id":"f7804c619dfda74259968a780c06e55aa3caa472","original_commit_id":"fdf92afae2828343f857e1f21a7b8a5695ffe463","user":{"login":"bbockelm","id":1093447,"node_id":"MDQ6VXNlcjEwOTM0NDc=","avatar_url":"https://avatars2.githubusercontent.com/u/1093447?v=4","gravatar_id":"","url":"https://api.github.com/users/bbockelm","html_url":"https://github.com/bbockelm","followers_url":"https://api.github.com/users/bbockelm/followers","following_url":"https://api.github.com/users/bbockelm/following{/other_user}","gists_url":"https://api.github.com/users/bbockelm/gists{/gist_id}","starred_url":"https://api.github.com/users/bbockelm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bbockelm/subscriptions","organizations_url":"https://api.github.com/users/bbockelm/orgs","repos_url":"https://api.github.com/users/bbockelm/repos","events_url":"https://api.github.com/users/bbockelm/events{/privacy}","received_events_url":"https://api.github.com/users/bbockelm/received_events","type":"User","site_admin":false},"body":"Maybe I confused this with another aspect of Oksana's work - but the idea was to simplify some of the interdependencies between modules as part of the \"ROOT base\" effort.\r\n\r\nAnyhow, I can make the change here.","created_at":"2018-08-10T01:52:17Z","updated_at":"2018-08-10T01:52:17Z","html_url":"https://github.com/root-project/root/pull/774#discussion_r209126392","pull_request_url":"https://api.github.com/repos/root-project/root/pulls/774","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/root-project/root/pulls/comments/209126392"},"html":{"href":"https://github.com/root-project/root/pull/774#discussion_r209126392"},"pull_request":{"href":"https://api.github.com/repos/root-project/root/pulls/774"}},"in_reply_to_id":208437918},"pull_request":{"url":"https://api.github.com/repos/root-project/root/pulls/774","id":130621978,"node_id":"MDExOlB1bGxSZXF1ZXN0MTMwNjIxOTc4","html_url":"https://github.com/root-project/root/pull/774","diff_url":"https://github.com/root-project/root/pull/774.diff","patch_url":"https://github.com/root-project/root/pull/774.patch","issue_url":"https://api.github.com/repos/root-project/root/issues/774","number":774,"state":"open","locked":false,"title":"[WIP] Add \"one-basket-per-cluster\" mode.","user":{"login":"bbockelm","id":1093447,"node_id":"MDQ6VXNlcjEwOTM0NDc=","avatar_url":"https://avatars2.githubusercontent.com/u/1093447?v=4","gravatar_id":"","url":"https://api.github.com/users/bbockelm","html_url":"https://github.com/bbockelm","followers_url":"https://api.github.com/users/bbockelm/followers","following_url":"https://api.github.com/users/bbockelm/following{/other_user}","gists_url":"https://api.github.com/users/bbockelm/gists{/gist_id}","starred_url":"https://api.github.com/users/bbockelm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bbockelm/subscriptions","organizations_url":"https://api.github.com/users/bbockelm/orgs","repos_url":"https://api.github.com/users/bbockelm/repos","events_url":"https://api.github.com/users/bbockelm/events{/privacy}","received_events_url":"https://api.github.com/users/bbockelm/received_events","type":"User","site_admin":false},"body":"By setting a new bit in the TTree object, `kFlushAtCluster`, one can enter the \"one basket per event cluster\" mode.  This forces baskets to line up with event clusters, at the cost of extra memory use.\r\n\r\nThis new mode simplifies the basket layout within the file, reducing the amount of time handling special cases for the bulk IO mode.\r\n\r\nBecause the baskets *must* grow to the size of an event cluster, we do not invoke `OptimizeBaskets` when one-basket-per-cluster mode is enabled.\r\n\r\nAs this mode is expected to cause increased memory usage (the memory utilized by the `TTree` is more strongly tied to the variations in event size), we combine this with a technique borrowed from a CMS patchset to more aggressively shrink basket sizes after very large objects.\r\n\r\nThe new basket shrinking algorithm will trigger whenever the basket is flushed.  If the actual object size in the last clusters is significantly below the buffer size, then the basket will be shrunk.  Given this tradeoff, I do not currently see this being enabled by default.\r\n\r\nThe ideal ratio of `(basket buffer size)/(occupied buffer size)` is controlled by a new tunable in the `TTree`, defaulting to 1.1.  A lower setting reduces overall memory usage at the cost of extra allocations; a higher setting increases aggregate memory usage.\r\n\r\nIn writing out a 10k event CMS file (total CPU time is 32 minutes):\r\n\r\n- Base case (without this patch):\r\n   - 888MB RSS\r\n   - 30 reallocations (shrinking of baskets due to low occupancy).\r\n   - 0.173ms taken for reallocation.\r\n- New shrinking algorithm (this patch with defaults):\r\n   - 866MB RSS\r\n   - 4434 reallocations\r\n   - 97.0ms\r\n- One basket per cluster mode with new shrinking algorithm:\r\n   - 902MB RSS\r\n   - 2882 reallocations\r\n   - 93.6ms\r\n\r\nThe CPU-time cost of the reallocation is 0.005% of total runtime (considering the file has to be read also, maybe 0.01% of output time?)\r\n\r\nThis patch purposely leaves in the code that measures reallocation time in order to allow others to experiment; the intent is to disable / remove the code before a final release.","created_at":"2017-07-14T16:08:27Z","updated_at":"2018-08-10T01:52:17Z","closed_at":null,"merged_at":null,"merge_commit_sha":"5d807e736750fb7c2d2d11c1a09059d2edda6d59","assignee":{"login":"pcanal","id":5175087,"node_id":"MDQ6VXNlcjUxNzUwODc=","avatar_url":"https://avatars3.githubusercontent.com/u/5175087?v=4","gravatar_id":"","url":"https://api.github.com/users/pcanal","html_url":"https://github.com/pcanal","followers_url":"https://api.github.com/users/pcanal/followers","following_url":"https://api.github.com/users/pcanal/following{/other_user}","gists_url":"https://api.github.com/users/pcanal/gists{/gist_id}","starred_url":"https://api.github.com/users/pcanal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcanal/subscriptions","organizations_url":"https://api.github.com/users/pcanal/orgs","repos_url":"https://api.github.com/users/pcanal/repos","events_url":"https://api.github.com/users/pcanal/events{/privacy}","received_events_url":"https://api.github.com/users/pcanal/received_events","type":"User","site_admin":false},"assignees":[{"login":"pcanal","id":5175087,"node_id":"MDQ6VXNlcjUxNzUwODc=","avatar_url":"https://avatars3.githubusercontent.com/u/5175087?v=4","gravatar_id":"","url":"https://api.github.com/users/pcanal","html_url":"https://github.com/pcanal","followers_url":"https://api.github.com/users/pcanal/followers","following_url":"https://api.github.com/users/pcanal/following{/other_user}","gists_url":"https://api.github.com/users/pcanal/gists{/gist_id}","starred_url":"https://api.github.com/users/pcanal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcanal/subscriptions","organizations_url":"https://api.github.com/users/pcanal/orgs","repos_url":"https://api.github.com/users/pcanal/repos","events_url":"https://api.github.com/users/pcanal/events{/privacy}","received_events_url":"https://api.github.com/users/pcanal/received_events","type":"User","site_admin":false}],"requested_reviewers":[{"login":"Axel-Naumann","id":144675,"node_id":"MDQ6VXNlcjE0NDY3NQ==","avatar_url":"https://avatars1.githubusercontent.com/u/144675?v=4","gravatar_id":"","url":"https://api.github.com/users/Axel-Naumann","html_url":"https://github.com/Axel-Naumann","followers_url":"https://api.github.com/users/Axel-Naumann/followers","following_url":"https://api.github.com/users/Axel-Naumann/following{/other_user}","gists_url":"https://api.github.com/users/Axel-Naumann/gists{/gist_id}","starred_url":"https://api.github.com/users/Axel-Naumann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Axel-Naumann/subscriptions","organizations_url":"https://api.github.com/users/Axel-Naumann/orgs","repos_url":"https://api.github.com/users/Axel-Naumann/repos","events_url":"https://api.github.com/users/Axel-Naumann/events{/privacy}","received_events_url":"https://api.github.com/users/Axel-Naumann/received_events","type":"User","site_admin":false},{"login":"amadio","id":249404,"node_id":"MDQ6VXNlcjI0OTQwNA==","avatar_url":"https://avatars2.githubusercontent.com/u/249404?v=4","gravatar_id":"","url":"https://api.github.com/users/amadio","html_url":"https://github.com/amadio","followers_url":"https://api.github.com/users/amadio/followers","following_url":"https://api.github.com/users/amadio/following{/other_user}","gists_url":"https://api.github.com/users/amadio/gists{/gist_id}","starred_url":"https://api.github.com/users/amadio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amadio/subscriptions","organizations_url":"https://api.github.com/users/amadio/orgs","repos_url":"https://api.github.com/users/amadio/repos","events_url":"https://api.github.com/users/amadio/events{/privacy}","received_events_url":"https://api.github.com/users/amadio/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[{"id":651913541,"node_id":"MDU6TGFiZWw2NTE5MTM1NDE=","url":"https://api.github.com/repos/root-project/root/labels/do%20not%20merge","name":"do not merge","color":"b60205","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/root-project/root/pulls/774/commits","review_comments_url":"https://api.github.com/repos/root-project/root/pulls/774/comments","review_comment_url":"https://api.github.com/repos/root-project/root/pulls/comments{/number}","comments_url":"https://api.github.com/repos/root-project/root/issues/774/comments","statuses_url":"https://api.github.com/repos/root-project/root/statuses/f7804c619dfda74259968a780c06e55aa3caa472","head":{"label":"bbockelm:one_basket_per_cluster_v2","ref":"one_basket_per_cluster_v2","sha":"f7804c619dfda74259968a780c06e55aa3caa472","user":{"login":"bbockelm","id":1093447,"node_id":"MDQ6VXNlcjEwOTM0NDc=","avatar_url":"https://avatars2.githubusercontent.com/u/1093447?v=4","gravatar_id":"","url":"https://api.github.com/users/bbockelm","html_url":"https://github.com/bbockelm","followers_url":"https://api.github.com/users/bbockelm/followers","following_url":"https://api.github.com/users/bbockelm/following{/other_user}","gists_url":"https://api.github.com/users/bbockelm/gists{/gist_id}","starred_url":"https://api.github.com/users/bbockelm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bbockelm/subscriptions","organizations_url":"https://api.github.com/users/bbockelm/orgs","repos_url":"https://api.github.com/users/bbockelm/repos","events_url":"https://api.github.com/users/bbockelm/events{/privacy}","received_events_url":"https://api.github.com/users/bbockelm/received_events","type":"User","site_admin":false},"repo":{"id":32016931,"node_id":"MDEwOlJlcG9zaXRvcnkzMjAxNjkzMQ==","name":"root","full_name":"bbockelm/root","owner":{"login":"bbockelm","id":1093447,"node_id":"MDQ6VXNlcjEwOTM0NDc=","avatar_url":"https://avatars2.githubusercontent.com/u/1093447?v=4","gravatar_id":"","url":"https://api.github.com/users/bbockelm","html_url":"https://github.com/bbockelm","followers_url":"https://api.github.com/users/bbockelm/followers","following_url":"https://api.github.com/users/bbockelm/following{/other_user}","gists_url":"https://api.github.com/users/bbockelm/gists{/gist_id}","starred_url":"https://api.github.com/users/bbockelm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bbockelm/subscriptions","organizations_url":"https://api.github.com/users/bbockelm/orgs","repos_url":"https://api.github.com/users/bbockelm/repos","events_url":"https://api.github.com/users/bbockelm/events{/privacy}","received_events_url":"https://api.github.com/users/bbockelm/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/bbockelm/root","description":"Mirror of official ROOT git repository","fork":true,"url":"https://api.github.com/repos/bbockelm/root","forks_url":"https://api.github.com/repos/bbockelm/root/forks","keys_url":"https://api.github.com/repos/bbockelm/root/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bbockelm/root/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bbockelm/root/teams","hooks_url":"https://api.github.com/repos/bbockelm/root/hooks","issue_events_url":"https://api.github.com/repos/bbockelm/root/issues/events{/number}","events_url":"https://api.github.com/repos/bbockelm/root/events","assignees_url":"https://api.github.com/repos/bbockelm/root/assignees{/user}","branches_url":"https://api.github.com/repos/bbockelm/root/branches{/branch}","tags_url":"https://api.github.com/repos/bbockelm/root/tags","blobs_url":"https://api.github.com/repos/bbockelm/root/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bbockelm/root/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bbockelm/root/git/refs{/sha}","trees_url":"https://api.github.com/repos/bbockelm/root/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bbockelm/root/statuses/{sha}","languages_url":"https://api.github.com/repos/bbockelm/root/languages","stargazers_url":"https://api.github.com/repos/bbockelm/root/stargazers","contributors_url":"https://api.github.com/repos/bbockelm/root/contributors","subscribers_url":"https://api.github.com/repos/bbockelm/root/subscribers","subscription_url":"https://api.github.com/repos/bbockelm/root/subscription","commits_url":"https://api.github.com/repos/bbockelm/root/commits{/sha}","git_commits_url":"https://api.github.com/repos/bbockelm/root/git/commits{/sha}","comments_url":"https://api.github.com/repos/bbockelm/root/comments{/number}","issue_comment_url":"https://api.github.com/repos/bbockelm/root/issues/comments{/number}","contents_url":"https://api.github.com/repos/bbockelm/root/contents/{+path}","compare_url":"https://api.github.com/repos/bbockelm/root/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bbockelm/root/merges","archive_url":"https://api.github.com/repos/bbockelm/root/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bbockelm/root/downloads","issues_url":"https://api.github.com/repos/bbockelm/root/issues{/number}","pulls_url":"https://api.github.com/repos/bbockelm/root/pulls{/number}","milestones_url":"https://api.github.com/repos/bbockelm/root/milestones{/number}","notifications_url":"https://api.github.com/repos/bbockelm/root/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bbockelm/root/labels{/name}","releases_url":"https://api.github.com/repos/bbockelm/root/releases{/id}","deployments_url":"https://api.github.com/repos/bbockelm/root/deployments","created_at":"2015-03-11T12:41:26Z","updated_at":"2017-03-01T14:12:58Z","pushed_at":"2018-08-10T01:51:24Z","git_url":"git://github.com/bbockelm/root.git","ssh_url":"git@github.com:bbockelm/root.git","clone_url":"https://github.com/bbockelm/root.git","svn_url":"https://github.com/bbockelm/root","homepage":"http://root.cern.ch","size":701695,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"lgpl-2.1","name":"GNU Lesser General Public License v2.1","spdx_id":"LGPL-2.1","url":"https://api.github.com/licenses/lgpl-2.1","node_id":"MDc6TGljZW5zZTEx"},"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"root-project:master","ref":"master","sha":"3ebc64b2063b619e69c3ebfda660f738973fb835","user":{"login":"root-project","id":6450093,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY0NTAwOTM=","avatar_url":"https://avatars1.githubusercontent.com/u/6450093?v=4","gravatar_id":"","url":"https://api.github.com/users/root-project","html_url":"https://github.com/root-project","followers_url":"https://api.github.com/users/root-project/followers","following_url":"https://api.github.com/users/root-project/following{/other_user}","gists_url":"https://api.github.com/users/root-project/gists{/gist_id}","starred_url":"https://api.github.com/users/root-project/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/root-project/subscriptions","organizations_url":"https://api.github.com/users/root-project/orgs","repos_url":"https://api.github.com/users/root-project/repos","events_url":"https://api.github.com/users/root-project/events{/privacy}","received_events_url":"https://api.github.com/users/root-project/received_events","type":"Organization","site_admin":false},"repo":{"id":10994345,"node_id":"MDEwOlJlcG9zaXRvcnkxMDk5NDM0NQ==","name":"root","full_name":"root-project/root","owner":{"login":"root-project","id":6450093,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY0NTAwOTM=","avatar_url":"https://avatars1.githubusercontent.com/u/6450093?v=4","gravatar_id":"","url":"https://api.github.com/users/root-project","html_url":"https://github.com/root-project","followers_url":"https://api.github.com/users/root-project/followers","following_url":"https://api.github.com/users/root-project/following{/other_user}","gists_url":"https://api.github.com/users/root-project/gists{/gist_id}","starred_url":"https://api.github.com/users/root-project/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/root-project/subscriptions","organizations_url":"https://api.github.com/users/root-project/orgs","repos_url":"https://api.github.com/users/root-project/repos","events_url":"https://api.github.com/users/root-project/events{/privacy}","received_events_url":"https://api.github.com/users/root-project/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/root-project/root","description":"The official repository for ROOT: analyzing, storing and visualizing big data","fork":false,"url":"https://api.github.com/repos/root-project/root","forks_url":"https://api.github.com/repos/root-project/root/forks","keys_url":"https://api.github.com/repos/root-project/root/keys{/key_id}","collaborators_url":"https://api.github.com/repos/root-project/root/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/root-project/root/teams","hooks_url":"https://api.github.com/repos/root-project/root/hooks","issue_events_url":"https://api.github.com/repos/root-project/root/issues/events{/number}","events_url":"https://api.github.com/repos/root-project/root/events","assignees_url":"https://api.github.com/repos/root-project/root/assignees{/user}","branches_url":"https://api.github.com/repos/root-project/root/branches{/branch}","tags_url":"https://api.github.com/repos/root-project/root/tags","blobs_url":"https://api.github.com/repos/root-project/root/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/root-project/root/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/root-project/root/git/refs{/sha}","trees_url":"https://api.github.com/repos/root-project/root/git/trees{/sha}","statuses_url":"https://api.github.com/repos/root-project/root/statuses/{sha}","languages_url":"https://api.github.com/repos/root-project/root/languages","stargazers_url":"https://api.github.com/repos/root-project/root/stargazers","contributors_url":"https://api.github.com/repos/root-project/root/contributors","subscribers_url":"https://api.github.com/repos/root-project/root/subscribers","subscription_url":"https://api.github.com/repos/root-project/root/subscription","commits_url":"https://api.github.com/repos/root-project/root/commits{/sha}","git_commits_url":"https://api.github.com/repos/root-project/root/git/commits{/sha}","comments_url":"https://api.github.com/repos/root-project/root/comments{/number}","issue_comment_url":"https://api.github.com/repos/root-project/root/issues/comments{/number}","contents_url":"https://api.github.com/repos/root-project/root/contents/{+path}","compare_url":"https://api.github.com/repos/root-project/root/compare/{base}...{head}","merges_url":"https://api.github.com/repos/root-project/root/merges","archive_url":"https://api.github.com/repos/root-project/root/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/root-project/root/downloads","issues_url":"https://api.github.com/repos/root-project/root/issues{/number}","pulls_url":"https://api.github.com/repos/root-project/root/pulls{/number}","milestones_url":"https://api.github.com/repos/root-project/root/milestones{/number}","notifications_url":"https://api.github.com/repos/root-project/root/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/root-project/root/labels{/name}","releases_url":"https://api.github.com/repos/root-project/root/releases{/id}","deployments_url":"https://api.github.com/repos/root-project/root/deployments","created_at":"2013-06-27T10:46:59Z","updated_at":"2018-08-09T21:21:56Z","pushed_at":"2018-08-10T01:51:26Z","git_url":"git://github.com/root-project/root.git","ssh_url":"git@github.com:root-project/root.git","clone_url":"https://github.com/root-project/root.git","svn_url":"https://github.com/root-project/root","homepage":"https://root.cern","size":706032,"stargazers_count":621,"watchers_count":621,"language":"C++","has_issues":false,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":446,"mirror_url":null,"archived":false,"open_issues_count":80,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":446,"open_issues":80,"watchers":621,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/root-project/root/pulls/774"},"html":{"href":"https://github.com/root-project/root/pull/774"},"issue":{"href":"https://api.github.com/repos/root-project/root/issues/774"},"comments":{"href":"https://api.github.com/repos/root-project/root/issues/774/comments"},"review_comments":{"href":"https://api.github.com/repos/root-project/root/pulls/774/comments"},"review_comment":{"href":"https://api.github.com/repos/root-project/root/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/root-project/root/pulls/774/commits"},"statuses":{"href":"https://api.github.com/repos/root-project/root/statuses/f7804c619dfda74259968a780c06e55aa3caa472"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"bbockelm"}}