{"action":"created","issue":{"url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/652","labels_url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/652/labels{/name}","comments_url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/652/comments","events_url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/652/events","html_url":"https://github.com/GRIFFINCollaboration/GRSISort/issues/652","id":126538745,"number":652,"title":"Memory usage during fragment tree creation","user":{"login":"jhenderson88","id":10636847,"avatar_url":"https://avatars.githubusercontent.com/u/10636847?v=3","gravatar_id":"","url":"https://api.github.com/users/jhenderson88","html_url":"https://github.com/jhenderson88","followers_url":"https://api.github.com/users/jhenderson88/followers","following_url":"https://api.github.com/users/jhenderson88/following{/other_user}","gists_url":"https://api.github.com/users/jhenderson88/gists{/gist_id}","starred_url":"https://api.github.com/users/jhenderson88/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhenderson88/subscriptions","organizations_url":"https://api.github.com/users/jhenderson88/orgs","repos_url":"https://api.github.com/users/jhenderson88/repos","events_url":"https://api.github.com/users/jhenderson88/events{/privacy}","received_events_url":"https://api.github.com/users/jhenderson88/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":22,"created_at":"2016-01-13T23:15:14Z","updated_at":"2016-01-22T17:23:36Z","closed_at":null,"body":"Working with the latest version of the master. GRIFFIN data (only clovers, no auxiliary detectors). Run number 5759 (grsmid00/data3/griffin/90Sr_2016).\r\n\r\nIf I convert a 4GB midas file to a fragment tree, GRSISort maxes out on my machine's (Belodon) RAM (32GB). Extrapolating to absolute RAM usage implies that to convert a 4GB .mid file to a fragment tree requires almost 40GB of memory. The final fragment tree size is also approximately 4GB. \r\n\r\nBased on the relative sizes of GRIFFIN events (calculated by @SmithJK) in .mid format and FragmentTree format, we expect that the uncompressed FragmentTree file should be about twice the size of the original midas file. Which is still a long way short of what is required to use up 32GB on a 4GB file.\r\n\r\nDo we understand why this is? It sounds like an issue in the unpacking to me (rather than just inefficient use of data types).\r\n\r\nFor large data sets the .mid to FragmentTree conversion seems to be the bottleneck in terms of data processing (see, e.g. Adam's ELog: https://elog.triumf.ca/Tigress/GRIFFIN/4752). \r\n\r\nThe same thing happens when sorting the files on grifuser/grsmid01 (although the memory doesn't max out due to there being more of it)."},"comment":{"url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/comments/173984137","html_url":"https://github.com/GRIFFINCollaboration/GRSISort/issues/652#issuecomment-173984137","issue_url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/652","id":173984137,"user":{"login":"VinzenzBildstein","id":7277216,"avatar_url":"https://avatars.githubusercontent.com/u/7277216?v=3","gravatar_id":"","url":"https://api.github.com/users/VinzenzBildstein","html_url":"https://github.com/VinzenzBildstein","followers_url":"https://api.github.com/users/VinzenzBildstein/followers","following_url":"https://api.github.com/users/VinzenzBildstein/following{/other_user}","gists_url":"https://api.github.com/users/VinzenzBildstein/gists{/gist_id}","starred_url":"https://api.github.com/users/VinzenzBildstein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VinzenzBildstein/subscriptions","organizations_url":"https://api.github.com/users/VinzenzBildstein/orgs","repos_url":"https://api.github.com/users/VinzenzBildstein/repos","events_url":"https://api.github.com/users/VinzenzBildstein/events{/privacy}","received_events_url":"https://api.github.com/users/VinzenzBildstein/received_events","type":"User","site_admin":false},"created_at":"2016-01-22T17:23:36Z","updated_at":"2016-01-22T17:23:36Z","body":"I don't think that custom streamers help with this, as far as I understand it root writes the vectors as arrays anyway. That is the reason that empty vectors take 12 byte in the tree, because the array (which is a pointer) needs 8 bytes (64bit addressing), and the integer that determines the length of the array needs 4 bytes. The only change we could do in a customer streamer is to use a single integer for multiple arrays, which is essentially what I did when using arrays in the TFragment.\r\n\r\nBut using arrays also reduces the memory usage of grsisort which is important if we want to run multiple instances (to increase the sorting speed). I think this is due to the fact that vectors have an overhead of 24 bytes compared to the 12 bytes of the array plus integer. So I think that at least for the TFragment we should think about switching to using arrays. We can still keep vectors in the analysis classes, because I agree with you, they make life much easier in the analysis.\r\n\r\nIf we decide to go this way, the changing of the TFragment would most likely require a re-sorting of all old data as well, unless we write custom streamers that can handle reading the older TFragment data as well as the new. Of course at that point the question whether we want to make all TFragment members private (and use our normal format of starting members with lower case f) becomes relevant again. As far as I remember we decided against that, when the question was originally raised, because it would require re-sorting, but if we need to do that anyway ..."}}