{"action":"created","issue":{"url":"https://api.github.com/repos/XENON1T/hax/issues/32","repository_url":"https://api.github.com/repos/XENON1T/hax","labels_url":"https://api.github.com/repos/XENON1T/hax/issues/32/labels{/name}","comments_url":"https://api.github.com/repos/XENON1T/hax/issues/32/comments","events_url":"https://api.github.com/repos/XENON1T/hax/issues/32/events","html_url":"https://github.com/XENON1T/hax/pull/32","id":173101946,"number":32,"title":"Minitree save arrays and pickle","user":{"login":"jhowl01","id":16269427,"avatar_url":"https://avatars.githubusercontent.com/u/16269427?v=3","gravatar_id":"","url":"https://api.github.com/users/jhowl01","html_url":"https://github.com/jhowl01","followers_url":"https://api.github.com/users/jhowl01/followers","following_url":"https://api.github.com/users/jhowl01/following{/other_user}","gists_url":"https://api.github.com/users/jhowl01/gists{/gist_id}","starred_url":"https://api.github.com/users/jhowl01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhowl01/subscriptions","organizations_url":"https://api.github.com/users/jhowl01/orgs","repos_url":"https://api.github.com/users/jhowl01/repos","events_url":"https://api.github.com/users/jhowl01/events{/privacy}","received_events_url":"https://api.github.com/users/jhowl01/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-08-25T02:42:23Z","updated_at":"2016-08-25T11:09:07Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/XENON1T/hax/pulls/32","html_url":"https://github.com/XENON1T/hax/pull/32","diff_url":"https://github.com/XENON1T/hax/pull/32.diff","patch_url":"https://github.com/XENON1T/hax/pull/32.patch"},"body":"# Changes in this branch\r\nThere are a couple changes to minitrees.py in this branch, and one new jupyter notebook containing a usage example (\"minitree_with_arrays\"). Here I will explain the motivation for these changes.\r\n\r\n## Pickling vs Root\r\nThe main hax output is currently a pandas DataFrame which is returned by `minitrees.load()`, as well as a `.root` file produced from this DataFrame and saved in the `minitree_paths` directory. While one can use the returned DataFrame for pandas analysis, some prefer to do further analysis using the reduced saved dataset.\r\n\r\nSince pickling allows one to serialize a pandas DataFrame without bringing root into the mix at all, we thought it may be useful to be able to save in this format as well or instead, especially since some people prefer to do analysis without root.\r\n\r\n**Pickle -** Analysis with pandas/numpy/matplotlib\r\n\r\n**Root -** Analysis with root, backward compatibility\r\n\r\n## Saving arrays in a minitree\r\nCurrently, minitrees are flat, in that each branch takes a single value per event. If you make a TreeMaker with an `extract_data()` function that returns a dictionary containing arrays, the array-type fields will not show up as branches in the output `.root` file, nor will they appear in the returned DataFrame. This is fine for many applications, but becomes difficult if you want to look at more than a few interactions per event (e.g. peak- and hit-level analyses).\r\n\r\n### DataExtractor vs minitrees with arrays\r\nHax now includes the `DataExtractor()` class for analyses of this type. Its output is a numpy structured array which is also flat in that none of its columns contain arrays, but it does allow one to save information about _all_ of the peaks or hits in a dataset.\r\n\r\n**DataExtractor Output**\r\n<img width=\"307\" alt=\"dataextractor_table\" src=\"https://cloud.githubusercontent.com/assets/16269427/17954701/a444084a-6a4a-11e6-9cff-f25a88b9bedc.png\">\r\n\r\n**Minitrees w/ Arrays Output**\r\n<img width=\"576\" alt=\"branch_table\" src=\"https://cloud.githubusercontent.com/assets/16269427/17954704/a96de49e-6a4a-11e6-82bc-c1dce100439a.png\">\r\n\r\nThere are a few reasons some might prefer to use a minitree with arrays rather than DataExtractor to look at peak-level info. First, the array branches are space-saving, since in the DataExtractor output, the values of each event-level field are repeated for each peak in an event. Also, the DataExtractor method does not save the results, and if they were saved to a `.root` file it would not have the usual structure where each entry is an event and peak information is represented as array-type branches. For these reasons, the minitree method may be ideal for those of us who use hax to produce a reduced copy of the processed `.root` file on midway, then do our analysis on the reduced file wherever we want.\r\n\r\nIt may also be advantageous that this branch contains no change on the user side, except that array-type fields no longer disappear - so if an analysis using minitrees expands to the point where the user prefers arrays to many dictionary keys, this can be accommodated without any change in code. Note that the new function `dataframe_to_root()` is only used if the dictionaries returned by `extract_data()` contain arrays, and otherwise there is no change except those made to allow pickling.\r\n\r\n## Structure of changes\r\n![minitree_edits 001](https://cloud.githubusercontent.com/assets/16269427/17954699/9a4eb182-6a4a-11e6-949f-a87c38617571.png)\r\nBasically, rather than automatically passing the DataFrame to a `.root` using `root_numpy.array2root()`, and then retrieving it using `root_numpy.root2array()` before returning it, in my branch the DataFrame is returned directly. First though, it checks for the options `save_root` (defaults true) and `save_pickle` (defaults false). If `save_root` is true, it will check if the dictionaries contain arrays. If they do, it will use my new function `dataframe_to_root()`, and if not it will use `array2root()` as normal. If `save_pickle` is true, a dictionary containing both the DataFrame and the same metadata dictionary will be saved to a `.pkl` with the same header.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/XENON1T/hax/issues/comments/242351321","html_url":"https://github.com/XENON1T/hax/pull/32#issuecomment-242351321","issue_url":"https://api.github.com/repos/XENON1T/hax/issues/32","id":242351321,"user":{"login":"JelleAalbers","id":4354311,"avatar_url":"https://avatars.githubusercontent.com/u/4354311?v=3","gravatar_id":"","url":"https://api.github.com/users/JelleAalbers","html_url":"https://github.com/JelleAalbers","followers_url":"https://api.github.com/users/JelleAalbers/followers","following_url":"https://api.github.com/users/JelleAalbers/following{/other_user}","gists_url":"https://api.github.com/users/JelleAalbers/gists{/gist_id}","starred_url":"https://api.github.com/users/JelleAalbers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JelleAalbers/subscriptions","organizations_url":"https://api.github.com/users/JelleAalbers/orgs","repos_url":"https://api.github.com/users/JelleAalbers/repos","events_url":"https://api.github.com/users/JelleAalbers/events{/privacy}","received_events_url":"https://api.github.com/users/JelleAalbers/received_events","type":"User","site_admin":false},"created_at":"2016-08-25T11:09:07Z","updated_at":"2016-08-25T11:09:07Z","body":"Hi Joey,\r\n\r\nIt's great to see someone new has read the code, actually understood it, identified a problem, then contributed something useful. You also explain it in a clear way so it's easy to review. The code looks quite readable, and the example notebook also helps understand what's going on. So thanks a lot!\r\n\r\nThe array fields feature address a big problem several analysts are having. DataExtractor exists, but it's interface is less nice, it lacks some features (most importantly file-based caching) and has less worked examples, so people don't often use it yet. Instead, I've seen people packing their stuff in multiple columns (interaction_0, interaction_1, ...) of a minitree... :-(\r\n\r\nStill, I think DataExtractor is what people usually want, even though they don't know it yet :-). Take a single electron analysis. With your feature I can collect properties of some peaks (say area, width, x, y) in array fields in a minitree. But after I made the minitree, how would I select, say, only peaks with an area larger than 10?  Remaking the minitree for every selection is very slow. I think I would have to iterate over the rows, get a boolean mask from the area field, then apply it to all of the peak property fields in the array. \r\n\r\nIf this makes people's life easier we should definitely merge it, but I'm worried it's actually quite difficult to work with once you go beyond just creating a minitree and looking at it. Maybe you can give an example of how you would use this in your analysis?\r\n\r\nAbout save_pickle: any move away from formats used only in the particle physics bubbleverse of course has to be welcomed :-) In pax we use pickle as the raw data format, but we compress the pickles with zlib first. I'm not sure if compression is needed for minitrees, but maybe if we start storing arrays in them they will become large. Might be worth a look.\r\n\r\nAm I correct though, that hax itself still only reads from the .root cached minitrees? So you can save to pickle, but then have to deal with reading it in, concatenating it with other minitrees etc. yourself. Then I'm not sure why you would like to have this in hax.minitrees. Doesn't really hurt anyone if it's just an option, but...\r\n\r\nThanks for making this pull request, again I really appreciate that you just dove into this and came out with something useful. "}}
{"actor":{"display_login":"JelleAalbers"}}