{"action":"created","issue":{"url":"https://api.github.com/repos/BlueBrain/NeuroM/issues/301","repository_url":"https://api.github.com/repos/BlueBrain/NeuroM","labels_url":"https://api.github.com/repos/BlueBrain/NeuroM/issues/301/labels{/name}","comments_url":"https://api.github.com/repos/BlueBrain/NeuroM/issues/301/comments","events_url":"https://api.github.com/repos/BlueBrain/NeuroM/issues/301/events","html_url":"https://github.com/BlueBrain/NeuroM/issues/301","id":148620501,"number":301,"title":"Thoughts on design","user":{"login":"eleftherioszisis","id":15011025,"avatar_url":"https://avatars.githubusercontent.com/u/15011025?v=3","gravatar_id":"","url":"https://api.github.com/users/eleftherioszisis","html_url":"https://github.com/eleftherioszisis","followers_url":"https://api.github.com/users/eleftherioszisis/followers","following_url":"https://api.github.com/users/eleftherioszisis/following{/other_user}","gists_url":"https://api.github.com/users/eleftherioszisis/gists{/gist_id}","starred_url":"https://api.github.com/users/eleftherioszisis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eleftherioszisis/subscriptions","organizations_url":"https://api.github.com/users/eleftherioszisis/orgs","repos_url":"https://api.github.com/users/eleftherioszisis/repos","events_url":"https://api.github.com/users/eleftherioszisis/events{/privacy}","received_events_url":"https://api.github.com/users/eleftherioszisis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":33,"created_at":"2016-04-15T10:15:08Z","updated_at":"2016-04-21T10:52:57Z","closed_at":null,"body":"I would like to discuss some things that I have noticed in NeuroM. The suggestion that I present requires a lot of changes in order to be applied, thus I would like make clear that this post is more on thinking into the future than making any changes in the current structure of NeuroM. Let's begin:\r\n\r\nWe have adopted a tree representation of our data which is used throughout the modules via iterators, maps etc. \r\n\r\nI understand that in languages where the trees are expressed with pointers, creating iterators and recursively generating stuff is super fast. In python however, generators are not fast, and functional programming will always lag behind the not so cheap function calls (although as far as we do not put them in loops, it's not that bad).\r\n\r\nIterators in Python are memory efficient because they do not store everything in memory at once when iterative calculations are yielded, but in our case that we first create the tree in memory and then return the parent node, we are in a lose lose situation. Plus, we always need to apply an O(n) traversal for operations where this could be avoided by a lookup table structure.\r\n\r\nOur loaders and processing are slow when the order of magnitude of number of cells is hundred and up, needing a lot of minutes to load or process a population. When I tried to profile and optimize the loading I stumbled upon the scary truth: Our bottleneck is mainly the make_tree function which iteratively generates a tree from the data. Unfortunately, the recursive alternative is even worse due to the unoptimized recursion in Python.\r\n\r\nI also noticed that in a lot of functions we convert the tree to a list of elements (```as_elements```), and I am wondering what would be the ratio of the times that we need a tree as a tree over the times that we need it as a array of points. Maybe we would need to write less code if we had the points to begin with.\r\n\r\nBeing stuck on the python level, we will always find these overheads in front of us, which kills the performance of NeuroM and causes serious problems when populations need to be analyzed.\r\n\r\nYou will ask: what alternatives do we have? Well, we can always go full numpy and exploit the speed of light (c). Instead of creating trees we can load the data through numpy, and generate a connectivity matrix for the \"parenting\" (I think that Pneumatk uses this design, not sure though). I suspect that this will eliminate the majority of the overheads.\r\n\r\nWhat do you think?"},"comment":{"url":"https://api.github.com/repos/BlueBrain/NeuroM/issues/comments/212854711","html_url":"https://github.com/BlueBrain/NeuroM/issues/301#issuecomment-212854711","issue_url":"https://api.github.com/repos/BlueBrain/NeuroM/issues/301","id":212854711,"user":{"login":"juanchopanza","id":675013,"avatar_url":"https://avatars.githubusercontent.com/u/675013?v=3","gravatar_id":"","url":"https://api.github.com/users/juanchopanza","html_url":"https://github.com/juanchopanza","followers_url":"https://api.github.com/users/juanchopanza/followers","following_url":"https://api.github.com/users/juanchopanza/following{/other_user}","gists_url":"https://api.github.com/users/juanchopanza/gists{/gist_id}","starred_url":"https://api.github.com/users/juanchopanza/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juanchopanza/subscriptions","organizations_url":"https://api.github.com/users/juanchopanza/orgs","repos_url":"https://api.github.com/users/juanchopanza/repos","events_url":"https://api.github.com/users/juanchopanza/events{/privacy}","received_events_url":"https://api.github.com/users/juanchopanza/received_events","type":"User","site_admin":false},"created_at":"2016-04-21T10:52:57Z","updated_at":"2016-04-21T10:52:57Z","body":"FYI, with the latest changes, including #311, a quick profiling of `neurom.io.load_data('big_neuron.h5')` shows an improvement from ~5s to ~0.7s. The data unpacking part no longer dominates the neuron building, but I want to optimize it further because we're likely to keep it. 0.7s for a single neuron, no matter how large, still seems slow to me."}}