{"action":"created","comment":{"url":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/comments/91861700","pull_request_review_id":12371681,"id":91861700,"diff_hunk":"@@ -331,3 +330,16 @@ def load_h5(filename, path='', lazy=False):\n                   \"will be removed in the next release. Please use load() instead.\",\n                   DeprecationWarning)\n     return load(filename, path=path, lazy=lazy)\n+\n+\n+# Helper function\n+def _accumulate(iterator):\n+    \"\"\"\n+    Create a generator to iterate over the accumulated\n+    values of the given iterator.\n+    \"\"\"\n+    total = 0\n+    yield total\n+    for item in iterator:\n+        total += item","path":"wrapper.py","position":29,"original_position":29,"commit_id":"501d9478198b66094e2e6b88cfa54a6c561f90ad","original_commit_id":"501d9478198b66094e2e6b88cfa54a6c561f90ad","user":{"login":"jakobj","id":1562742,"avatar_url":"https://avatars.githubusercontent.com/u/1562742?v=3","gravatar_id":"","url":"https://api.github.com/users/jakobj","html_url":"https://github.com/jakobj","followers_url":"https://api.github.com/users/jakobj/followers","following_url":"https://api.github.com/users/jakobj/following{/other_user}","gists_url":"https://api.github.com/users/jakobj/gists{/gist_id}","starred_url":"https://api.github.com/users/jakobj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakobj/subscriptions","organizations_url":"https://api.github.com/users/jakobj/orgs","repos_url":"https://api.github.com/users/jakobj/repos","events_url":"https://api.github.com/users/jakobj/events{/privacy}","received_events_url":"https://api.github.com/users/jakobj/received_events","type":"User","site_admin":false},"body":"if you switch `total += item` and `yield total` you don't need the `yield` before the loop.","created_at":"2016-12-11T18:30:44Z","updated_at":"2016-12-11T18:30:44Z","html_url":"https://github.com/INM-6/h5py_wrapper/pull/35#discussion_r91861700","pull_request_url":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/35","_links":{"self":{"href":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/comments/91861700"},"html":{"href":"https://github.com/INM-6/h5py_wrapper/pull/35#discussion_r91861700"},"pull_request":{"href":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/35"}}},"pull_request":{"url":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/35","id":97128767,"html_url":"https://github.com/INM-6/h5py_wrapper/pull/35","diff_url":"https://github.com/INM-6/h5py_wrapper/pull/35.diff","patch_url":"https://github.com/INM-6/h5py_wrapper/pull/35.patch","issue_url":"https://api.github.com/repos/INM-6/h5py_wrapper/issues/35","number":35,"state":"open","locked":false,"title":"Improve performance of _load_custom_shape function","user":{"login":"mschmidt87","id":6512628,"avatar_url":"https://avatars.githubusercontent.com/u/6512628?v=3","gravatar_id":"","url":"https://api.github.com/users/mschmidt87","html_url":"https://github.com/mschmidt87","followers_url":"https://api.github.com/users/mschmidt87/followers","following_url":"https://api.github.com/users/mschmidt87/following{/other_user}","gists_url":"https://api.github.com/users/mschmidt87/gists{/gist_id}","starred_url":"https://api.github.com/users/mschmidt87/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mschmidt87/subscriptions","organizations_url":"https://api.github.com/users/mschmidt87/orgs","repos_url":"https://api.github.com/users/mschmidt87/repos","events_url":"https://api.github.com/users/mschmidt87/events{/privacy}","received_events_url":"https://api.github.com/users/mschmidt87/received_events","type":"User","site_admin":false},"body":"This PR leads to a significant improvement of the `_load_custom_shape` function.\r\nIt addresses #34 .\r\nI measured the performance of the current master vs. the 0.0.1-release for loading a 20 MB data file containing arrays with custom shapes. The result was\r\n\r\n|||\r\n|---------|--------|\r\n| 0.0.1 | 0.06 s |\r\n| master | 2.06 s |\r\n| enhancement/custom_shape_performance | 0.05 s |\r\n\r\nUsing python's line_profiler, I found that retrieving the value of an hdf5 group is quite costly. In the current master, this retrieval is inside the loop in which the loaded array is shaped to the required shape. Simply moving this out of the loop leads to a huge reduction of runtime.\r\n\r\n**master**\r\n```\r\nFunction: _load_custom_shape at line 306\r\n\r\nLine #      Hits         Time  Per Hit   % Time  Line Contents\r\n==============================================================\r\n   306                                           @profile\r\n   307                                           def _load_custom_shape(f):\r\n   308                                               \"\"\"\r\n   309                                               Reshape array with unequal dimensions into original shape.\r\n   310                                               \"\"\"\r\n   311         2            2      1.0      0.0      data_reshaped = []\r\n   312         2            0      0.0      0.0      counter = 0\r\n   313      1002         1610      1.6      0.1      for l in f.attrs['oldshape']:\r\n   314      1000      2095013   2095.0     99.8          data_reshaped.append(np.array(f.value[counter:counter + l]))\r\n   315      1000         1803      1.8      0.1          counter += l\r\n   316         2           41     20.5      0.0      return np.array(data_reshaped, dtype=object)\r\n\r\nTotal time: 2.15489 s\r\n```\r\n\r\n**enhancement/custom_shape_performance**\r\n```\r\nFunction: _load_custom_shape at line 306\r\n\r\nLine #      Hits         Time  Per Hit   % Time  Line Contents\r\n==============================================================\r\n   306                                           @profile\r\n   307                                           def _load_custom_shape(f):\r\n   308                                               \"\"\"\r\n   309                                               Reshape array with unequal dimensions into original shape.\r\n   310                                               \"\"\"\r\n   311         2            3      1.5      0.0      data_reshaped = []\r\n   312         2            2      1.0      0.0      counter = 0\r\n   313         2         4065   2032.5     28.0      value = f.value\r\n   314      1002          598      0.6      4.1      for l in f.attrs['oldshape']:\r\n   315      1000         9298      9.3     64.0          data_reshaped.append(np.array(value[counter:counter + l]))\r\n   316      1000          520      0.5      3.6          counter += l\r\n   317         2           51     25.5      0.4      return np.array(data_reshaped, dtype=object)\r\nTotal time: 0.072391 s\r\n```\r\n\r\n\r\n","created_at":"2016-12-08T14:12:19Z","updated_at":"2016-12-11T18:30:44Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bd2eca45d31cbca279792daaaa07e64135a06f92","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/35/commits","review_comments_url":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/35/comments","review_comment_url":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/comments{/number}","comments_url":"https://api.github.com/repos/INM-6/h5py_wrapper/issues/35/comments","statuses_url":"https://api.github.com/repos/INM-6/h5py_wrapper/statuses/501d9478198b66094e2e6b88cfa54a6c561f90ad","head":{"label":"mschmidt87:enhancement/custom_shape_performance","ref":"enhancement/custom_shape_performance","sha":"501d9478198b66094e2e6b88cfa54a6c561f90ad","user":{"login":"mschmidt87","id":6512628,"avatar_url":"https://avatars.githubusercontent.com/u/6512628?v=3","gravatar_id":"","url":"https://api.github.com/users/mschmidt87","html_url":"https://github.com/mschmidt87","followers_url":"https://api.github.com/users/mschmidt87/followers","following_url":"https://api.github.com/users/mschmidt87/following{/other_user}","gists_url":"https://api.github.com/users/mschmidt87/gists{/gist_id}","starred_url":"https://api.github.com/users/mschmidt87/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mschmidt87/subscriptions","organizations_url":"https://api.github.com/users/mschmidt87/orgs","repos_url":"https://api.github.com/users/mschmidt87/repos","events_url":"https://api.github.com/users/mschmidt87/events{/privacy}","received_events_url":"https://api.github.com/users/mschmidt87/received_events","type":"User","site_admin":false},"repo":{"id":35635276,"name":"h5py_wrapper","full_name":"mschmidt87/h5py_wrapper","owner":{"login":"mschmidt87","id":6512628,"avatar_url":"https://avatars.githubusercontent.com/u/6512628?v=3","gravatar_id":"","url":"https://api.github.com/users/mschmidt87","html_url":"https://github.com/mschmidt87","followers_url":"https://api.github.com/users/mschmidt87/followers","following_url":"https://api.github.com/users/mschmidt87/following{/other_user}","gists_url":"https://api.github.com/users/mschmidt87/gists{/gist_id}","starred_url":"https://api.github.com/users/mschmidt87/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mschmidt87/subscriptions","organizations_url":"https://api.github.com/users/mschmidt87/orgs","repos_url":"https://api.github.com/users/mschmidt87/repos","events_url":"https://api.github.com/users/mschmidt87/events{/privacy}","received_events_url":"https://api.github.com/users/mschmidt87/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mschmidt87/h5py_wrapper","description":"A wrapper to conveniently store nested python dictionaries in hdf5 files.","fork":true,"url":"https://api.github.com/repos/mschmidt87/h5py_wrapper","forks_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/forks","keys_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/teams","hooks_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/hooks","issue_events_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/issues/events{/number}","events_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/events","assignees_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/assignees{/user}","branches_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/branches{/branch}","tags_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/tags","blobs_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/git/refs{/sha}","trees_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/statuses/{sha}","languages_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/languages","stargazers_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/stargazers","contributors_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/contributors","subscribers_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/subscribers","subscription_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/subscription","commits_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/commits{/sha}","git_commits_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/git/commits{/sha}","comments_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/comments{/number}","issue_comment_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/issues/comments{/number}","contents_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/contents/{+path}","compare_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/merges","archive_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/downloads","issues_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/issues{/number}","pulls_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/pulls{/number}","milestones_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/milestones{/number}","notifications_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/labels{/name}","releases_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/releases{/id}","deployments_url":"https://api.github.com/repos/mschmidt87/h5py_wrapper/deployments","created_at":"2015-05-14T20:30:05Z","updated_at":"2016-12-02T18:45:04Z","pushed_at":"2016-12-11T18:17:23Z","git_url":"git://github.com/mschmidt87/h5py_wrapper.git","ssh_url":"git@github.com:mschmidt87/h5py_wrapper.git","clone_url":"https://github.com/mschmidt87/h5py_wrapper.git","svn_url":"https://github.com/mschmidt87/h5py_wrapper","homepage":null,"size":84,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"INM-6:master","ref":"master","sha":"25834fbc60c08f9a116d80447c1afdd24336da7b","user":{"login":"INM-6","id":10058055,"avatar_url":"https://avatars.githubusercontent.com/u/10058055?v=3","gravatar_id":"","url":"https://api.github.com/users/INM-6","html_url":"https://github.com/INM-6","followers_url":"https://api.github.com/users/INM-6/followers","following_url":"https://api.github.com/users/INM-6/following{/other_user}","gists_url":"https://api.github.com/users/INM-6/gists{/gist_id}","starred_url":"https://api.github.com/users/INM-6/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/INM-6/subscriptions","organizations_url":"https://api.github.com/users/INM-6/orgs","repos_url":"https://api.github.com/users/INM-6/repos","events_url":"https://api.github.com/users/INM-6/events{/privacy}","received_events_url":"https://api.github.com/users/INM-6/received_events","type":"Organization","site_admin":false},"repo":{"id":32864601,"name":"h5py_wrapper","full_name":"INM-6/h5py_wrapper","owner":{"login":"INM-6","id":10058055,"avatar_url":"https://avatars.githubusercontent.com/u/10058055?v=3","gravatar_id":"","url":"https://api.github.com/users/INM-6","html_url":"https://github.com/INM-6","followers_url":"https://api.github.com/users/INM-6/followers","following_url":"https://api.github.com/users/INM-6/following{/other_user}","gists_url":"https://api.github.com/users/INM-6/gists{/gist_id}","starred_url":"https://api.github.com/users/INM-6/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/INM-6/subscriptions","organizations_url":"https://api.github.com/users/INM-6/orgs","repos_url":"https://api.github.com/users/INM-6/repos","events_url":"https://api.github.com/users/INM-6/events{/privacy}","received_events_url":"https://api.github.com/users/INM-6/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/INM-6/h5py_wrapper","description":"A wrapper to conveniently store nested python dictionaries in hdf5 files.","fork":false,"url":"https://api.github.com/repos/INM-6/h5py_wrapper","forks_url":"https://api.github.com/repos/INM-6/h5py_wrapper/forks","keys_url":"https://api.github.com/repos/INM-6/h5py_wrapper/keys{/key_id}","collaborators_url":"https://api.github.com/repos/INM-6/h5py_wrapper/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/INM-6/h5py_wrapper/teams","hooks_url":"https://api.github.com/repos/INM-6/h5py_wrapper/hooks","issue_events_url":"https://api.github.com/repos/INM-6/h5py_wrapper/issues/events{/number}","events_url":"https://api.github.com/repos/INM-6/h5py_wrapper/events","assignees_url":"https://api.github.com/repos/INM-6/h5py_wrapper/assignees{/user}","branches_url":"https://api.github.com/repos/INM-6/h5py_wrapper/branches{/branch}","tags_url":"https://api.github.com/repos/INM-6/h5py_wrapper/tags","blobs_url":"https://api.github.com/repos/INM-6/h5py_wrapper/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/INM-6/h5py_wrapper/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/INM-6/h5py_wrapper/git/refs{/sha}","trees_url":"https://api.github.com/repos/INM-6/h5py_wrapper/git/trees{/sha}","statuses_url":"https://api.github.com/repos/INM-6/h5py_wrapper/statuses/{sha}","languages_url":"https://api.github.com/repos/INM-6/h5py_wrapper/languages","stargazers_url":"https://api.github.com/repos/INM-6/h5py_wrapper/stargazers","contributors_url":"https://api.github.com/repos/INM-6/h5py_wrapper/contributors","subscribers_url":"https://api.github.com/repos/INM-6/h5py_wrapper/subscribers","subscription_url":"https://api.github.com/repos/INM-6/h5py_wrapper/subscription","commits_url":"https://api.github.com/repos/INM-6/h5py_wrapper/commits{/sha}","git_commits_url":"https://api.github.com/repos/INM-6/h5py_wrapper/git/commits{/sha}","comments_url":"https://api.github.com/repos/INM-6/h5py_wrapper/comments{/number}","issue_comment_url":"https://api.github.com/repos/INM-6/h5py_wrapper/issues/comments{/number}","contents_url":"https://api.github.com/repos/INM-6/h5py_wrapper/contents/{+path}","compare_url":"https://api.github.com/repos/INM-6/h5py_wrapper/compare/{base}...{head}","merges_url":"https://api.github.com/repos/INM-6/h5py_wrapper/merges","archive_url":"https://api.github.com/repos/INM-6/h5py_wrapper/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/INM-6/h5py_wrapper/downloads","issues_url":"https://api.github.com/repos/INM-6/h5py_wrapper/issues{/number}","pulls_url":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls{/number}","milestones_url":"https://api.github.com/repos/INM-6/h5py_wrapper/milestones{/number}","notifications_url":"https://api.github.com/repos/INM-6/h5py_wrapper/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/INM-6/h5py_wrapper/labels{/name}","releases_url":"https://api.github.com/repos/INM-6/h5py_wrapper/releases{/id}","deployments_url":"https://api.github.com/repos/INM-6/h5py_wrapper/deployments","created_at":"2015-03-25T12:58:14Z","updated_at":"2016-02-27T21:58:37Z","pushed_at":"2016-12-11T18:17:23Z","git_url":"git://github.com/INM-6/h5py_wrapper.git","ssh_url":"git@github.com:INM-6/h5py_wrapper.git","clone_url":"https://github.com/INM-6/h5py_wrapper.git","svn_url":"https://github.com/INM-6/h5py_wrapper","homepage":null,"size":63,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":10,"mirror_url":null,"open_issues_count":8,"forks":10,"open_issues":8,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/35"},"html":{"href":"https://github.com/INM-6/h5py_wrapper/pull/35"},"issue":{"href":"https://api.github.com/repos/INM-6/h5py_wrapper/issues/35"},"comments":{"href":"https://api.github.com/repos/INM-6/h5py_wrapper/issues/35/comments"},"review_comments":{"href":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/35/comments"},"review_comment":{"href":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/INM-6/h5py_wrapper/pulls/35/commits"},"statuses":{"href":"https://api.github.com/repos/INM-6/h5py_wrapper/statuses/501d9478198b66094e2e6b88cfa54a6c561f90ad"}}}}
{"actor":{"display_login":"jakobj"}}