{"action":"created","comment":{"url":"https://api.github.com/repos/nipy/nibabel/pulls/comments/61931608","id":61931608,"diff_hunk":"@@ -62,6 +65,22 @@ def assert_allclose_safely(a, b, match_nans=True, rtol=1e-5, atol=1e-8):\n     assert_true(np.allclose(a, b, rtol=rtol, atol=atol))\n \n \n+def check_iteration(iterable):\n+    \"\"\" Checks that an object can be iterated through without errors. \"\"\"\n+    try:\n+        for _ in iterable:\n+            pass\n+    except:\n+        return False\n+\n+    return True\n+\n+\n+def assert_arrays_equal(arrays1, arrays2):\n+    for arr1, arr2 in zip_longest(arrays1, arrays2, fillvalue=None):","path":"nibabel/testing/__init__.py","position":33,"original_position":33,"commit_id":"713ef7b64a3f1fb1b84dcdb03f070bdb6272940b","original_commit_id":"713ef7b64a3f1fb1b84dcdb03f070bdb6272940b","user":{"login":"matthew-brett","id":67612,"avatar_url":"https://avatars.githubusercontent.com/u/67612?v=3","gravatar_id":"","url":"https://api.github.com/users/matthew-brett","html_url":"https://github.com/matthew-brett","followers_url":"https://api.github.com/users/matthew-brett/followers","following_url":"https://api.github.com/users/matthew-brett/following{/other_user}","gists_url":"https://api.github.com/users/matthew-brett/gists{/gist_id}","starred_url":"https://api.github.com/users/matthew-brett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthew-brett/subscriptions","organizations_url":"https://api.github.com/users/matthew-brett/orgs","repos_url":"https://api.github.com/users/matthew-brett/repos","events_url":"https://api.github.com/users/matthew-brett/events{/privacy}","received_events_url":"https://api.github.com/users/matthew-brett/received_events","type":"User","site_admin":false},"body":"I guess this one is for comparing sequences of arrays?  I guess too that the `fillvalue=None` is intended to make this fail when the sequences are different length?   If so, it would be easier to understand written as:\r\n\r\n```\r\nassert_equal(len(arrays1), len(arrays2))\r\nfor arr1, arr2 in zip(arrays1, arrays2):\r\n    assert_array_equal(arr1, arr2)\r\n```\r\n\r\nThat will also avoid hitting the numpy deprecation warning comparing arrays with None:\r\n\r\n```\r\nIn [6]: None == np.arange(10)\r\n/Users/mb312/Library/Python/2.7/bin/ipython:1: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.\r\n  #!/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python\r\nOut[6]: False\r\n```\r\n","created_at":"2016-05-03T18:31:07Z","updated_at":"2016-05-03T18:31:07Z","html_url":"https://github.com/nipy/nibabel/pull/391#discussion_r61931608","pull_request_url":"https://api.github.com/repos/nipy/nibabel/pulls/391","_links":{"self":{"href":"https://api.github.com/repos/nipy/nibabel/pulls/comments/61931608"},"html":{"href":"https://github.com/nipy/nibabel/pull/391#discussion_r61931608"},"pull_request":{"href":"https://api.github.com/repos/nipy/nibabel/pulls/391"}}},"pull_request":{"url":"https://api.github.com/repos/nipy/nibabel/pulls/391","id":52565146,"html_url":"https://github.com/nipy/nibabel/pull/391","diff_url":"https://github.com/nipy/nibabel/pull/391.diff","patch_url":"https://github.com/nipy/nibabel/pull/391.patch","issue_url":"https://api.github.com/repos/nipy/nibabel/issues/391","number":391,"state":"open","locked":false,"title":"Streamlines API","user":{"login":"MarcCote","id":660004,"avatar_url":"https://avatars.githubusercontent.com/u/660004?v=3","gravatar_id":"","url":"https://api.github.com/users/MarcCote","html_url":"https://github.com/MarcCote","followers_url":"https://api.github.com/users/MarcCote/followers","following_url":"https://api.github.com/users/MarcCote/following{/other_user}","gists_url":"https://api.github.com/users/MarcCote/gists{/gist_id}","starred_url":"https://api.github.com/users/MarcCote/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MarcCote/subscriptions","organizations_url":"https://api.github.com/users/MarcCote/orgs","repos_url":"https://api.github.com/users/MarcCote/repos","events_url":"https://api.github.com/users/MarcCote/events{/privacy}","received_events_url":"https://api.github.com/users/MarcCote/received_events","type":"User","site_admin":false},"body":"This is take two at providing a general API to manage different streamlines file format (the code can all be found in `nibabel.streamlines.`). This is related to this somewhat outdated [BIAP5](https://github.com/nipy/nibabel/wiki/BIAP5). I initially wanted to make two PRs but as @Garyfallidis [pointed out](https://github.com/nipy/nibabel/pull/243#issuecomment-160810062), without the entire PR \"it will make it difficult to try this with real data\".\r\n\r\n\r\nThis new streamline API is divided in two main parts:\r\n\r\n1. Representation of streamlines in memory (`Tractogram` and `ArraySequence` classes)\r\n2. Loading and saving streamlines from/to different file format (`TractogramFile` class)\r\n\r\n*We use the term 'Tractogram' to refer to a set of streamlines, usually the output of a tractography algorithm.*\r\n\r\n![streamlines_api](https://cloud.githubusercontent.com/assets/660004/12959160/4606db04-d003-11e5-9161-778f488c8bfd.png)\r\n\r\n#### ArraySequence\r\nThe `ArraySequence` class needs to be explained first as it is at the heart of the streamlines representation. I guess it could be seen as a sort of jagged/ragged array to store a list of 2D arrays that have different length but a same number of dimensions (e.g. a list of list of points). For efficiency, we store a list of streamlines contiguously in memory using what we call a `ArraySequence`. This was needed as @Garyfallidis and I realized that a list of numpy arrays has too much overhead in term of memory when handling >1M streamlines (loading a tractogram of 2.5Gb could take up to 5Gb in RAM!).\r\n\r\nA `ArraySequence` behaves exactly like a `list` of `2D ndarray`. That is, it supports appending, extending, indexing (supports slicing too), iterating, etc. as a `list` object. For example, indexing a `ArraySequence` object will return a 2D `ndarray`, as one would expect, representing the 3D points of the selected streamline. Note that, while `ArraySequence` does not support deletion (i.e. no `del` nor `remove`), one can efficiently use slicing to achieve the same goal.\r\n\r\nA `ArraySequence` object can be instantiated using any iterable yielding `2D array`:\r\n```\r\nimport numpy as np\r\nimport nibabel as nib\r\ndata = [np.random.rand(10, 3), np.random.rand(3, 3)]\r\nseq = nib.streamlines.ArraySequence(data)\r\n```\r\n\r\nSlicing a `ArraySequence` object returns a new `ArraySequence` object acting as a view (*no memory duplication*). Of course, one can use the `copy` method to \"force\" the memory duplication.\r\n\r\n\r\n#### Tractogram\r\nThis part handles streamlines in memory using the Python class `Tractogram`. This class provides a mechanism to keep data associated to points of every streamline (`data_per_point` property). It can also keep data associated to each streamline (`data_per_streamline` property). \r\n\r\nRegarding the reference space we used for the streamlines of a tractogram live in, we decided to follow the NiBabel convention: the streamlines are assumed to be in **RAS+**, points are expressed in **millimetre** and the coordinate of a voxel (eg. (0,0,0) or (1,2,3)) refers to the **centre of that voxel**.\r\n\r\nA `Tractogram` objects supports indexing, slicing and iterating. For example, indexing a `Tractogram` object will return a `TractogramItem` object, representing a single streamline with its associated data. Note that, while `Tractogram` does not support deletion (i.e. no `del` nor `remove`), one can efficiently use slicing to achieve the same goal. Again, slicing a `Tractogram` object returns a new `Tractogram` object acting as a view (*no memory duplication*).\r\n\r\nHere a small example of instantiating a `Tractogram` object:\r\n```\r\nimport numpy as np\r\nimport nibabel as nib\r\ndata = [np.random.rand(10, 3), np.random.rand(3, 3)]\r\ntractogram = nib.streamlines.Tractogram(streamlines=data)\r\ncolors = [(1, 0, 0), (0, 1, 0)]\r\ntractogram.data_per_streamlines['colors'] = colors\r\n```\r\n\r\n\r\n#### Tractogram File\r\nThis part handles the saving and loading of `Tractogram` objects from/to the disk. Right now, as a proof of concept, only the TRK file format is supported (support for TCK is already in progress and will be added in a following PR).\r\n\r\nThe `TractogramFile` class is abstract. Each file format should have its own class inheriting `TractogramFile` (e.g., `TrkFile`).\r\n\r\n\r\n\r\n#### Realistic example\r\n\r\nHere a small example of loading a TRK file, computing the length of each streamlines, adding that information to the tractogram and saving it back.\r\n```\r\nimport nibabel as nib\r\ntrk = nib.streamlines.load('my_streamlines.trk')\r\n\r\nfrom dipy.tracking.streamline import length\r\nlengths = length(trk.tractogram.streamlines)\r\n\r\ntrk.tractogram.data_per_streamline['lengths'] = lengths\r\n\r\nnew_trk = nib.streamlines.TrkFile(trk.tractogram, trk.header)\r\nnib.streamlines.save(new_trk, \"my_streamlines.trk\")\r\n```\r\nEDIT: Renamed `CompactList` to `ArraySequence`","created_at":"2015-12-03T17:25:04Z","updated_at":"2016-05-03T18:31:07Z","closed_at":null,"merged_at":null,"merge_commit_sha":"4f860d7c11c89661cd9fa60afae91c3f839b1f76","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/nipy/nibabel/pulls/391/commits","review_comments_url":"https://api.github.com/repos/nipy/nibabel/pulls/391/comments","review_comment_url":"https://api.github.com/repos/nipy/nibabel/pulls/comments{/number}","comments_url":"https://api.github.com/repos/nipy/nibabel/issues/391/comments","statuses_url":"https://api.github.com/repos/nipy/nibabel/statuses/713ef7b64a3f1fb1b84dcdb03f070bdb6272940b","head":{"label":"MarcCote:streamlines_api","ref":"streamlines_api","sha":"713ef7b64a3f1fb1b84dcdb03f070bdb6272940b","user":{"login":"MarcCote","id":660004,"avatar_url":"https://avatars.githubusercontent.com/u/660004?v=3","gravatar_id":"","url":"https://api.github.com/users/MarcCote","html_url":"https://github.com/MarcCote","followers_url":"https://api.github.com/users/MarcCote/followers","following_url":"https://api.github.com/users/MarcCote/following{/other_user}","gists_url":"https://api.github.com/users/MarcCote/gists{/gist_id}","starred_url":"https://api.github.com/users/MarcCote/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MarcCote/subscriptions","organizations_url":"https://api.github.com/users/MarcCote/orgs","repos_url":"https://api.github.com/users/MarcCote/repos","events_url":"https://api.github.com/users/MarcCote/events{/privacy}","received_events_url":"https://api.github.com/users/MarcCote/received_events","type":"User","site_admin":false},"repo":{"id":12565378,"name":"nibabel","full_name":"MarcCote/nibabel","owner":{"login":"MarcCote","id":660004,"avatar_url":"https://avatars.githubusercontent.com/u/660004?v=3","gravatar_id":"","url":"https://api.github.com/users/MarcCote","html_url":"https://github.com/MarcCote","followers_url":"https://api.github.com/users/MarcCote/followers","following_url":"https://api.github.com/users/MarcCote/following{/other_user}","gists_url":"https://api.github.com/users/MarcCote/gists{/gist_id}","starred_url":"https://api.github.com/users/MarcCote/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MarcCote/subscriptions","organizations_url":"https://api.github.com/users/MarcCote/orgs","repos_url":"https://api.github.com/users/MarcCote/repos","events_url":"https://api.github.com/users/MarcCote/events{/privacy}","received_events_url":"https://api.github.com/users/MarcCote/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/MarcCote/nibabel","description":"Python package to access a cacophony of neuro-imaging file formats","fork":true,"url":"https://api.github.com/repos/MarcCote/nibabel","forks_url":"https://api.github.com/repos/MarcCote/nibabel/forks","keys_url":"https://api.github.com/repos/MarcCote/nibabel/keys{/key_id}","collaborators_url":"https://api.github.com/repos/MarcCote/nibabel/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/MarcCote/nibabel/teams","hooks_url":"https://api.github.com/repos/MarcCote/nibabel/hooks","issue_events_url":"https://api.github.com/repos/MarcCote/nibabel/issues/events{/number}","events_url":"https://api.github.com/repos/MarcCote/nibabel/events","assignees_url":"https://api.github.com/repos/MarcCote/nibabel/assignees{/user}","branches_url":"https://api.github.com/repos/MarcCote/nibabel/branches{/branch}","tags_url":"https://api.github.com/repos/MarcCote/nibabel/tags","blobs_url":"https://api.github.com/repos/MarcCote/nibabel/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/MarcCote/nibabel/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/MarcCote/nibabel/git/refs{/sha}","trees_url":"https://api.github.com/repos/MarcCote/nibabel/git/trees{/sha}","statuses_url":"https://api.github.com/repos/MarcCote/nibabel/statuses/{sha}","languages_url":"https://api.github.com/repos/MarcCote/nibabel/languages","stargazers_url":"https://api.github.com/repos/MarcCote/nibabel/stargazers","contributors_url":"https://api.github.com/repos/MarcCote/nibabel/contributors","subscribers_url":"https://api.github.com/repos/MarcCote/nibabel/subscribers","subscription_url":"https://api.github.com/repos/MarcCote/nibabel/subscription","commits_url":"https://api.github.com/repos/MarcCote/nibabel/commits{/sha}","git_commits_url":"https://api.github.com/repos/MarcCote/nibabel/git/commits{/sha}","comments_url":"https://api.github.com/repos/MarcCote/nibabel/comments{/number}","issue_comment_url":"https://api.github.com/repos/MarcCote/nibabel/issues/comments{/number}","contents_url":"https://api.github.com/repos/MarcCote/nibabel/contents/{+path}","compare_url":"https://api.github.com/repos/MarcCote/nibabel/compare/{base}...{head}","merges_url":"https://api.github.com/repos/MarcCote/nibabel/merges","archive_url":"https://api.github.com/repos/MarcCote/nibabel/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/MarcCote/nibabel/downloads","issues_url":"https://api.github.com/repos/MarcCote/nibabel/issues{/number}","pulls_url":"https://api.github.com/repos/MarcCote/nibabel/pulls{/number}","milestones_url":"https://api.github.com/repos/MarcCote/nibabel/milestones{/number}","notifications_url":"https://api.github.com/repos/MarcCote/nibabel/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/MarcCote/nibabel/labels{/name}","releases_url":"https://api.github.com/repos/MarcCote/nibabel/releases{/id}","deployments_url":"https://api.github.com/repos/MarcCote/nibabel/deployments","created_at":"2013-09-03T14:05:03Z","updated_at":"2015-02-19T12:09:52Z","pushed_at":"2016-04-12T04:41:24Z","git_url":"git://github.com/MarcCote/nibabel.git","ssh_url":"git@github.com:MarcCote/nibabel.git","clone_url":"https://github.com/MarcCote/nibabel.git","svn_url":"https://github.com/MarcCote/nibabel","homepage":"http://nipy.org/nibabel/","size":7900,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"nipy:master","ref":"master","sha":"83d32494e310ec3644a4a56967f8c8c0cd84bdfb","user":{"login":"nipy","id":233707,"avatar_url":"https://avatars.githubusercontent.com/u/233707?v=3","gravatar_id":"","url":"https://api.github.com/users/nipy","html_url":"https://github.com/nipy","followers_url":"https://api.github.com/users/nipy/followers","following_url":"https://api.github.com/users/nipy/following{/other_user}","gists_url":"https://api.github.com/users/nipy/gists{/gist_id}","starred_url":"https://api.github.com/users/nipy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nipy/subscriptions","organizations_url":"https://api.github.com/users/nipy/orgs","repos_url":"https://api.github.com/users/nipy/repos","events_url":"https://api.github.com/users/nipy/events{/privacy}","received_events_url":"https://api.github.com/users/nipy/received_events","type":"Organization","site_admin":false},"repo":{"id":791352,"name":"nibabel","full_name":"nipy/nibabel","owner":{"login":"nipy","id":233707,"avatar_url":"https://avatars.githubusercontent.com/u/233707?v=3","gravatar_id":"","url":"https://api.github.com/users/nipy","html_url":"https://github.com/nipy","followers_url":"https://api.github.com/users/nipy/followers","following_url":"https://api.github.com/users/nipy/following{/other_user}","gists_url":"https://api.github.com/users/nipy/gists{/gist_id}","starred_url":"https://api.github.com/users/nipy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nipy/subscriptions","organizations_url":"https://api.github.com/users/nipy/orgs","repos_url":"https://api.github.com/users/nipy/repos","events_url":"https://api.github.com/users/nipy/events{/privacy}","received_events_url":"https://api.github.com/users/nipy/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/nipy/nibabel","description":"Python package to access a cacophony of neuro-imaging file formats","fork":false,"url":"https://api.github.com/repos/nipy/nibabel","forks_url":"https://api.github.com/repos/nipy/nibabel/forks","keys_url":"https://api.github.com/repos/nipy/nibabel/keys{/key_id}","collaborators_url":"https://api.github.com/repos/nipy/nibabel/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/nipy/nibabel/teams","hooks_url":"https://api.github.com/repos/nipy/nibabel/hooks","issue_events_url":"https://api.github.com/repos/nipy/nibabel/issues/events{/number}","events_url":"https://api.github.com/repos/nipy/nibabel/events","assignees_url":"https://api.github.com/repos/nipy/nibabel/assignees{/user}","branches_url":"https://api.github.com/repos/nipy/nibabel/branches{/branch}","tags_url":"https://api.github.com/repos/nipy/nibabel/tags","blobs_url":"https://api.github.com/repos/nipy/nibabel/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/nipy/nibabel/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/nipy/nibabel/git/refs{/sha}","trees_url":"https://api.github.com/repos/nipy/nibabel/git/trees{/sha}","statuses_url":"https://api.github.com/repos/nipy/nibabel/statuses/{sha}","languages_url":"https://api.github.com/repos/nipy/nibabel/languages","stargazers_url":"https://api.github.com/repos/nipy/nibabel/stargazers","contributors_url":"https://api.github.com/repos/nipy/nibabel/contributors","subscribers_url":"https://api.github.com/repos/nipy/nibabel/subscribers","subscription_url":"https://api.github.com/repos/nipy/nibabel/subscription","commits_url":"https://api.github.com/repos/nipy/nibabel/commits{/sha}","git_commits_url":"https://api.github.com/repos/nipy/nibabel/git/commits{/sha}","comments_url":"https://api.github.com/repos/nipy/nibabel/comments{/number}","issue_comment_url":"https://api.github.com/repos/nipy/nibabel/issues/comments{/number}","contents_url":"https://api.github.com/repos/nipy/nibabel/contents/{+path}","compare_url":"https://api.github.com/repos/nipy/nibabel/compare/{base}...{head}","merges_url":"https://api.github.com/repos/nipy/nibabel/merges","archive_url":"https://api.github.com/repos/nipy/nibabel/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/nipy/nibabel/downloads","issues_url":"https://api.github.com/repos/nipy/nibabel/issues{/number}","pulls_url":"https://api.github.com/repos/nipy/nibabel/pulls{/number}","milestones_url":"https://api.github.com/repos/nipy/nibabel/milestones{/number}","notifications_url":"https://api.github.com/repos/nipy/nibabel/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/nipy/nibabel/labels{/name}","releases_url":"https://api.github.com/repos/nipy/nibabel/releases{/id}","deployments_url":"https://api.github.com/repos/nipy/nibabel/deployments","created_at":"2010-07-22T16:28:30Z","updated_at":"2016-04-28T17:11:42Z","pushed_at":"2016-04-30T21:45:10Z","git_url":"git://github.com/nipy/nibabel.git","ssh_url":"git@github.com:nipy/nibabel.git","clone_url":"https://github.com/nipy/nibabel.git","svn_url":"https://github.com/nipy/nibabel","homepage":"http://nipy.org/nibabel/","size":19926,"stargazers_count":106,"watchers_count":106,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":106,"mirror_url":null,"open_issues_count":64,"forks":106,"open_issues":64,"watchers":106,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/nipy/nibabel/pulls/391"},"html":{"href":"https://github.com/nipy/nibabel/pull/391"},"issue":{"href":"https://api.github.com/repos/nipy/nibabel/issues/391"},"comments":{"href":"https://api.github.com/repos/nipy/nibabel/issues/391/comments"},"review_comments":{"href":"https://api.github.com/repos/nipy/nibabel/pulls/391/comments"},"review_comment":{"href":"https://api.github.com/repos/nipy/nibabel/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/nipy/nibabel/pulls/391/commits"},"statuses":{"href":"https://api.github.com/repos/nipy/nibabel/statuses/713ef7b64a3f1fb1b84dcdb03f070bdb6272940b"}}}}