{"action":"created","comment":{"url":"https://api.github.com/repos/craffel/mir_eval/pulls/comments/71670818","id":71670818,"diff_hunk":"@@ -66,11 +66,27 @@ def validate(reference_sources, estimated_sources):\n                          '= {}'.format(reference_sources.shape,\n                                        estimated_sources.shape))\n \n+    if reference_sources.ndim > 3 or estimated_sources.ndim > 3:\n+        raise ValueError('The number of dimensions is too high (must be less '\n+                         'than 3). reference_sources.ndim = {}, '\n+                         'estimated_sources.ndim '\n+                         '= {}'.format(reference_sources.ndim,\n+                                       estimated_sources.ndim))\n+\n     if reference_sources.size == 0:\n         warnings.warn(\"reference_sources is empty, should be of size \"\n                       \"(nsrc, nsample).  sdr, sir, sar, and perm will all \"\n                       \"be empty np.ndarrays\")\n-    elif np.any(np.all(reference_sources == 0, axis=1)):\n+    elif (reference_sources.ndim == 2 and\n+            np.any(np.all(reference_sources == 0, axis=1))):\n+        raise ValueError('All the reference sources should be non-silent (not '\n+                         'all-zeros), but at least one of the reference '\n+                         'sources is all 0s, which introduces ambiguity to the'\n+                         ' evaluation. (Otherwise we can add infinitely many '\n+                         'all-zero sources.)')\n+    elif (reference_sources.ndim > 2 and\n+            np.any(np.all(np.sum(reference_sources, axis=-1) == 0,\n+                          axis=1))):","path":"mir_eval/separation.py","position":25,"original_position":25,"commit_id":"42f03a03c5b430fc2dfea45995dc423802adc6df","original_commit_id":"42f03a03c5b430fc2dfea45995dc423802adc6df","user":{"login":"ErikJohnsonCU","id":6766142,"avatar_url":"https://avatars.githubusercontent.com/u/6766142?v=3","gravatar_id":"","url":"https://api.github.com/users/ErikJohnsonCU","html_url":"https://github.com/ErikJohnsonCU","followers_url":"https://api.github.com/users/ErikJohnsonCU/followers","following_url":"https://api.github.com/users/ErikJohnsonCU/following{/other_user}","gists_url":"https://api.github.com/users/ErikJohnsonCU/gists{/gist_id}","starred_url":"https://api.github.com/users/ErikJohnsonCU/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ErikJohnsonCU/subscriptions","organizations_url":"https://api.github.com/users/ErikJohnsonCU/orgs","repos_url":"https://api.github.com/users/ErikJohnsonCU/repos","events_url":"https://api.github.com/users/ErikJohnsonCU/events{/privacy}","received_events_url":"https://api.github.com/users/ErikJohnsonCU/received_events","type":"User","site_admin":false},"body":"The reason this is a complicated check is that we want to accept any channel being entirely 0 (in the `ndim >= 2` case). Hence, the addition of the `np.sum(reference_sources, axis=-1)` which sums all the channels for the `ndim >= 2` case. Since the trailing dimension for `ndim >= 2` is the channels, a check of trailing dimension all-zeros will almost always be true (one sample has zeros on both channels).","created_at":"2016-07-21T09:05:52Z","updated_at":"2016-07-21T09:05:52Z","html_url":"https://github.com/craffel/mir_eval/pull/207#discussion_r71670818","pull_request_url":"https://api.github.com/repos/craffel/mir_eval/pulls/207","_links":{"self":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/comments/71670818"},"html":{"href":"https://github.com/craffel/mir_eval/pull/207#discussion_r71670818"},"pull_request":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/207"}}},"pull_request":{"url":"https://api.github.com/repos/craffel/mir_eval/pulls/207","id":76454159,"html_url":"https://github.com/craffel/mir_eval/pull/207","diff_url":"https://github.com/craffel/mir_eval/pull/207.diff","patch_url":"https://github.com/craffel/mir_eval/pull/207.patch","issue_url":"https://api.github.com/repos/craffel/mir_eval/issues/207","number":207,"state":"open","locked":false,"title":"Bss eval images","user":{"login":"ErikJohnsonCU","id":6766142,"avatar_url":"https://avatars.githubusercontent.com/u/6766142?v=3","gravatar_id":"","url":"https://api.github.com/users/ErikJohnsonCU","html_url":"https://github.com/ErikJohnsonCU","followers_url":"https://api.github.com/users/ErikJohnsonCU/followers","following_url":"https://api.github.com/users/ErikJohnsonCU/following{/other_user}","gists_url":"https://api.github.com/users/ErikJohnsonCU/gists{/gist_id}","starred_url":"https://api.github.com/users/ErikJohnsonCU/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ErikJohnsonCU/subscriptions","organizations_url":"https://api.github.com/users/ErikJohnsonCU/orgs","repos_url":"https://api.github.com/users/ErikJohnsonCU/repos","events_url":"https://api.github.com/users/ErikJohnsonCU/events{/privacy}","received_events_url":"https://api.github.com/users/ErikJohnsonCU/received_events","type":"User","site_admin":false},"body":"## Images Separation Evaluation\r\n\r\nA function has been added in `separation.py` (`bss_eval_images`) that allows for image separation evaluation as discussed in #68. This implementation is identical to that of the `BSS_eval` MATLAB implementation.\r\n\r\n## Implementation Testing\r\n\r\nIn order to validate the consistency of `bss_eval_images` with that of the MATLAB implementation, comparison testing has been performed. This comparison testing is available for anyone wishing to test it [here](https://github.com/ErikJohnsonCU/mir_eval_images_compare) (please let me know if there are any issues installing/running it!).\r\n\r\n## Regression Testing\r\n\r\nA suite of tests have been added to `test_separation.py` for regression testing. These tests follow the same style as for `bss_eval_sources` and `bss_eval_sources_framewise` and the `output0*.json` files have been updated accordingly. Generation of dual channel audio is done by expanding both references and estimates to the correct number of channels and then manipulating the estimates in order to generate non-identical channels (see the `__generate_multichannel` function in `test_separation.py`).\r\n\r\n**All feedback is welcome!**","created_at":"2016-07-06T15:41:41Z","updated_at":"2016-07-21T09:05:52Z","closed_at":null,"merged_at":null,"merge_commit_sha":"cb913a97816cd752d8a0e3b84d724075a0ed8d91","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/craffel/mir_eval/pulls/207/commits","review_comments_url":"https://api.github.com/repos/craffel/mir_eval/pulls/207/comments","review_comment_url":"https://api.github.com/repos/craffel/mir_eval/pulls/comments{/number}","comments_url":"https://api.github.com/repos/craffel/mir_eval/issues/207/comments","statuses_url":"https://api.github.com/repos/craffel/mir_eval/statuses/42f03a03c5b430fc2dfea45995dc423802adc6df","head":{"label":"faroit:bss_eval_images","ref":"bss_eval_images","sha":"42f03a03c5b430fc2dfea45995dc423802adc6df","user":{"login":"faroit","id":72940,"avatar_url":"https://avatars.githubusercontent.com/u/72940?v=3","gravatar_id":"","url":"https://api.github.com/users/faroit","html_url":"https://github.com/faroit","followers_url":"https://api.github.com/users/faroit/followers","following_url":"https://api.github.com/users/faroit/following{/other_user}","gists_url":"https://api.github.com/users/faroit/gists{/gist_id}","starred_url":"https://api.github.com/users/faroit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/faroit/subscriptions","organizations_url":"https://api.github.com/users/faroit/orgs","repos_url":"https://api.github.com/users/faroit/repos","events_url":"https://api.github.com/users/faroit/events{/privacy}","received_events_url":"https://api.github.com/users/faroit/received_events","type":"User","site_admin":false},"repo":{"id":32523873,"name":"mir_eval","full_name":"faroit/mir_eval","owner":{"login":"faroit","id":72940,"avatar_url":"https://avatars.githubusercontent.com/u/72940?v=3","gravatar_id":"","url":"https://api.github.com/users/faroit","html_url":"https://github.com/faroit","followers_url":"https://api.github.com/users/faroit/followers","following_url":"https://api.github.com/users/faroit/following{/other_user}","gists_url":"https://api.github.com/users/faroit/gists{/gist_id}","starred_url":"https://api.github.com/users/faroit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/faroit/subscriptions","organizations_url":"https://api.github.com/users/faroit/orgs","repos_url":"https://api.github.com/users/faroit/repos","events_url":"https://api.github.com/users/faroit/events{/privacy}","received_events_url":"https://api.github.com/users/faroit/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/faroit/mir_eval","description":"Evaluation functions for music/audio information retrieval/signal processing algorithms.","fork":true,"url":"https://api.github.com/repos/faroit/mir_eval","forks_url":"https://api.github.com/repos/faroit/mir_eval/forks","keys_url":"https://api.github.com/repos/faroit/mir_eval/keys{/key_id}","collaborators_url":"https://api.github.com/repos/faroit/mir_eval/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/faroit/mir_eval/teams","hooks_url":"https://api.github.com/repos/faroit/mir_eval/hooks","issue_events_url":"https://api.github.com/repos/faroit/mir_eval/issues/events{/number}","events_url":"https://api.github.com/repos/faroit/mir_eval/events","assignees_url":"https://api.github.com/repos/faroit/mir_eval/assignees{/user}","branches_url":"https://api.github.com/repos/faroit/mir_eval/branches{/branch}","tags_url":"https://api.github.com/repos/faroit/mir_eval/tags","blobs_url":"https://api.github.com/repos/faroit/mir_eval/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/faroit/mir_eval/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/faroit/mir_eval/git/refs{/sha}","trees_url":"https://api.github.com/repos/faroit/mir_eval/git/trees{/sha}","statuses_url":"https://api.github.com/repos/faroit/mir_eval/statuses/{sha}","languages_url":"https://api.github.com/repos/faroit/mir_eval/languages","stargazers_url":"https://api.github.com/repos/faroit/mir_eval/stargazers","contributors_url":"https://api.github.com/repos/faroit/mir_eval/contributors","subscribers_url":"https://api.github.com/repos/faroit/mir_eval/subscribers","subscription_url":"https://api.github.com/repos/faroit/mir_eval/subscription","commits_url":"https://api.github.com/repos/faroit/mir_eval/commits{/sha}","git_commits_url":"https://api.github.com/repos/faroit/mir_eval/git/commits{/sha}","comments_url":"https://api.github.com/repos/faroit/mir_eval/comments{/number}","issue_comment_url":"https://api.github.com/repos/faroit/mir_eval/issues/comments{/number}","contents_url":"https://api.github.com/repos/faroit/mir_eval/contents/{+path}","compare_url":"https://api.github.com/repos/faroit/mir_eval/compare/{base}...{head}","merges_url":"https://api.github.com/repos/faroit/mir_eval/merges","archive_url":"https://api.github.com/repos/faroit/mir_eval/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/faroit/mir_eval/downloads","issues_url":"https://api.github.com/repos/faroit/mir_eval/issues{/number}","pulls_url":"https://api.github.com/repos/faroit/mir_eval/pulls{/number}","milestones_url":"https://api.github.com/repos/faroit/mir_eval/milestones{/number}","notifications_url":"https://api.github.com/repos/faroit/mir_eval/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/faroit/mir_eval/labels{/name}","releases_url":"https://api.github.com/repos/faroit/mir_eval/releases{/id}","deployments_url":"https://api.github.com/repos/faroit/mir_eval/deployments","created_at":"2015-03-19T13:45:57Z","updated_at":"2016-06-01T10:39:11Z","pushed_at":"2016-07-19T11:57:20Z","git_url":"git://github.com/faroit/mir_eval.git","ssh_url":"git@github.com:faroit/mir_eval.git","clone_url":"https://github.com/faroit/mir_eval.git","svn_url":"https://github.com/faroit/mir_eval","homepage":null,"size":12893,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"craffel:master","ref":"master","sha":"aa06df166435f1a96bd5f111b7bb480e146d3429","user":{"login":"craffel","id":417568,"avatar_url":"https://avatars.githubusercontent.com/u/417568?v=3","gravatar_id":"","url":"https://api.github.com/users/craffel","html_url":"https://github.com/craffel","followers_url":"https://api.github.com/users/craffel/followers","following_url":"https://api.github.com/users/craffel/following{/other_user}","gists_url":"https://api.github.com/users/craffel/gists{/gist_id}","starred_url":"https://api.github.com/users/craffel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craffel/subscriptions","organizations_url":"https://api.github.com/users/craffel/orgs","repos_url":"https://api.github.com/users/craffel/repos","events_url":"https://api.github.com/users/craffel/events{/privacy}","received_events_url":"https://api.github.com/users/craffel/received_events","type":"User","site_admin":false},"repo":{"id":11929631,"name":"mir_eval","full_name":"craffel/mir_eval","owner":{"login":"craffel","id":417568,"avatar_url":"https://avatars.githubusercontent.com/u/417568?v=3","gravatar_id":"","url":"https://api.github.com/users/craffel","html_url":"https://github.com/craffel","followers_url":"https://api.github.com/users/craffel/followers","following_url":"https://api.github.com/users/craffel/following{/other_user}","gists_url":"https://api.github.com/users/craffel/gists{/gist_id}","starred_url":"https://api.github.com/users/craffel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craffel/subscriptions","organizations_url":"https://api.github.com/users/craffel/orgs","repos_url":"https://api.github.com/users/craffel/repos","events_url":"https://api.github.com/users/craffel/events{/privacy}","received_events_url":"https://api.github.com/users/craffel/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/craffel/mir_eval","description":"Evaluation functions for music/audio information retrieval/signal processing algorithms.","fork":false,"url":"https://api.github.com/repos/craffel/mir_eval","forks_url":"https://api.github.com/repos/craffel/mir_eval/forks","keys_url":"https://api.github.com/repos/craffel/mir_eval/keys{/key_id}","collaborators_url":"https://api.github.com/repos/craffel/mir_eval/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/craffel/mir_eval/teams","hooks_url":"https://api.github.com/repos/craffel/mir_eval/hooks","issue_events_url":"https://api.github.com/repos/craffel/mir_eval/issues/events{/number}","events_url":"https://api.github.com/repos/craffel/mir_eval/events","assignees_url":"https://api.github.com/repos/craffel/mir_eval/assignees{/user}","branches_url":"https://api.github.com/repos/craffel/mir_eval/branches{/branch}","tags_url":"https://api.github.com/repos/craffel/mir_eval/tags","blobs_url":"https://api.github.com/repos/craffel/mir_eval/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/craffel/mir_eval/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/craffel/mir_eval/git/refs{/sha}","trees_url":"https://api.github.com/repos/craffel/mir_eval/git/trees{/sha}","statuses_url":"https://api.github.com/repos/craffel/mir_eval/statuses/{sha}","languages_url":"https://api.github.com/repos/craffel/mir_eval/languages","stargazers_url":"https://api.github.com/repos/craffel/mir_eval/stargazers","contributors_url":"https://api.github.com/repos/craffel/mir_eval/contributors","subscribers_url":"https://api.github.com/repos/craffel/mir_eval/subscribers","subscription_url":"https://api.github.com/repos/craffel/mir_eval/subscription","commits_url":"https://api.github.com/repos/craffel/mir_eval/commits{/sha}","git_commits_url":"https://api.github.com/repos/craffel/mir_eval/git/commits{/sha}","comments_url":"https://api.github.com/repos/craffel/mir_eval/comments{/number}","issue_comment_url":"https://api.github.com/repos/craffel/mir_eval/issues/comments{/number}","contents_url":"https://api.github.com/repos/craffel/mir_eval/contents/{+path}","compare_url":"https://api.github.com/repos/craffel/mir_eval/compare/{base}...{head}","merges_url":"https://api.github.com/repos/craffel/mir_eval/merges","archive_url":"https://api.github.com/repos/craffel/mir_eval/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/craffel/mir_eval/downloads","issues_url":"https://api.github.com/repos/craffel/mir_eval/issues{/number}","pulls_url":"https://api.github.com/repos/craffel/mir_eval/pulls{/number}","milestones_url":"https://api.github.com/repos/craffel/mir_eval/milestones{/number}","notifications_url":"https://api.github.com/repos/craffel/mir_eval/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/craffel/mir_eval/labels{/name}","releases_url":"https://api.github.com/repos/craffel/mir_eval/releases{/id}","deployments_url":"https://api.github.com/repos/craffel/mir_eval/deployments","created_at":"2013-08-06T17:04:10Z","updated_at":"2016-07-21T08:54:59Z","pushed_at":"2016-07-19T11:57:20Z","git_url":"git://github.com/craffel/mir_eval.git","ssh_url":"git@github.com:craffel/mir_eval.git","clone_url":"https://github.com/craffel/mir_eval.git","svn_url":"https://github.com/craffel/mir_eval","homepage":null,"size":13844,"stargazers_count":63,"watchers_count":63,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":23,"mirror_url":null,"open_issues_count":13,"forks":23,"open_issues":13,"watchers":63,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/207"},"html":{"href":"https://github.com/craffel/mir_eval/pull/207"},"issue":{"href":"https://api.github.com/repos/craffel/mir_eval/issues/207"},"comments":{"href":"https://api.github.com/repos/craffel/mir_eval/issues/207/comments"},"review_comments":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/207/comments"},"review_comment":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/207/commits"},"statuses":{"href":"https://api.github.com/repos/craffel/mir_eval/statuses/42f03a03c5b430fc2dfea45995dc423802adc6df"}}}}
{"actor":{"display_login":"ErikJohnsonCU"}}