{"action":"created","comment":{"url":"https://api.github.com/repos/craffel/mir_eval/pulls/comments/52940035","id":52940035,"diff_hunk":"@@ -0,0 +1,243 @@\n+'''\n+The aim of a transcription algorithm is to produce a symbolic representation of\n+a recorded piece of music in the form of a set of discrete notes. There are\n+different ways to represent notes symbolically. Here we use the piano-roll\n+convention, meaning each notes has a start time, a duration (or end time), and\n+a single, constant, pitch value. Pitch values can be quantized (e.g. to a\n+semitone grid tuned to 440 Hz), but do not have to be. Also, the transcription\n+can contain the notes of a single instrument or voice (for example the melody),\n+or the notes of all instruments/voices in the recording. This module is\n+instrument agnostic: all notes in the estimate are compared against all notes\n+in the reference.\n+\n+There are many metrics for evaluating transcription algorithms. Here we limit\n+ourselves to the most simple and commonly used: given two sets of notes, we\n+count how many estimate notes match the reference, and how many do not. Based\n+on these counts we compute the precision, recall, and f-measure of the estimate\n+given the reference. The default criteria for considering two notes to be a\n+match are adopted from the MIREX \"Multiple fundamental frequency estimation and\n+tracking, task 2\" criteria:\n+(http://www.music-ir.org/mirex/wiki/2015:Multiple_Fundamental_Frequency_\\\n+Estimation_%26_Tracking_Results_-_MIREX_Dataset#Task_2:Note_Tracking_.28NT.29):\n+\n+\"This subtask is evaluated in two different ways. In the first setup , a\n+returned note is assumed correct if its onset is within +-50ms of a ref note\n+and its F0 is within +- quarter tone of the corresponding reference note,\n+ignoring the returned offset values. In the second setup, on top of the above\n+requirements, a correct returned note is required to have an offset value\n+within 20% of the ref notes duration around the ref note's offset, or within\n+50ms whichever is larger.\"\n+\n+In short, we compute precision, recall and f-measure, once without taking\n+offsets into account, and the second time with.\n+\n+For further details see Salamon, 2013 (page 186), and references therein:\n+\n+    Salamon, J. (2013). Melody Extraction from Polyphonic Music Signals.\n+    Ph.D. thesis, Universitat Pompeu Fabra, Barcelona, Spain, 2013.\n+\n+\n+Conventions\n+-----------\n+\n+Notes should be provided in the form of an interval array and a pitch list.\n+The interval array contains two columns, one for note onsets and the second\n+for note offsets (each row represents a single note). The pitch list contains\n+the corresponding note pitch values (one value per note), represented by their\n+fundamental frequency (f0) in Hertz.\n+\n+Metrics\n+-------\n+\n+* :func:`mir_eval.transcription.precision_recall_f1`: The precision,\n+  recall, and F-measure of the note transcription, where an estimated note is\n+  considered correct if its pitch, onset and (optionally) offset are\n+  sufficiently close to a reference note\n+\n+'''\n+\n+import numpy as np\n+import collections\n+from . import util\n+import warnings\n+\n+\n+def validate(ref_intervals, ref_pitches, est_intervals, est_pitches):\n+    \"\"\"Checks that the input annotations to a metric look like time intervals\n+    and a pitch list, and throws helpful errors if not.\n+\n+    Parameters\n+    ----------\n+    ref_intervals : np.ndarray, shape=(n,2)\n+        Array of reference notes time intervals (onset and offset times)\n+    ref_pitches: list, len=n\n+        List of reference pitch values in Hertz\n+    est_intervals : np.ndarray, shape=(m,2)\n+        Array of estimated notes time intervals (onset and offset times)\n+    est_pitches : list, len=m\n+        List of estimated pitch values in Hertz\n+    \"\"\"\n+    # If reference or estimated notes are empty, warn\n+    if ref_intervals.size == 0:\n+        warnings.warn(\"Reference note intervals are empty.\")\n+    if len(ref_pitches) == 0:\n+        warnings.warn(\"Reference note pitches are empty.\")\n+    if est_intervals.size == 0:\n+        warnings.warn(\"Estimate note intervals are empty.\")\n+    if len(est_pitches) == 0:\n+        warnings.warn(\"Estimate note pitches are empty.\")\n+\n+    # Make sure intervals and pitches match in length\n+    if not len(ref_intervals) == len(ref_pitches):\n+        warnings.warn(\"Reference intervals and pitches have different \"\n+                      \"lengths.\")\n+    if not len(est_intervals) == len(est_pitches):\n+        warnings.warn(\"Estimate intervals and pitches have different lengths.\")\n+\n+    # Make sure all pitch values are positive\n+    if np.min(ref_pitches) <= 0:\n+        warnings.warn(\"Reference contains at least one non-positive pitch \"\n+                      \"value\")\n+    if np.min(est_pitches) <= 0:\n+        warnings.warn(\"Estimate contains at least one non-positive pitch \"\n+                      \"value\")\n+\n+\n+def precision_recall_f1(ref_intervals, ref_pitches, est_intervals, est_pitches,\n+                        onset_tolerance=0.05, pitch_tolerance=50.0,\n+                        offset_ratio=0.2, with_offset=False):\n+    \"\"\"Compute the Precision, Recall and F-measure of correct vs incorrectly\n+    transcribed notes. \"Correctness\" is determined based on note onset, pitch\n+    and (optionally) offset: an estimated note is assumed correct if its onset\n+    is within +-50ms of a ref note and its pitch (F0) is within +- quarter tone\n+    (50 cents) of the corresponding reference note. If with_offset is False,\n+    note offsets are ignored in the comparison. It with_offset is True,\n+    on top of the above requirements, a correct returned note is required to\n+    have an offset value within 20% (by default, adjustable via the\n+    offset_ratio parameter) of the ref note's duration around the ref note's\n+    offset, or within 50ms, whichever is larger.","path":"mir_eval/transcription.py","position":null,"original_position":118,"commit_id":"606fa79ba1c572cf8cace6026e5460209b7e880b","original_commit_id":"a8f4d6845b5016a3625b65845cfb803fb690acbb","user":{"login":"justinsalamon","id":3009670,"avatar_url":"https://avatars.githubusercontent.com/u/3009670?v=3","gravatar_id":"","url":"https://api.github.com/users/justinsalamon","html_url":"https://github.com/justinsalamon","followers_url":"https://api.github.com/users/justinsalamon/followers","following_url":"https://api.github.com/users/justinsalamon/following{/other_user}","gists_url":"https://api.github.com/users/justinsalamon/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsalamon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsalamon/subscriptions","organizations_url":"https://api.github.com/users/justinsalamon/orgs","repos_url":"https://api.github.com/users/justinsalamon/repos","events_url":"https://api.github.com/users/justinsalamon/events{/privacy}","received_events_url":"https://api.github.com/users/justinsalamon/received_events","type":"User","site_admin":false},"body":"I've changed the minimum-offset-tolerance to be user settable in 606fa79 (in fact it turns out no minimum offset tolerance was being used, so this also caught a bug that made it through the tests).\r\n\r\nRegarding `with_offset`, I had it the way you propose before, but I actually prefer to have an explicit argument for this (hence `with_offset`). The `evaluate` method always computes the metrics both with and without offsets. If ignoring offsets is made implicit (by `offset_ratio=None`), then when `offset_ratio=None` the two sets of metrics will be identical, even though one is supposedly with offsets and the other without (unless I add some clunky logic into `evaluate`, which I'd rather not).\r\n\r\nBy having an explicit argument, this difference is made as clear as possible to the user, and since I expect most users to use the `evaluate` method, I think this avoids potential confusion. Long story short, unless there's serious opposition, I'd rather keep it as is.","created_at":"2016-02-15T20:21:00Z","updated_at":"2016-02-15T20:21:00Z","html_url":"https://github.com/craffel/mir_eval/pull/170#discussion_r52940035","pull_request_url":"https://api.github.com/repos/craffel/mir_eval/pulls/170","_links":{"self":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/comments/52940035"},"html":{"href":"https://github.com/craffel/mir_eval/pull/170#discussion_r52940035"},"pull_request":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/170"}}},"pull_request":{"url":"https://api.github.com/repos/craffel/mir_eval/pulls/170","id":58773839,"html_url":"https://github.com/craffel/mir_eval/pull/170","diff_url":"https://github.com/craffel/mir_eval/pull/170.diff","patch_url":"https://github.com/craffel/mir_eval/pull/170.patch","issue_url":"https://api.github.com/repos/craffel/mir_eval/issues/170","number":170,"state":"open","locked":false,"title":"Transcription [wip]","user":{"login":"justinsalamon","id":3009670,"avatar_url":"https://avatars.githubusercontent.com/u/3009670?v=3","gravatar_id":"","url":"https://api.github.com/users/justinsalamon","html_url":"https://github.com/justinsalamon","followers_url":"https://api.github.com/users/justinsalamon/followers","following_url":"https://api.github.com/users/justinsalamon/following{/other_user}","gists_url":"https://api.github.com/users/justinsalamon/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsalamon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsalamon/subscriptions","organizations_url":"https://api.github.com/users/justinsalamon/orgs","repos_url":"https://api.github.com/users/justinsalamon/repos","events_url":"https://api.github.com/users/justinsalamon/events{/privacy}","received_events_url":"https://api.github.com/users/justinsalamon/received_events","type":"User","site_admin":false},"body":"This will (eventually) contain all the code for transcription evaluation (and tests)","created_at":"2016-02-09T16:41:46Z","updated_at":"2016-02-15T20:21:01Z","closed_at":null,"merged_at":null,"merge_commit_sha":"61ac9c4a6a7814c4b0a85e35c99e17b88e8cd2c5","assignee":null,"milestone":{"url":"https://api.github.com/repos/craffel/mir_eval/milestones/6","html_url":"https://github.com/craffel/mir_eval/milestones/0.3.0","labels_url":"https://api.github.com/repos/craffel/mir_eval/milestones/6/labels","id":1578395,"number":6,"title":"0.3.0","description":"","creator":{"login":"bmcfee","id":1190540,"avatar_url":"https://avatars.githubusercontent.com/u/1190540?v=3","gravatar_id":"","url":"https://api.github.com/users/bmcfee","html_url":"https://github.com/bmcfee","followers_url":"https://api.github.com/users/bmcfee/followers","following_url":"https://api.github.com/users/bmcfee/following{/other_user}","gists_url":"https://api.github.com/users/bmcfee/gists{/gist_id}","starred_url":"https://api.github.com/users/bmcfee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmcfee/subscriptions","organizations_url":"https://api.github.com/users/bmcfee/orgs","repos_url":"https://api.github.com/users/bmcfee/repos","events_url":"https://api.github.com/users/bmcfee/events{/privacy}","received_events_url":"https://api.github.com/users/bmcfee/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":1,"state":"open","created_at":"2016-02-11T19:53:17Z","updated_at":"2016-02-15T19:35:41Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/craffel/mir_eval/pulls/170/commits","review_comments_url":"https://api.github.com/repos/craffel/mir_eval/pulls/170/comments","review_comment_url":"https://api.github.com/repos/craffel/mir_eval/pulls/comments{/number}","comments_url":"https://api.github.com/repos/craffel/mir_eval/issues/170/comments","statuses_url":"https://api.github.com/repos/craffel/mir_eval/statuses/606fa79ba1c572cf8cace6026e5460209b7e880b","head":{"label":"craffel:transcription","ref":"transcription","sha":"606fa79ba1c572cf8cace6026e5460209b7e880b","user":{"login":"craffel","id":417568,"avatar_url":"https://avatars.githubusercontent.com/u/417568?v=3","gravatar_id":"","url":"https://api.github.com/users/craffel","html_url":"https://github.com/craffel","followers_url":"https://api.github.com/users/craffel/followers","following_url":"https://api.github.com/users/craffel/following{/other_user}","gists_url":"https://api.github.com/users/craffel/gists{/gist_id}","starred_url":"https://api.github.com/users/craffel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craffel/subscriptions","organizations_url":"https://api.github.com/users/craffel/orgs","repos_url":"https://api.github.com/users/craffel/repos","events_url":"https://api.github.com/users/craffel/events{/privacy}","received_events_url":"https://api.github.com/users/craffel/received_events","type":"User","site_admin":false},"repo":{"id":11929631,"name":"mir_eval","full_name":"craffel/mir_eval","owner":{"login":"craffel","id":417568,"avatar_url":"https://avatars.githubusercontent.com/u/417568?v=3","gravatar_id":"","url":"https://api.github.com/users/craffel","html_url":"https://github.com/craffel","followers_url":"https://api.github.com/users/craffel/followers","following_url":"https://api.github.com/users/craffel/following{/other_user}","gists_url":"https://api.github.com/users/craffel/gists{/gist_id}","starred_url":"https://api.github.com/users/craffel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craffel/subscriptions","organizations_url":"https://api.github.com/users/craffel/orgs","repos_url":"https://api.github.com/users/craffel/repos","events_url":"https://api.github.com/users/craffel/events{/privacy}","received_events_url":"https://api.github.com/users/craffel/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/craffel/mir_eval","description":"Evaluation functions for music/audio information retrieval/signal processing algorithms.","fork":false,"url":"https://api.github.com/repos/craffel/mir_eval","forks_url":"https://api.github.com/repos/craffel/mir_eval/forks","keys_url":"https://api.github.com/repos/craffel/mir_eval/keys{/key_id}","collaborators_url":"https://api.github.com/repos/craffel/mir_eval/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/craffel/mir_eval/teams","hooks_url":"https://api.github.com/repos/craffel/mir_eval/hooks","issue_events_url":"https://api.github.com/repos/craffel/mir_eval/issues/events{/number}","events_url":"https://api.github.com/repos/craffel/mir_eval/events","assignees_url":"https://api.github.com/repos/craffel/mir_eval/assignees{/user}","branches_url":"https://api.github.com/repos/craffel/mir_eval/branches{/branch}","tags_url":"https://api.github.com/repos/craffel/mir_eval/tags","blobs_url":"https://api.github.com/repos/craffel/mir_eval/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/craffel/mir_eval/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/craffel/mir_eval/git/refs{/sha}","trees_url":"https://api.github.com/repos/craffel/mir_eval/git/trees{/sha}","statuses_url":"https://api.github.com/repos/craffel/mir_eval/statuses/{sha}","languages_url":"https://api.github.com/repos/craffel/mir_eval/languages","stargazers_url":"https://api.github.com/repos/craffel/mir_eval/stargazers","contributors_url":"https://api.github.com/repos/craffel/mir_eval/contributors","subscribers_url":"https://api.github.com/repos/craffel/mir_eval/subscribers","subscription_url":"https://api.github.com/repos/craffel/mir_eval/subscription","commits_url":"https://api.github.com/repos/craffel/mir_eval/commits{/sha}","git_commits_url":"https://api.github.com/repos/craffel/mir_eval/git/commits{/sha}","comments_url":"https://api.github.com/repos/craffel/mir_eval/comments{/number}","issue_comment_url":"https://api.github.com/repos/craffel/mir_eval/issues/comments{/number}","contents_url":"https://api.github.com/repos/craffel/mir_eval/contents/{+path}","compare_url":"https://api.github.com/repos/craffel/mir_eval/compare/{base}...{head}","merges_url":"https://api.github.com/repos/craffel/mir_eval/merges","archive_url":"https://api.github.com/repos/craffel/mir_eval/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/craffel/mir_eval/downloads","issues_url":"https://api.github.com/repos/craffel/mir_eval/issues{/number}","pulls_url":"https://api.github.com/repos/craffel/mir_eval/pulls{/number}","milestones_url":"https://api.github.com/repos/craffel/mir_eval/milestones{/number}","notifications_url":"https://api.github.com/repos/craffel/mir_eval/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/craffel/mir_eval/labels{/name}","releases_url":"https://api.github.com/repos/craffel/mir_eval/releases{/id}","deployments_url":"https://api.github.com/repos/craffel/mir_eval/deployments","created_at":"2013-08-06T17:04:10Z","updated_at":"2016-02-05T20:27:00Z","pushed_at":"2016-02-15T20:17:47Z","git_url":"git://github.com/craffel/mir_eval.git","ssh_url":"git@github.com:craffel/mir_eval.git","clone_url":"https://github.com/craffel/mir_eval.git","svn_url":"https://github.com/craffel/mir_eval","homepage":null,"size":11553,"stargazers_count":43,"watchers_count":43,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":17,"mirror_url":null,"open_issues_count":9,"forks":17,"open_issues":9,"watchers":43,"default_branch":"master"}},"base":{"label":"craffel:master","ref":"master","sha":"1676798fc325886168801ac3090c1febcea497a2","user":{"login":"craffel","id":417568,"avatar_url":"https://avatars.githubusercontent.com/u/417568?v=3","gravatar_id":"","url":"https://api.github.com/users/craffel","html_url":"https://github.com/craffel","followers_url":"https://api.github.com/users/craffel/followers","following_url":"https://api.github.com/users/craffel/following{/other_user}","gists_url":"https://api.github.com/users/craffel/gists{/gist_id}","starred_url":"https://api.github.com/users/craffel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craffel/subscriptions","organizations_url":"https://api.github.com/users/craffel/orgs","repos_url":"https://api.github.com/users/craffel/repos","events_url":"https://api.github.com/users/craffel/events{/privacy}","received_events_url":"https://api.github.com/users/craffel/received_events","type":"User","site_admin":false},"repo":{"id":11929631,"name":"mir_eval","full_name":"craffel/mir_eval","owner":{"login":"craffel","id":417568,"avatar_url":"https://avatars.githubusercontent.com/u/417568?v=3","gravatar_id":"","url":"https://api.github.com/users/craffel","html_url":"https://github.com/craffel","followers_url":"https://api.github.com/users/craffel/followers","following_url":"https://api.github.com/users/craffel/following{/other_user}","gists_url":"https://api.github.com/users/craffel/gists{/gist_id}","starred_url":"https://api.github.com/users/craffel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craffel/subscriptions","organizations_url":"https://api.github.com/users/craffel/orgs","repos_url":"https://api.github.com/users/craffel/repos","events_url":"https://api.github.com/users/craffel/events{/privacy}","received_events_url":"https://api.github.com/users/craffel/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/craffel/mir_eval","description":"Evaluation functions for music/audio information retrieval/signal processing algorithms.","fork":false,"url":"https://api.github.com/repos/craffel/mir_eval","forks_url":"https://api.github.com/repos/craffel/mir_eval/forks","keys_url":"https://api.github.com/repos/craffel/mir_eval/keys{/key_id}","collaborators_url":"https://api.github.com/repos/craffel/mir_eval/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/craffel/mir_eval/teams","hooks_url":"https://api.github.com/repos/craffel/mir_eval/hooks","issue_events_url":"https://api.github.com/repos/craffel/mir_eval/issues/events{/number}","events_url":"https://api.github.com/repos/craffel/mir_eval/events","assignees_url":"https://api.github.com/repos/craffel/mir_eval/assignees{/user}","branches_url":"https://api.github.com/repos/craffel/mir_eval/branches{/branch}","tags_url":"https://api.github.com/repos/craffel/mir_eval/tags","blobs_url":"https://api.github.com/repos/craffel/mir_eval/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/craffel/mir_eval/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/craffel/mir_eval/git/refs{/sha}","trees_url":"https://api.github.com/repos/craffel/mir_eval/git/trees{/sha}","statuses_url":"https://api.github.com/repos/craffel/mir_eval/statuses/{sha}","languages_url":"https://api.github.com/repos/craffel/mir_eval/languages","stargazers_url":"https://api.github.com/repos/craffel/mir_eval/stargazers","contributors_url":"https://api.github.com/repos/craffel/mir_eval/contributors","subscribers_url":"https://api.github.com/repos/craffel/mir_eval/subscribers","subscription_url":"https://api.github.com/repos/craffel/mir_eval/subscription","commits_url":"https://api.github.com/repos/craffel/mir_eval/commits{/sha}","git_commits_url":"https://api.github.com/repos/craffel/mir_eval/git/commits{/sha}","comments_url":"https://api.github.com/repos/craffel/mir_eval/comments{/number}","issue_comment_url":"https://api.github.com/repos/craffel/mir_eval/issues/comments{/number}","contents_url":"https://api.github.com/repos/craffel/mir_eval/contents/{+path}","compare_url":"https://api.github.com/repos/craffel/mir_eval/compare/{base}...{head}","merges_url":"https://api.github.com/repos/craffel/mir_eval/merges","archive_url":"https://api.github.com/repos/craffel/mir_eval/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/craffel/mir_eval/downloads","issues_url":"https://api.github.com/repos/craffel/mir_eval/issues{/number}","pulls_url":"https://api.github.com/repos/craffel/mir_eval/pulls{/number}","milestones_url":"https://api.github.com/repos/craffel/mir_eval/milestones{/number}","notifications_url":"https://api.github.com/repos/craffel/mir_eval/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/craffel/mir_eval/labels{/name}","releases_url":"https://api.github.com/repos/craffel/mir_eval/releases{/id}","deployments_url":"https://api.github.com/repos/craffel/mir_eval/deployments","created_at":"2013-08-06T17:04:10Z","updated_at":"2016-02-05T20:27:00Z","pushed_at":"2016-02-15T20:17:47Z","git_url":"git://github.com/craffel/mir_eval.git","ssh_url":"git@github.com:craffel/mir_eval.git","clone_url":"https://github.com/craffel/mir_eval.git","svn_url":"https://github.com/craffel/mir_eval","homepage":null,"size":11553,"stargazers_count":43,"watchers_count":43,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":17,"mirror_url":null,"open_issues_count":9,"forks":17,"open_issues":9,"watchers":43,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/170"},"html":{"href":"https://github.com/craffel/mir_eval/pull/170"},"issue":{"href":"https://api.github.com/repos/craffel/mir_eval/issues/170"},"comments":{"href":"https://api.github.com/repos/craffel/mir_eval/issues/170/comments"},"review_comments":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/170/comments"},"review_comment":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/craffel/mir_eval/pulls/170/commits"},"statuses":{"href":"https://api.github.com/repos/craffel/mir_eval/statuses/606fa79ba1c572cf8cace6026e5460209b7e880b"}}}}