{"action":"created","issue":{"url":"https://api.github.com/repos/cosmir/openmic-2018/issues/24","repository_url":"https://api.github.com/repos/cosmir/openmic-2018","labels_url":"https://api.github.com/repos/cosmir/openmic-2018/issues/24/labels{/name}","comments_url":"https://api.github.com/repos/cosmir/openmic-2018/issues/24/comments","events_url":"https://api.github.com/repos/cosmir/openmic-2018/issues/24/events","html_url":"https://github.com/cosmir/openmic-2018/issues/24","id":355385538,"node_id":"MDU6SXNzdWUzNTUzODU1Mzg=","number":24,"title":"De-duping","user":{"login":"bmcfee","id":1190540,"node_id":"MDQ6VXNlcjExOTA1NDA=","avatar_url":"https://avatars3.githubusercontent.com/u/1190540?v=4","gravatar_id":"","url":"https://api.github.com/users/bmcfee","html_url":"https://github.com/bmcfee","followers_url":"https://api.github.com/users/bmcfee/followers","following_url":"https://api.github.com/users/bmcfee/following{/other_user}","gists_url":"https://api.github.com/users/bmcfee/gists{/gist_id}","starred_url":"https://api.github.com/users/bmcfee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmcfee/subscriptions","organizations_url":"https://api.github.com/users/bmcfee/orgs","repos_url":"https://api.github.com/users/bmcfee/repos","events_url":"https://api.github.com/users/bmcfee/events{/privacy}","received_events_url":"https://api.github.com/users/bmcfee/received_events","type":"User","site_admin":false},"labels":[{"id":1026381045,"node_id":"MDU6TGFiZWwxMDI2MzgxMDQ1","url":"https://api.github.com/repos/cosmir/openmic-2018/labels/bug","name":"bug","color":"d73a4a","default":true},{"id":1026381050,"node_id":"MDU6TGFiZWwxMDI2MzgxMDUw","url":"https://api.github.com/repos/cosmir/openmic-2018/labels/invalid","name":"invalid","color":"e4e669","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-30T01:02:25Z","updated_at":"2018-08-30T01:38:41Z","closed_at":null,"author_association":"COLLABORATOR","body":"Inspired by a recent tweet on deduping in cifar, I went and performed a similar analysis on our proposed train-test split.  We indeed have some dupes in our dataset, though it seems like they're relatively few.  This does already occur in our (current) train-test split though, so we'll need to do some cleanup to prevent that from happening.\r\n\r\n### Procedure\r\n\r\n1. Map all tracks to vggish quantized codewords, and flatten the time dimension.  This gives for train a 15000x1280 matrix; for test, 5000x1280.\r\n2. Perform a nearest neighbor search from test to train using L1 distance (though any distance would work).\r\n3. Look for any suspiciously small distances to nearest neighbors.\r\n\r\n### Results\r\n\r\n![image](https://user-images.githubusercontent.com/1190540/44823187-f2346c00-abcc-11e8-80b4-571877fbf227.png)\r\n\r\nThis plot shows the distribution of distances from test to 1st, 2nd, ..., 5th nearest neighbor from the training set.  As you can see from the fliers on column 1, there are a few suspiciously low values.  In fact, two of them are identically 0.  The offending culprits are\r\n\r\n```\r\n007954_184320 -> 120195_184320\r\n074658_372480 -> 074328_372480\r\n```\r\nChecking the metadata fields, these appear to be exact duplicates with dirty metadata.\r\n\r\nThere's one more soft near-match, which has distance 9057 (median is about 54000):\r\n```\r\n021334_11520 -> 020898_11520\r\n```\r\nwhich again, by metadata, seems like a like match.  (The artist ids are identical, but the other metadata is garbage.)\r\n\r\nSince in all of these cases, an artist-id filter using metadata would have caught this, I'm wondering if that's sufficient to fix it?  I didn't include this originally because I was under the impression that no artist had more than one track in the collection.  Apparently that assumption was incorrect.\r\n\r\nI haven't done a similar analysis on the train-train data, but I suspect there are many more dupes than are caught here.\r\n\r\nWhat do you recommend @ejhumphrey @simondurand ?  I could try to resolve this in the splitter, but I'm not sure how artist conditioning will play with stratified sampling -- in sklearn splitters, we can do either stratification or grouping, but not both simultaneously.  I'll take a look and see if we just got unlucky with this random split, but it'd be nice to have a proper solution in place."},"comment":{"url":"https://api.github.com/repos/cosmir/openmic-2018/issues/comments/417161246","html_url":"https://github.com/cosmir/openmic-2018/issues/24#issuecomment-417161246","issue_url":"https://api.github.com/repos/cosmir/openmic-2018/issues/24","id":417161246,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNzE2MTI0Ng==","user":{"login":"ejhumphrey","id":926379,"node_id":"MDQ6VXNlcjkyNjM3OQ==","avatar_url":"https://avatars0.githubusercontent.com/u/926379?v=4","gravatar_id":"","url":"https://api.github.com/users/ejhumphrey","html_url":"https://github.com/ejhumphrey","followers_url":"https://api.github.com/users/ejhumphrey/followers","following_url":"https://api.github.com/users/ejhumphrey/following{/other_user}","gists_url":"https://api.github.com/users/ejhumphrey/gists{/gist_id}","starred_url":"https://api.github.com/users/ejhumphrey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ejhumphrey/subscriptions","organizations_url":"https://api.github.com/users/ejhumphrey/orgs","repos_url":"https://api.github.com/users/ejhumphrey/repos","events_url":"https://api.github.com/users/ejhumphrey/events{/privacy}","received_events_url":"https://api.github.com/users/ejhumphrey/received_events","type":"User","site_admin":false},"created_at":"2018-08-30T01:38:41Z","updated_at":"2018-08-30T01:38:41Z","author_association":"CONTRIBUTOR","body":"all grumbling aside, yea, I thought we did an artist_id split as well ... guess that's on me.\r\n\r\nfirst, scope of the problem: why not do a full 20k x 20k distance matrix, all splits aside? I'm curious to know what the full duplicate count is. same goes for artist IDs...\r\n\r\nsecond, fixing it: what's the right corrective action here? I see three possibilities, in increasing correctness:\r\n1. leave it, flag it as a known issue, and split the data to avoid train/test contamination\r\n2. drop the dupes and let it be openmic-2018, containing 19980+ examples\r\n3. figure out which positive labels we'd lose for each example, and try to manually backfill from the disjoint set of artists not already covered in the collection.\r\n\r\nI think you're suggesting (1) @bmcfee? I don't know which I prefer. (3) sounds like a lot of work, and I think making a point of this at ISMIR is a fine lesson to share."}}
{"actor":{"display_login":"ejhumphrey"}}