{"action":"created","issue":{"url":"https://api.github.com/repos/CGATOxford/UMI-tools/issues/36","repository_url":"https://api.github.com/repos/CGATOxford/UMI-tools","labels_url":"https://api.github.com/repos/CGATOxford/UMI-tools/issues/36/labels{/name}","comments_url":"https://api.github.com/repos/CGATOxford/UMI-tools/issues/36/comments","events_url":"https://api.github.com/repos/CGATOxford/UMI-tools/issues/36/events","html_url":"https://github.com/CGATOxford/UMI-tools/pull/36","id":158159059,"number":36,"title":"Ts speed up stats","user":{"login":"TomSmithCGAT","id":6096414,"avatar_url":"https://avatars.githubusercontent.com/u/6096414?v=3","gravatar_id":"","url":"https://api.github.com/users/TomSmithCGAT","html_url":"https://github.com/TomSmithCGAT","followers_url":"https://api.github.com/users/TomSmithCGAT/followers","following_url":"https://api.github.com/users/TomSmithCGAT/following{/other_user}","gists_url":"https://api.github.com/users/TomSmithCGAT/gists{/gist_id}","starred_url":"https://api.github.com/users/TomSmithCGAT/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TomSmithCGAT/subscriptions","organizations_url":"https://api.github.com/users/TomSmithCGAT/orgs","repos_url":"https://api.github.com/users/TomSmithCGAT/repos","events_url":"https://api.github.com/users/TomSmithCGAT/events{/privacy}","received_events_url":"https://api.github.com/users/TomSmithCGAT/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-06-02T14:45:00Z","updated_at":"2016-06-03T12:31:54Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/CGATOxford/UMI-tools/pulls/36","html_url":"https://github.com/CGATOxford/UMI-tools/pull/36","diff_url":"https://github.com/CGATOxford/UMI-tools/pull/36.diff","patch_url":"https://github.com/CGATOxford/UMI-tools/pull/36.patch"},"body":"@IanSudbery After discussing with @MikeDMorgan I've implemented a two-stage random sampling since the original array of probabilities contains the same probability multiple times (e.g 1/n, 2/n, 3/n, ..., where n is the total umi counts) This speeds up the random sampling ~5-fold for the BAM from Roy Francis (see below) so the total run time decreased from ~28 minutes to ~10 minutes. This may be a particular case where the gains are larger since the distribution of UMIs is a long way off uniform. I'd expect the improvement to be marginal where the distribution of umis is more \"random\".\r\n\r\nI've checked the stats output is consistent. \r\n\r\nCan you think of any scenario where this method of sampling would be significantly less efficient?\r\n\r\npreviously...\r\nncalls  tottime  percall  cumtime  percall filename:lineno(function)\r\n  4400216 1373.266    0.000 1426.919    0.000 {method 'choice' of 'mtrand.RandomState' objects}\r\n  4400216   18.888    0.000 1445.807    0.000 dedup.py:776(getUmis)\r\n\r\ntwo-stage sampling...\r\n  8995437  219.319    0.000  268.225    0.000 {method 'choice' of 'mtrand.RandomState' objects}\r\n  4400216   24.821    0.000  300.054    0.000 dedup.py:789(getUmis)"},"comment":{"url":"https://api.github.com/repos/CGATOxford/UMI-tools/issues/comments/223566485","html_url":"https://github.com/CGATOxford/UMI-tools/pull/36#issuecomment-223566485","issue_url":"https://api.github.com/repos/CGATOxford/UMI-tools/issues/36","id":223566485,"user":{"login":"IanSudbery","id":5391758,"avatar_url":"https://avatars.githubusercontent.com/u/5391758?v=3","gravatar_id":"","url":"https://api.github.com/users/IanSudbery","html_url":"https://github.com/IanSudbery","followers_url":"https://api.github.com/users/IanSudbery/followers","following_url":"https://api.github.com/users/IanSudbery/following{/other_user}","gists_url":"https://api.github.com/users/IanSudbery/gists{/gist_id}","starred_url":"https://api.github.com/users/IanSudbery/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IanSudbery/subscriptions","organizations_url":"https://api.github.com/users/IanSudbery/orgs","repos_url":"https://api.github.com/users/IanSudbery/repos","events_url":"https://api.github.com/users/IanSudbery/events{/privacy}","received_events_url":"https://api.github.com/users/IanSudbery/received_events","type":"User","site_admin":false},"created_at":"2016-06-03T12:31:54Z","updated_at":"2016-06-03T12:31:54Z","body":"memory usage looks good. I recommend merging."}}
{"actor":{"display_login":"IanSudbery"}}