{"action":"created","issue":{"url":"https://api.github.com/repos/dstreett/Super-Deduper/issues/40","repository_url":"https://api.github.com/repos/dstreett/Super-Deduper","labels_url":"https://api.github.com/repos/dstreett/Super-Deduper/issues/40/labels{/name}","comments_url":"https://api.github.com/repos/dstreett/Super-Deduper/issues/40/comments","events_url":"https://api.github.com/repos/dstreett/Super-Deduper/issues/40/events","html_url":"https://github.com/dstreett/Super-Deduper/issues/40","id":167638616,"number":40,"title":"Many duplicates (unexpected)","user":{"login":"adomingues","id":2607577,"avatar_url":"https://avatars.githubusercontent.com/u/2607577?v=3","gravatar_id":"","url":"https://api.github.com/users/adomingues","html_url":"https://github.com/adomingues","followers_url":"https://api.github.com/users/adomingues/followers","following_url":"https://api.github.com/users/adomingues/following{/other_user}","gists_url":"https://api.github.com/users/adomingues/gists{/gist_id}","starred_url":"https://api.github.com/users/adomingues/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adomingues/subscriptions","organizations_url":"https://api.github.com/users/adomingues/orgs","repos_url":"https://api.github.com/users/adomingues/repos","events_url":"https://api.github.com/users/adomingues/events{/privacy}","received_events_url":"https://api.github.com/users/adomingues/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-07-26T15:34:58Z","updated_at":"2016-07-28T14:02:14Z","closed_at":null,"body":"Hi David,\r\nI am trying super-deduper in some very very large datasets (like hundreds of millions reads). The reads have a random barcodes (unique molecule identifiers) in 5' and my goal is to collapse those reads that have the same barcode+sequence to get rid of PCR duplicates pre-mapping.\r\n\r\nI thought I could do it with:\r\n\r\n```bash\r\nsuper_deduper -U data/NET-seq_HeLa_S3_Rep1.fastq.gz --output-prefix results/processed_reads/NET-seq_HeLa_S3_Rep1.de_ --gzip-output --length 6\r\n```\r\n\r\nwhich was impressively fast and used very little memory - great job! However the output has relatively few reads:\r\n\r\n> Final:| reads: 175226879 | duplicates: 174702121 | reads_written: 524758 | percent: 99.70 | discarded: 76297 | total_seconds: 757.36 | reads/sec: 231365.37\r\n\r\nIs this an expected outcome or am I not used the correct settings. Assuming I can even use super-deduper for this particular task.\r\n\r\nCheers\r\n"},"comment":{"url":"https://api.github.com/repos/dstreett/Super-Deduper/issues/comments/235903431","html_url":"https://github.com/dstreett/Super-Deduper/issues/40#issuecomment-235903431","issue_url":"https://api.github.com/repos/dstreett/Super-Deduper/issues/40","id":235903431,"user":{"login":"adomingues","id":2607577,"avatar_url":"https://avatars.githubusercontent.com/u/2607577?v=3","gravatar_id":"","url":"https://api.github.com/users/adomingues","html_url":"https://github.com/adomingues","followers_url":"https://api.github.com/users/adomingues/followers","following_url":"https://api.github.com/users/adomingues/following{/other_user}","gists_url":"https://api.github.com/users/adomingues/gists{/gist_id}","starred_url":"https://api.github.com/users/adomingues/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adomingues/subscriptions","organizations_url":"https://api.github.com/users/adomingues/orgs","repos_url":"https://api.github.com/users/adomingues/repos","events_url":"https://api.github.com/users/adomingues/events{/privacy}","received_events_url":"https://api.github.com/users/adomingues/received_events","type":"User","site_admin":false},"created_at":"2016-07-28T14:02:14Z","updated_at":"2016-07-28T14:02:14Z","body":"Thank you for the quick reply Matt,\r\n\r\n> super deduper uses a key sequence within the read and duplicate keys are expected to be PCR duplicates, your 6bp molecular barcode alone however is not enough to determine a duplicate (there are only 4096 possible combinations and you have many, many more reads).\r\n\r\nOf course! I missed that even though I read the super-deduper paper :/\r\n\r\n>  In this dataset you have the inclusion of a random hexamer on each fragment to help in the PCR dup call, it will be interesting to see how\r\n\r\nIn our lab we routinely use 2x4bp barcodes, one at the 5' end and another at the 3', for small RNAseq experiments in which we expect small sequence diversity and want to distinguish that from true PCR duplicates. For these libraries, the deduplication is then done simply (and naively) by collapsing all reads that have the exact same sequence. This usually works well, just using unix tools, but this dataset is so large that I had to try something more efficient.  That said, if super-deduper works well for this data-set, I will probably include it in my small RNAseq pipeline.\r\n\r\nAnd to be clear, this is not my data or experimental design. This has been previously published by another lab and I am just trying to re-analyse it. \r\n\r\n----\r\nNow for the test results:\r\n\r\n>  LSBATCH: User input\r\n> /home/adomingu/imb-kettinggr/common_bin/Super-Deduper/super_deduper -U data/NET-seq_HeLa_S3_Rep1.fastq.gz --output-prefix results/processed_reads/NET-seq_HeLa_S3_Rep1.de_7-15 --gzip-output **--start 7 --length 15**\r\n> \r\n> \r\n> Resource usage summary:\r\n> \r\n>     CPU time :               10855.24 sec.\r\n>     Max Memory :             15186.68 MB\r\n>     Average Memory :         14514.08 MB\r\n>     Total Requested Memory : 25600.00 MB\r\n>     Delta Memory :           10413.32 MB\r\n>     (Delta: the difference between total requested memory and actual max usage.)\r\n>     Max Swap :               15229 MB\r\n> \r\n>     Max Processes :          4\r\n>     Max Threads :            5\r\n> \r\n> The output (if any) follows:\r\n> \r\n> Final:| reads: 175224149 | duplicates: 147473141 | reads_written: 27751008 | **percent: 84.16** | discarded: 79027 | total_seconds: 10787.34 | reads/sec: 16243.50\r\n\r\n>  LSBATCH: User input\r\n> /home/adomingu/imb-kettinggr/common_bin/Super-Deduper/super_deduper -U data/NET-seq_HeLa_S3_Rep1.fastq.gz --output-prefix results/processed_reads/NET-seq_HeLa_S3_Rep1.de_1-21 --gzip-output **--start 1 --length 21**\r\n> \r\n> Resource usage summary:\r\n> \r\n>     CPU time :               14958.32 sec.\r\n>     Max Memory :             21254.66 MB\r\n>     Average Memory :         20434.38 MB\r\n>     Total Requested Memory : 25600.00 MB\r\n>     Delta Memory :           4345.34 MB\r\n>     (Delta: the difference between total requested memory and actual max usage.)\r\n>     Max Swap :               21297 MB\r\n> \r\n>     Max Processes :          4\r\n>     Max Threads :            5\r\n> \r\n> The output (if any) follows:\r\n> \r\n> Final:| reads: 175221640 | duplicates: 135016167 | reads_written: 40205473 | **percent: 77.05** | discarded: 81536 | total_seconds: 14864.68 | reads/sec: 11787.78\r\n\r\nI have also posted the memory usage in case you are curious about it. It seems like extending the size of the region used to determine duplication, and including the barcode, did increase the number of reads retained to a more acceptable number, but the effect is not has striking as I anticipated. I am running another test with `--start 1 --length 26`. The idea is to find a large enough portion of the read to detect unique sequences.\r\n\r\nAntónio "}}
{"actor":{"display_login":"adomingues"}}