{"action":"created","issue":{"url":"https://api.github.com/repos/biosails/pheniqs/issues/16","repository_url":"https://api.github.com/repos/biosails/pheniqs","labels_url":"https://api.github.com/repos/biosails/pheniqs/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/biosails/pheniqs/issues/16/comments","events_url":"https://api.github.com/repos/biosails/pheniqs/issues/16/events","html_url":"https://github.com/biosails/pheniqs/issues/16","id":369635767,"node_id":"MDU6SXNzdWUzNjk2MzU3Njc=","number":16,"title":"Pheniqs only processes a small fraction of reads","user":{"login":"rspreafico-sgi","id":34698433,"node_id":"MDQ6VXNlcjM0Njk4NDMz","avatar_url":"https://avatars2.githubusercontent.com/u/34698433?v=4","gravatar_id":"","url":"https://api.github.com/users/rspreafico-sgi","html_url":"https://github.com/rspreafico-sgi","followers_url":"https://api.github.com/users/rspreafico-sgi/followers","following_url":"https://api.github.com/users/rspreafico-sgi/following{/other_user}","gists_url":"https://api.github.com/users/rspreafico-sgi/gists{/gist_id}","starred_url":"https://api.github.com/users/rspreafico-sgi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rspreafico-sgi/subscriptions","organizations_url":"https://api.github.com/users/rspreafico-sgi/orgs","repos_url":"https://api.github.com/users/rspreafico-sgi/repos","events_url":"https://api.github.com/users/rspreafico-sgi/events{/privacy}","received_events_url":"https://api.github.com/users/rspreafico-sgi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-10-12T17:09:15Z","updated_at":"2018-11-08T08:42:32Z","closed_at":null,"author_association":"NONE","body":"Hi, I am trying trying to demux a PE run using inline barcodes on read 2:\r\n\r\n```\r\n{\r\n    \"input\": [\r\n        \"LS01-2018-09-24-LS-UA-CAAF-001_R1_001.fastq.gz\",\r\n        \"LS01-2018-09-24-LS-UA-CAAF-001_R2_001.fastq.gz\"\r\n    ],\r\n    \"transform\": { \r\n        \"token\": [ \"0::\", \"1:14:\" ],\r\n        \"segment pattern\": [ \"0\", \"1\" ]\r\n    },\r\n    \"multiplex\": {\r\n        \"transform\": { \"token\": [ \"1::12\" ] },\r\n        \"codec\": {\r\n            \"@TATGAACGTCCG\": { \"barcode\": [ \"TATGAACGTCCG\" ], \"output\": [ \"BC4CAAF1-10m-20180912-1120_L001_R1_001.fastq.gz\", \"BC4CAAF1-10m-20180912-1120_L001_R2_001.fastq.gz\" ] },\r\n            \"@CCACATTGGGTC\": { \"barcode\": [ \"CCACATTGGGTC\" ], \"output\": [ \"BC1CAAF1-200m-20180912-0920_L001_R1_001.fastq.gz\", \"BC1CAAF1-200m-20180912-0920_L001_R2_001.fastq.gz\" ] },\r\n            \"@TCAGTCAGATGA\": { \"barcode\": [ \"TCAGTCAGATGA\" ], \"output\": [ \"BC4CAAF1-10m-20180912-1015_L001_R1_001.fastq.gz\", \"BC4CAAF1-10m-20180912-1015_L001_R2_001.fastq.gz\" ] },\r\n            \"@AAGTCACACACA\": { \"barcode\": [ \"AAGTCACACACA\" ], \"output\": [ \"BC1CAAF1-200m-20180912-1020_L001_R1_001.fastq.gz\", \"BC1CAAF1-200m-20180912-1020_L001_R2_001.fastq.gz\" ] },\r\n            \"@GCTGTGATTCGA\": { \"barcode\": [ \"GCTGTGATTCGA\" ], \"output\": [ \"BC2CAAF2-200m-20180912-0920_L001_R1_001.fastq.gz\", \"BC2CAAF2-200m-20180912-0920_L001_R2_001.fastq.gz\" ] }\r\n        },\r\n        \"undetermined\": { \"output\": [ \"LS01-undetermined_L001_R1_001.fastq.gz\", \"LS01-undetermined_L001_R2_001.fastq.gz\" ] },\r\n        \"algorithm\": \"mdd\",\r\n        \"include filtered\": true\r\n    }\r\n}\r\n```\r\n\r\nHowever, I always get back only 116 reads, despite the fact that the two input files have thousands of reads. I have tried with PAML too, same results.\r\n\r\n```\r\n{\r\n    \"multiplex\": {\r\n        \"average classified distance\": 0.219298245614035,\r\n        \"average pf classified distance\": 0.219298245614035,\r\n        \"classified\": [\r\n            {\r\n                \"ID\": \"AAGTCACACACA\",\r\n                \"PU\": \"AAGTCACACACA\",\r\n                \"average distance\": 0.461538461538461,\r\n                \"average pf distance\": 0.461538461538461,\r\n                \"barcode\": [\r\n                    \"AAGTCACACACA\"\r\n                ],\r\n                \"concentration\": 0.198,\r\n                \"count\": 13,\r\n                \"index\": 1,\r\n                \"pf count\": 13,\r\n                \"pf fraction\": 1.0,\r\n                \"pf pooled classified fraction\": 0.114035087719298,\r\n                \"pf pooled fraction\": 0.112068965517241,\r\n                \"pooled classified fraction\": 0.114035087719298,\r\n                \"pooled fraction\": 0.112068965517241\r\n            },\r\n            {\r\n                \"ID\": \"CCACATTGGGTC\",\r\n                \"PU\": \"CCACATTGGGTC\",\r\n                \"average distance\": 0.071428571428571,\r\n                \"average pf distance\": 0.071428571428571,\r\n                \"barcode\": [\r\n                    \"CCACATTGGGTC\"\r\n                ],\r\n                \"concentration\": 0.198,\r\n                \"count\": 28,\r\n                \"index\": 2,\r\n                \"pf count\": 28,\r\n                \"pf fraction\": 1.0,\r\n                \"pf pooled classified fraction\": 0.245614035087719,\r\n                \"pf pooled fraction\": 0.241379310344827,\r\n                \"pooled classified fraction\": 0.245614035087719,\r\n                \"pooled fraction\": 0.241379310344827\r\n            },\r\n            {\r\n                \"ID\": \"GCTGTGATTCGA\",\r\n                \"PU\": \"GCTGTGATTCGA\",\r\n                \"average distance\": 0.260869565217391,\r\n                \"average pf distance\": 0.260869565217391,\r\n                \"barcode\": [\r\n                    \"GCTGTGATTCGA\"\r\n                ],\r\n                \"concentration\": 0.198,\r\n                \"count\": 46,\r\n                \"index\": 3,\r\n                \"pf count\": 46,\r\n                \"pf fraction\": 1.0,\r\n                \"pf pooled classified fraction\": 0.403508771929824,\r\n                \"pf pooled fraction\": 0.396551724137931,\r\n                \"pooled classified fraction\": 0.403508771929824,\r\n                \"pooled fraction\": 0.396551724137931\r\n            },\r\n            {\r\n                \"ID\": \"TATGAACGTCCG\",\r\n                \"PU\": \"TATGAACGTCCG\",\r\n                \"average distance\": 1.0,\r\n                \"average pf distance\": 1.0,\r\n                \"barcode\": [\r\n                    \"TATGAACGTCCG\"\r\n                ],\r\n                \"concentration\": 0.198,\r\n                \"count\": 1,\r\n                \"index\": 4,\r\n                \"pf count\": 1,\r\n                \"pf fraction\": 1.0,\r\n                \"pf pooled classified fraction\": 0.008771929824561,\r\n                \"pf pooled fraction\": 0.008620689655172,\r\n                \"pooled classified fraction\": 0.008771929824561,\r\n                \"pooled fraction\": 0.008620689655172\r\n            },\r\n            {\r\n                \"ID\": \"TCAGTCAGATGA\",\r\n                \"PU\": \"TCAGTCAGATGA\",\r\n                \"average distance\": 0.153846153846153,\r\n                \"average pf distance\": 0.153846153846153,\r\n                \"barcode\": [\r\n                    \"TCAGTCAGATGA\"\r\n                ],\r\n                \"concentration\": 0.198,\r\n                \"count\": 26,\r\n                \"index\": 5,\r\n                \"pf count\": 26,\r\n                \"pf fraction\": 1.0,\r\n                \"pf pooled classified fraction\": 0.228070175438596,\r\n                \"pf pooled fraction\": 0.224137931034482,\r\n                \"pooled classified fraction\": 0.228070175438596,\r\n                \"pooled fraction\": 0.224137931034482\r\n            }\r\n        ],\r\n        \"classified count\": 114,\r\n        \"classified fraction\": 0.982758620689655,\r\n        \"classified pf fraction\": 1.0,\r\n        \"count\": 116,\r\n        \"pf classified count\": 114,\r\n        \"pf classified fraction\": 0.982758620689655,\r\n        \"pf count\": 116,\r\n        \"pf fraction\": 1.0,\r\n        \"unclassified\": {\r\n            \"ID\": \"undetermined\",\r\n            \"PU\": \"undetermined\",\r\n            \"count\": 2,\r\n            \"index\": 0,\r\n            \"pf count\": 2,\r\n            \"pf fraction\": 1.0,\r\n            \"pf pooled classified fraction\": 0.017543859649122,\r\n            \"pf pooled fraction\": 0.017241379310344,\r\n            \"pooled classified fraction\": 0.017543859649122,\r\n            \"pooled fraction\": 0.017241379310344\r\n        }\r\n    }\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/biosails/pheniqs/issues/comments/436916079","html_url":"https://github.com/biosails/pheniqs/issues/16#issuecomment-436916079","issue_url":"https://api.github.com/repos/biosails/pheniqs/issues/16","id":436916079,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNjkxNjA3OQ==","user":{"login":"moonwatcher","id":214621,"node_id":"MDQ6VXNlcjIxNDYyMQ==","avatar_url":"https://avatars2.githubusercontent.com/u/214621?v=4","gravatar_id":"","url":"https://api.github.com/users/moonwatcher","html_url":"https://github.com/moonwatcher","followers_url":"https://api.github.com/users/moonwatcher/followers","following_url":"https://api.github.com/users/moonwatcher/following{/other_user}","gists_url":"https://api.github.com/users/moonwatcher/gists{/gist_id}","starred_url":"https://api.github.com/users/moonwatcher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/moonwatcher/subscriptions","organizations_url":"https://api.github.com/users/moonwatcher/orgs","repos_url":"https://api.github.com/users/moonwatcher/repos","events_url":"https://api.github.com/users/moonwatcher/events{/privacy}","received_events_url":"https://api.github.com/users/moonwatcher/received_events","type":"User","site_admin":false},"created_at":"2018-11-08T08:42:32Z","updated_at":"2018-11-08T08:42:32Z","author_association":"CONTRIBUTOR","body":"To be clear. It is theoretically possible that it’s a more subtle bug. The multi threaded double buffering mechanism in Pheniqs is quite a complex thing...  it has been in production for quite a while and those kind of bugs are VERY rare these days. \r\n\r\nI have seen many zip files that were generated with slight violations that are not compatible. For instance the gzip on MacOS and the gzip on an an average gnu based Linux machine are incompatible is some scenarios and it can make the compression engine in HTSLib read a few blocks fine and stop. Usually either decompressing the file with standard gzip will work and you can then recomporess it. You can also feed Pheniqs from stdin but only one input... (for instance and interleaved FASTQ or SAM."}}
{"actor":{"display_login":"moonwatcher"}}