{"action":"created","issue":{"url":"https://api.github.com/repos/disq-bio/disq/issues/34","repository_url":"https://api.github.com/repos/disq-bio/disq","labels_url":"https://api.github.com/repos/disq-bio/disq/issues/34/labels{/name}","comments_url":"https://api.github.com/repos/disq-bio/disq/issues/34/comments","events_url":"https://api.github.com/repos/disq-bio/disq/issues/34/events","html_url":"https://github.com/disq-bio/disq/pull/34","id":360879641,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE1OTg5MzEz","number":34,"title":"[DISQ-33] Duplicate reads bug","user":{"login":"tomwhite","id":85085,"node_id":"MDQ6VXNlcjg1MDg1","avatar_url":"https://avatars3.githubusercontent.com/u/85085?v=4","gravatar_id":"","url":"https://api.github.com/users/tomwhite","html_url":"https://github.com/tomwhite","followers_url":"https://api.github.com/users/tomwhite/followers","following_url":"https://api.github.com/users/tomwhite/following{/other_user}","gists_url":"https://api.github.com/users/tomwhite/gists{/gist_id}","starred_url":"https://api.github.com/users/tomwhite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomwhite/subscriptions","organizations_url":"https://api.github.com/users/tomwhite/orgs","repos_url":"https://api.github.com/users/tomwhite/repos","events_url":"https://api.github.com/users/tomwhite/events{/privacy}","received_events_url":"https://api.github.com/users/tomwhite/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-09-17T13:52:20Z","updated_at":"2018-09-25T14:47:47Z","closed_at":"2018-09-25T14:47:23Z","author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/disq-bio/disq/pulls/34","html_url":"https://github.com/disq-bio/disq/pull/34","diff_url":"https://github.com/disq-bio/disq/pull/34.diff","patch_url":"https://github.com/disq-bio/disq/pull/34.patch"},"body":"This reproduces the problem (#33) and provides a fix. It also adds log debugging to provide diagnostic information about splits and BGZF chunk boundaries to help with debugging issues like this in the future.\r\n\r\nThe bug occurs when a BGZF block boundary coincides with a split boundary. Here's a log file for a run using a file that produced an errant count:\r\n\r\n```\r\n10:49:42.990 DEBUG BamSource - PathChunk for partition 294: PathChunk{path=file:/home/tom/bamaftersparksortsize.bam, span=9865016001:489-9898557440:65535}\r\n10:49:43.239 DEBUG BamSource - PathChunk for partition 295: PathChunk{path=file:/home/tom/bamaftersparksortsize.bam, span=9898557440:190-9932111872:65535}\r\n```\r\nFrom the log we see that the spans for the two paths overlap, since the first ends at 9898557440:65535 which is *after* 9898557440:190, where the second starts. This results in duplicate reads in the two partitions. The first span should end at the beginning of the BGZF block where the next span starts, i.e. 9898557440:0.\r\n\r\nThe problem was observed since 9898557440 is 9440MB, which is a multiple of the default block size, 32MB for local files.  (It is *not* a multiple of the default block size for HDFS, which is 128MB, so the problem was not observed on HDFS unless the split size was set explicitly to 32MB.)\r\n\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/disq-bio/disq/issues/comments/424374324","html_url":"https://github.com/disq-bio/disq/pull/34#issuecomment-424374324","issue_url":"https://api.github.com/repos/disq-bio/disq/issues/34","id":424374324,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNDM3NDMyNA==","user":{"login":"tomwhite","id":85085,"node_id":"MDQ6VXNlcjg1MDg1","avatar_url":"https://avatars3.githubusercontent.com/u/85085?v=4","gravatar_id":"","url":"https://api.github.com/users/tomwhite","html_url":"https://github.com/tomwhite","followers_url":"https://api.github.com/users/tomwhite/followers","following_url":"https://api.github.com/users/tomwhite/following{/other_user}","gists_url":"https://api.github.com/users/tomwhite/gists{/gist_id}","starred_url":"https://api.github.com/users/tomwhite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomwhite/subscriptions","organizations_url":"https://api.github.com/users/tomwhite/orgs","repos_url":"https://api.github.com/users/tomwhite/repos","events_url":"https://api.github.com/users/tomwhite/events{/privacy}","received_events_url":"https://api.github.com/users/tomwhite/received_events","type":"User","site_admin":false},"created_at":"2018-09-25T14:47:47Z","updated_at":"2018-09-25T14:47:47Z","author_association":"CONTRIBUTOR","body":"Thanks @jamesemery"}}
{"actor":{"display_login":"tomwhite"}}