{"action":"created","issue":{"url":"https://api.github.com/repos/hammerlab/guacamole/issues/386","repository_url":"https://api.github.com/repos/hammerlab/guacamole","labels_url":"https://api.github.com/repos/hammerlab/guacamole/issues/386/labels{/name}","comments_url":"https://api.github.com/repos/hammerlab/guacamole/issues/386/comments","events_url":"https://api.github.com/repos/hammerlab/guacamole/issues/386/events","html_url":"https://github.com/hammerlab/guacamole/issues/386","id":135853396,"number":386,"title":"avoid reshuffling all reads at startup","user":{"login":"timodonnell","id":467896,"avatar_url":"https://avatars.githubusercontent.com/u/467896?v=3","gravatar_id":"","url":"https://api.github.com/users/timodonnell","html_url":"https://github.com/timodonnell","followers_url":"https://api.github.com/users/timodonnell/followers","following_url":"https://api.github.com/users/timodonnell/following{/other_user}","gists_url":"https://api.github.com/users/timodonnell/gists{/gist_id}","starred_url":"https://api.github.com/users/timodonnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timodonnell/subscriptions","organizations_url":"https://api.github.com/users/timodonnell/orgs","repos_url":"https://api.github.com/users/timodonnell/repos","events_url":"https://api.github.com/users/timodonnell/events{/privacy}","received_events_url":"https://api.github.com/users/timodonnell/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-02-23T20:22:04Z","updated_at":"2016-03-18T22:30:54Z","closed_at":null,"body":"Our BAMs are generally sorted by alignment position, so our current method of doing a full repartition at startup (which I believe more or less sends every read to another node) is very inefficient. Most of the reads could just stay where they are if we partitioned the data more intelligently (e.g. in the best case would only have to move reads that overlap two partitions).\r\n\r\nHere's the function where we do this:\r\nhttps://github.com/hammerlab/guacamole/blob/master/src/main/scala/org/hammerlab/guacamole/DistributedUtil.scala#L558\r\n\r\nWe assign reads to arbitrary \"task numbers,\" partition by task number and then map over the partitions. Unless Spark is more clever than I think, this is shuffling more or less every read over the network.\r\n\r\nFirst step here would be to check if what I'm talking about is true by looking at how many reads are getting sent across the network in the above function.\r\n\r\nGetting this to work right is going to require more Spark knowledge than I have, so I'm filing this in case @ryan-williams or @e5c are interested in taking a crack at it sometime. I think it could substantially improve Guacamole usability on the cluster. "},"comment":{"url":"https://api.github.com/repos/hammerlab/guacamole/issues/comments/198566300","html_url":"https://github.com/hammerlab/guacamole/issues/386#issuecomment-198566300","issue_url":"https://api.github.com/repos/hammerlab/guacamole/issues/386","id":198566300,"user":{"login":"jstjohn","id":459014,"avatar_url":"https://avatars.githubusercontent.com/u/459014?v=3","gravatar_id":"","url":"https://api.github.com/users/jstjohn","html_url":"https://github.com/jstjohn","followers_url":"https://api.github.com/users/jstjohn/followers","following_url":"https://api.github.com/users/jstjohn/following{/other_user}","gists_url":"https://api.github.com/users/jstjohn/gists{/gist_id}","starred_url":"https://api.github.com/users/jstjohn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jstjohn/subscriptions","organizations_url":"https://api.github.com/users/jstjohn/orgs","repos_url":"https://api.github.com/users/jstjohn/repos","events_url":"https://api.github.com/users/jstjohn/events{/privacy}","received_events_url":"https://api.github.com/users/jstjohn/received_events","type":"User","site_admin":false},"created_at":"2016-03-18T22:30:54Z","updated_at":"2016-03-18T22:30:54Z","body":"I have been thinking about this a bit. In a sorted stream context you can do really nice things with the sorted property. For example a given pileup element can be created as soon as all reads overlapping the a locus are read in, this is a knowable thing from the sorted stream b/c once you encounter a read with start location beyond some pileup locus, you have definitely seen every read you need to see to construct those previous loci. Some kind of priorityQueue of pileups as you stream through the reads which you can keep popping off as you encounter reads at more distant loci.\r\n\r\nYou could probably also do something slick with having different iterators seek to different chunks of the bam file in a spark way, and have N sorted streams going in parallel. Basically just farm out chunks of the sorted bam file. \r\n\r\nI started trying to hack together a version of the pileupFlatMap function that works on a sorted stream of reads, rather than an RDD. Have not had time to make much progress though, and I am definitely not ready to claim ownership of that task."}}