{"action":"created","issue":{"url":"https://api.github.com/repos/samtools/htsjdk/issues/744","repository_url":"https://api.github.com/repos/samtools/htsjdk","labels_url":"https://api.github.com/repos/samtools/htsjdk/issues/744/labels{/name}","comments_url":"https://api.github.com/repos/samtools/htsjdk/issues/744/comments","events_url":"https://api.github.com/repos/samtools/htsjdk/issues/744/events","html_url":"https://github.com/samtools/htsjdk/pull/744","id":190094390,"number":744,"title":"Fix for issue #432 in Picard Tools. CreateSequenceDictionary stalls indefinitely with large genomes ","user":{"login":"SilinPavel","id":11179595,"avatar_url":"https://avatars.githubusercontent.com/u/11179595?v=3","gravatar_id":"","url":"https://api.github.com/users/SilinPavel","html_url":"https://github.com/SilinPavel","followers_url":"https://api.github.com/users/SilinPavel/followers","following_url":"https://api.github.com/users/SilinPavel/following{/other_user}","gists_url":"https://api.github.com/users/SilinPavel/gists{/gist_id}","starred_url":"https://api.github.com/users/SilinPavel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SilinPavel/subscriptions","organizations_url":"https://api.github.com/users/SilinPavel/orgs","repos_url":"https://api.github.com/users/SilinPavel/repos","events_url":"https://api.github.com/users/SilinPavel/events{/privacy}","received_events_url":"https://api.github.com/users/SilinPavel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-11-17T16:24:39Z","updated_at":"2016-11-23T12:51:15Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/samtools/htsjdk/pulls/744","html_url":"https://github.com/samtools/htsjdk/pull/744","diff_url":"https://github.com/samtools/htsjdk/pull/744.diff","patch_url":"https://github.com/samtools/htsjdk/pull/744.patch"},"body":"We introduce two pull requests to Htsjdk and Picard Tools(https://github.com/broadinstitute/picard/pull/687) to resolve the issue https://github.com/broadinstitute/picard/issues/432 .\r\n\r\nOriginal version of CreateSequenceDictionary creates and stores all reference sequences entirely in memory in a SAMHeader object and then writes it to a file.\r\n\r\nIn order to check sequence's names for uniqueness, they are added to a HashSet. This might lead to \"Out of memory\" exceptions for files with a large number of sequences .\r\n\r\n\r\n**Changes in Htsjdk:**\r\n\r\nIn our implementation we introduce a new \"on fly\" codec SAMSequenceDictionaryCodec, which encodes each sequence and directly writes it to the Dictionary file. This allows us not to store sequences in memory.\r\n\r\n\r\n**Changes in Picard Tools:**\r\n\r\nCreateSequenceDictionary now uses new SAMSequenceDictionaryCodec to write dictionary file.\r\n\r\nSortingCollection is used instead of a HashSet to verify the sequences names uniqueness , this allows us not to keep all the names in memory since identical names will be stored in the collection one after another and we can find duplicates in one pass. \r\n\r\nPlease pay attention, that now CreateSequenceDictionary doesn't write HD line, since we think that it is not required, SequenceDictionary must contain only SQ line, not HD. If our assumption is wrong, correct us, please.\r\n\r\n \r\n\r\n \r\n\r\nSo, if we look at the plot \"GC Times\" from JMC: FlightRecorder, we will see that standard version of CreateSequenceDictionary have a big trouble with a large amount of records (several millions):\r\n\r\n**The plot \"GC Times\" for a  Standard version:**\r\n \r\n![selection_041](https://cloud.githubusercontent.com/assets/11179595/20395648/16c18ec0-acfd-11e6-8576-bf4c52013e4f.png)\r\n![selection_043](https://cloud.githubusercontent.com/assets/11179595/20395649/16c275a6-acfd-11e6-9d54-63f1f20c50a4.png)\r\n\r\n\r\n**The plot \"GC Times\" for a fixed version:**\r\n![selection_040](https://cloud.githubusercontent.com/assets/11179595/20395646/16990892-acfd-11e6-829a-832b63679b73.png)\r\n![selection_042](https://cloud.githubusercontent.com/assets/11179595/20395647/16be48c8-acfd-11e6-8eac-2982b09fe183.png)\r\n\r\nHow we can see, now CreateSequenceDictionary processes a big reference without troubles, for example time of execution for file from issue https://github.com/broadinstitute/picard/issues/432 is:\r\n\r\n```\r\nreal 762.83\r\nuser 626.94\r\nsys 67.02\r\n```\r\n\r\nFull GC reports (jmc flight recorder):\r\n[JMC FR logs.zip](https://github.com/broadinstitute/picard/files/597773/JMC.FR.logs.zip)\r\n\r\n\r\n### Checklist\r\n\r\n- [x] Code compiles correctly\r\n- [ ] New tests covering changes and new functionality\r\n- [x] All tests passing\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/samtools/htsjdk/issues/comments/262504868","html_url":"https://github.com/samtools/htsjdk/pull/744#issuecomment-262504868","issue_url":"https://api.github.com/repos/samtools/htsjdk/issues/744","id":262504868,"user":{"login":"coveralls","id":2354108,"avatar_url":"https://avatars.githubusercontent.com/u/2354108?v=3","gravatar_id":"","url":"https://api.github.com/users/coveralls","html_url":"https://github.com/coveralls","followers_url":"https://api.github.com/users/coveralls/followers","following_url":"https://api.github.com/users/coveralls/following{/other_user}","gists_url":"https://api.github.com/users/coveralls/gists{/gist_id}","starred_url":"https://api.github.com/users/coveralls/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coveralls/subscriptions","organizations_url":"https://api.github.com/users/coveralls/orgs","repos_url":"https://api.github.com/users/coveralls/repos","events_url":"https://api.github.com/users/coveralls/events{/privacy}","received_events_url":"https://api.github.com/users/coveralls/received_events","type":"User","site_admin":false},"created_at":"2016-11-23T12:51:15Z","updated_at":"2016-11-23T12:51:15Z","body":"\n[![Coverage Status](https://coveralls.io/builds/8964543/badge)](https://coveralls.io/builds/8964543)\n\nCoverage increased (+0.02%) to 70.026% when pulling **97fbe637a10422c3cecc37df59dc5026e2949497 on EpamLifeSciencesTeam:epam-ls_CreateSequenceDictionary_fix** into **de27f18c9e623bdad822c530a555721ed81089f5 on samtools:master**.\n"}}
{"actor":{"display_login":"coveralls"}}