{"action":"created","issue":{"url":"https://api.github.com/repos/nanoporetech/tombo/issues/36","repository_url":"https://api.github.com/repos/nanoporetech/tombo","labels_url":"https://api.github.com/repos/nanoporetech/tombo/issues/36/labels{/name}","comments_url":"https://api.github.com/repos/nanoporetech/tombo/issues/36/comments","events_url":"https://api.github.com/repos/nanoporetech/tombo/issues/36/events","html_url":"https://github.com/nanoporetech/tombo/issues/36","id":300770217,"number":36,"title":"Huge memory demand in resquiggle.","user":{"login":"JohnUrban","id":7783314,"avatar_url":"https://avatars2.githubusercontent.com/u/7783314?v=4","gravatar_id":"","url":"https://api.github.com/users/JohnUrban","html_url":"https://github.com/JohnUrban","followers_url":"https://api.github.com/users/JohnUrban/followers","following_url":"https://api.github.com/users/JohnUrban/following{/other_user}","gists_url":"https://api.github.com/users/JohnUrban/gists{/gist_id}","starred_url":"https://api.github.com/users/JohnUrban/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JohnUrban/subscriptions","organizations_url":"https://api.github.com/users/JohnUrban/orgs","repos_url":"https://api.github.com/users/JohnUrban/repos","events_url":"https://api.github.com/users/JohnUrban/events{/privacy}","received_events_url":"https://api.github.com/users/JohnUrban/received_events","type":"User","site_admin":false},"labels":[{"id":754143540,"url":"https://api.github.com/repos/nanoporetech/tombo/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-02-27T19:53:58Z","updated_at":"2018-03-02T21:06:03Z","closed_at":null,"author_association":"NONE","body":"Hi Marcus,\r\n\r\nI'm wondering if you have insight into what exactly is the memory hog in resquiggle (exceeding 67 GB).\r\n```\r\nWed Feb 21 16:40:21 EST 2018\r\nresquiggle\r\nGetting file list.\r\nUsing default canonical ***** DNA ***** model.\r\nEstimating global scale parameter\r\nCorrecting 8644 files with 1 subgroup(s)/read(s) each (Will print a dot for each 1000 reads completed).\r\n.slurmstepd: Job 18951741 exceeded memory limit (67181424 > 67108864), being killed\r\nslurmstepd: *** JOB 18951741 ON node850 CANCELLED AT 2018-02-23T20:32:32 ***\r\nslurmstepd: Exceeded job memory limit\r\n```\r\n\r\nI am using the much more sensitive minimap index I discussed previously and I did not previously have memory issues with the default index. Therefore, I am thinking it is either:\r\n(i) the mapping step, or \r\n(ii) a tombo step working with more difficult reads that mapped with the more sensitive index -- if that even makes sense.\r\n\r\nLet me know what you think. Many thanks as usual.\r\n\r\nBest,\r\n\r\n  John\r\n"},"comment":{"url":"https://api.github.com/repos/nanoporetech/tombo/issues/comments/370052288","html_url":"https://github.com/nanoporetech/tombo/issues/36#issuecomment-370052288","issue_url":"https://api.github.com/repos/nanoporetech/tombo/issues/36","id":370052288,"user":{"login":"marcus1487","id":13755444,"avatar_url":"https://avatars1.githubusercontent.com/u/13755444?v=4","gravatar_id":"","url":"https://api.github.com/users/marcus1487","html_url":"https://github.com/marcus1487","followers_url":"https://api.github.com/users/marcus1487/followers","following_url":"https://api.github.com/users/marcus1487/following{/other_user}","gists_url":"https://api.github.com/users/marcus1487/gists{/gist_id}","starred_url":"https://api.github.com/users/marcus1487/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcus1487/subscriptions","organizations_url":"https://api.github.com/users/marcus1487/orgs","repos_url":"https://api.github.com/users/marcus1487/repos","events_url":"https://api.github.com/users/marcus1487/events{/privacy}","received_events_url":"https://api.github.com/users/marcus1487/received_events","type":"User","site_admin":false},"created_at":"2018-03-02T21:06:03Z","updated_at":"2018-03-02T21:06:03Z","author_association":"COLLABORATOR","body":"The reads are not mapped before the estimate scale parameter step. Reads are mapped one at a time right before the correction step (following the `Correcting #### files` line). So I am not sure how your command was able to complete so much faster with the new index. My first guess would be that the read information was cached on what ever machine you were using and thus the speed up was actually from the processes not having to access the disk."}}
{"actor":{"display_login":"marcus1487"}}