{"action":"created","issue":{"url":"https://api.github.com/repos/marbl/canu/issues/211","repository_url":"https://api.github.com/repos/marbl/canu","labels_url":"https://api.github.com/repos/marbl/canu/issues/211/labels{/name}","comments_url":"https://api.github.com/repos/marbl/canu/issues/211/comments","events_url":"https://api.github.com/repos/marbl/canu/issues/211/events","html_url":"https://github.com/marbl/canu/issues/211","id":168297016,"number":211,"title":"Parts of pipeline where SLURM is killing Canu for exceeding memory ","user":{"login":"JohnUrban","id":7783314,"avatar_url":"https://avatars.githubusercontent.com/u/7783314?v=3","gravatar_id":"","url":"https://api.github.com/users/JohnUrban","html_url":"https://github.com/JohnUrban","followers_url":"https://api.github.com/users/JohnUrban/followers","following_url":"https://api.github.com/users/JohnUrban/following{/other_user}","gists_url":"https://api.github.com/users/JohnUrban/gists{/gist_id}","starred_url":"https://api.github.com/users/JohnUrban/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JohnUrban/subscriptions","organizations_url":"https://api.github.com/users/JohnUrban/orgs","repos_url":"https://api.github.com/users/JohnUrban/repos","events_url":"https://api.github.com/users/JohnUrban/events{/privacy}","received_events_url":"https://api.github.com/users/JohnUrban/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-29T10:32:51Z","updated_at":"2016-08-02T17:13:29Z","closed_at":null,"body":"Hi,\r\n\r\nI recently began running a lot of Canu jobs again. In case it is of interest, I am sharing here parts of the pipeline that are being killed due to exceeding memory. Recall, I'm on SLURM (if that matters).\r\n\r\n1. Meryl in correction phase\r\nIn at least 1 run with a set of reads filtered some various way, it reproducibly fails to ask for enough memory.\r\n```\r\n##   Mem/CPU : 2432 MB\r\n##  Mem/Node : 16384 MB\r\n.\r\n.\r\n.\r\nslurmstepd: Job 13015929 exceeded memory limit (19924520 > 19922944), being killed\r\nslurmstepd: Exceeded job memory limiters --  6.66 Mmers/second\r\nslurmstepd: *** JOB 13015929 ON node413 CANCELLED AT 2016-07-28T22:55:01 ***\r\nslurmstepd: Exceeded step memory limit at some point.\r\n```\r\nI had to add ``` corMemory=18``` to the canu command to pass through this meryl step. This changes --mem-per-cpu=2432 to  --mem-per-cpu=3840 in meryl.jobSubmit.sh. Here is top of slurm.out file\r\n```\r\n##   Mem/CPU : 3840 MB\r\n##  Mem/Node : 16384 MB\r\n```\r\n\r\n\r\n2. 2-correction in correction phase\r\nThis has failed on several different attempts (with different read mixtures).\r\nExample: part of header in correctRead._1.out:\r\n```\r\n##   Mem/CPU : 2304 MB\r\n##  Mem/Node : 16384 MB\r\n```\r\nI added corMemory=18 to the canu command.\r\nIt still failed - so I need more.\r\nExample slurm.out - piece of header:\r\n```\r\n##  Nodelist : node403\r\n##      CPUs : \r\n##   Mem/CPU : 4608 MB\r\n##  Mem/Node : 16384 MB\r\n.\r\n.\r\nRunning job 1 based on SLURM_ARRAY_TASK_ID=1 and offset=0.\r\nslurmstepd: Job 13020988 exceeded memory limit (18879048 > 18874368), being killed\r\nslurmstepd: Exceeded job memory limit\r\nslurmstepd: *** JOB 13020988 ON node403 CANCELLED AT 2016-07-29T05:49:25 ***\r\nslurmstepd: Exceeded step memory limit at some point.\r\n```\r\nIncreasing corMemory=24 worked for some, not others.\r\nIts now running at corMemory=30.\r\n\r\nI will let you know any other problematic areas of the pipeline as the runs progress.\r\n\r\nbest,\r\n\r\n  John\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/marbl/canu/issues/comments/236974090","html_url":"https://github.com/marbl/canu/issues/211#issuecomment-236974090","issue_url":"https://api.github.com/repos/marbl/canu/issues/211","id":236974090,"user":{"login":"brianwalenz","id":8249720,"avatar_url":"https://avatars.githubusercontent.com/u/8249720?v=3","gravatar_id":"","url":"https://api.github.com/users/brianwalenz","html_url":"https://github.com/brianwalenz","followers_url":"https://api.github.com/users/brianwalenz/followers","following_url":"https://api.github.com/users/brianwalenz/following{/other_user}","gists_url":"https://api.github.com/users/brianwalenz/gists{/gist_id}","starred_url":"https://api.github.com/users/brianwalenz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brianwalenz/subscriptions","organizations_url":"https://api.github.com/users/brianwalenz/orgs","repos_url":"https://api.github.com/users/brianwalenz/repos","events_url":"https://api.github.com/users/brianwalenz/events{/privacy}","received_events_url":"https://api.github.com/users/brianwalenz/received_events","type":"User","site_admin":false},"created_at":"2016-08-02T17:13:29Z","updated_at":"2016-08-02T17:13:29Z","body":"Is this failing when running meryl (the binary) or gnuplot?  I.e., what is in the 0-mercounts directory?  Serge has seem a few cases where gnuplot was the problem."}}
{"actor":{"display_login":"brianwalenz"}}