{"action":"opened","issue":{"url":"https://api.github.com/repos/marbl/canu/issues/56","repository_url":"https://api.github.com/repos/marbl/canu","labels_url":"https://api.github.com/repos/marbl/canu/issues/56/labels{/name}","comments_url":"https://api.github.com/repos/marbl/canu/issues/56/comments","events_url":"https://api.github.com/repos/marbl/canu/issues/56/events","html_url":"https://github.com/marbl/canu/issues/56","id":137217281,"number":56,"title":"mhap correction time of large genomes","user":{"login":"aliecs","id":9980279,"avatar_url":"https://avatars.githubusercontent.com/u/9980279?v=3","gravatar_id":"","url":"https://api.github.com/users/aliecs","html_url":"https://github.com/aliecs","followers_url":"https://api.github.com/users/aliecs/followers","following_url":"https://api.github.com/users/aliecs/following{/other_user}","gists_url":"https://api.github.com/users/aliecs/gists{/gist_id}","starred_url":"https://api.github.com/users/aliecs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aliecs/subscriptions","organizations_url":"https://api.github.com/users/aliecs/orgs","repos_url":"https://api.github.com/users/aliecs/repos","events_url":"https://api.github.com/users/aliecs/events{/privacy}","received_events_url":"https://api.github.com/users/aliecs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-02-29T10:48:46Z","updated_at":"2016-02-29T10:48:46Z","closed_at":null,"body":"Hi everyone,\r\n\r\nI'm assembling a large genome of 1.5Gb with canu. While the assembly part works fine if I feed it corrected data, the -correct portion is slow, slow, slow. \r\nI want to run in on a sigle machine of 1TB with 22 cores. My question is to know how to best optimise the memory, threads and concurrency. The data is a coverage of ~55x, so a ~150Gb fastq file of raw pacbio. \r\n\r\nSo far I have tried:\r\n1) - high memory: This results in over 150 precompute jobs and over 300 mhap jobs\r\n\r\n2) - high memory: corMemory=900, cormhapMemory=900\r\n- max threads: corThreads=22, cormhapThreads=22\r\n- low concurrency: corConcurrency=4, cormhapConcurrency=4\r\n\r\n3) Same as previous but with concurrencies at 16:\r\nBoth result in a 6 precompute jobs that are relatively fast (~ 24h) and 10 mhap jobs that are too long. My mhap job looks like this, and is stuck at first 1/10 of the reads since 2 days:\r\n`\r\njava -server -Xmx900g -jar /software/canu/canu-1.0.local/Linux-amd64/bin/mhap-1.5b1.jar --weighted -k 16 --num-hashes 512 --num-min-matches 3 --threshold 0.04 --filter-threshold 0.000005 --min-store-length 10611 --num-threads 22 -f /project/project1/abodrug/assembly_canu/refQ2-correction3/correction/0-mercounts/refQ2-raw.ms16.frequentMers.ignore -s /project/project1/abodrug/assembly_canu/refQ2-correction3/correction/1-overlapper/blocks/000001.dat -q /project/project1/abodrug/assembly_canu/refQ2-correction3/correction/1-overlapper/queries/000001`\r\n\r\nMy latest canu command looks like this:\r\n`/software/BIN/canu -correct -p refQ2-raw -d refQ2-correction3 genomeSize=1.45g -pacbio-raw /project/project1/abodrug/pacbio_template_filtered_nopal_size35.fastq useGrid=0 minThreads=22 maxThreads=22 corConcurrency=16 corMemory=900 corThreads=22 cormhapConcurrency=16 cormhapMemory=900 cormhapThreads=22 corovlConcurrency=16 corovlMemory=900 corovlThreads=22 merylThreads=22 merylMemory=900 merylConcurrency=1`\r\n\r\nIs there anything else I could do to speed up the precompute and mhap steps of canu on a single machine?\r\n\r\nWould using a grid engine be in any way faster if my settings are actually similar to the single machine ones (gridOptionscor, gridOptionscormhap, gridOptionscorovl)?\r\n\r\nThanks, \r\nAlex "}}