{"action":"created","issue":{"url":"https://api.github.com/repos/merenlab/anvio/issues/961","repository_url":"https://api.github.com/repos/merenlab/anvio","labels_url":"https://api.github.com/repos/merenlab/anvio/issues/961/labels{/name}","comments_url":"https://api.github.com/repos/merenlab/anvio/issues/961/comments","events_url":"https://api.github.com/repos/merenlab/anvio/issues/961/events","html_url":"https://github.com/merenlab/anvio/issues/961","id":357280719,"node_id":"MDU6SXNzdWUzNTcyODA3MTk=","number":961,"title":"anvi-profile performance","user":{"login":"arivers","id":368469,"node_id":"MDQ6VXNlcjM2ODQ2OQ==","avatar_url":"https://avatars0.githubusercontent.com/u/368469?v=4","gravatar_id":"","url":"https://api.github.com/users/arivers","html_url":"https://github.com/arivers","followers_url":"https://api.github.com/users/arivers/followers","following_url":"https://api.github.com/users/arivers/following{/other_user}","gists_url":"https://api.github.com/users/arivers/gists{/gist_id}","starred_url":"https://api.github.com/users/arivers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arivers/subscriptions","organizations_url":"https://api.github.com/users/arivers/orgs","repos_url":"https://api.github.com/users/arivers/repos","events_url":"https://api.github.com/users/arivers/events{/privacy}","received_events_url":"https://api.github.com/users/arivers/received_events","type":"User","site_admin":false},"labels":[{"id":488265072,"node_id":"MDU6TGFiZWw0ODgyNjUwNzI=","url":"https://api.github.com/repos/merenlab/anvio/labels/metagenomic%20workflow","name":"metagenomic workflow","color":"e99695","default":false},{"id":204498768,"node_id":"MDU6TGFiZWwyMDQ0OTg3Njg=","url":"https://api.github.com/repos/merenlab/anvio/labels/performance","name":"performance","color":"f7c6c7","default":false},{"id":797570819,"node_id":"MDU6TGFiZWw3OTc1NzA4MTk=","url":"https://api.github.com/repos/merenlab/anvio/labels/profile%20database","name":"profile database","color":"e26fe8","default":false}],"state":"open","locked":false,"assignee":{"login":"ozcan","id":940884,"node_id":"MDQ6VXNlcjk0MDg4NA==","avatar_url":"https://avatars3.githubusercontent.com/u/940884?v=4","gravatar_id":"","url":"https://api.github.com/users/ozcan","html_url":"https://github.com/ozcan","followers_url":"https://api.github.com/users/ozcan/followers","following_url":"https://api.github.com/users/ozcan/following{/other_user}","gists_url":"https://api.github.com/users/ozcan/gists{/gist_id}","starred_url":"https://api.github.com/users/ozcan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ozcan/subscriptions","organizations_url":"https://api.github.com/users/ozcan/orgs","repos_url":"https://api.github.com/users/ozcan/repos","events_url":"https://api.github.com/users/ozcan/events{/privacy}","received_events_url":"https://api.github.com/users/ozcan/received_events","type":"User","site_admin":false},"assignees":[{"login":"ozcan","id":940884,"node_id":"MDQ6VXNlcjk0MDg4NA==","avatar_url":"https://avatars3.githubusercontent.com/u/940884?v=4","gravatar_id":"","url":"https://api.github.com/users/ozcan","html_url":"https://github.com/ozcan","followers_url":"https://api.github.com/users/ozcan/followers","following_url":"https://api.github.com/users/ozcan/following{/other_user}","gists_url":"https://api.github.com/users/ozcan/gists{/gist_id}","starred_url":"https://api.github.com/users/ozcan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ozcan/subscriptions","organizations_url":"https://api.github.com/users/ozcan/orgs","repos_url":"https://api.github.com/users/ozcan/repos","events_url":"https://api.github.com/users/ozcan/events{/privacy}","received_events_url":"https://api.github.com/users/ozcan/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-09-05T15:21:43Z","updated_at":"2018-10-31T20:54:39Z","closed_at":null,"author_association":"NONE","body":"# Performance issue\r\nI am using `anvi-merge` to process sorted indexed bam files  and a contig database.  I have tested multiple samples but a typical sample looks like this:\r\n\r\n```\r\ninput_bam ..........................: /project/gbru_fy18_rice_methane/mapping/FRCS-D1-R1-238.bam\r\noutput_dir .........................: /project/gbru_fy18_rice_methane/anvio/profiles/FRCS_D1_R1_238\r\ntotal_reads_mapped .................: 6,061,299\r\nnum_contigs ........................: 2,218,523\r\nnum_contigs_after_M ................: 145,320\r\nnum_splits .........................: 147,532\r\ntotal_length .......................: 1,407,546,361\r\n```\r\n\r\nA typical run command is\r\n```\r\nanvi-profile \r\n  -i /project/gbru_fy18_rice_methane/mapping/FRCS-D1-R1-238.bam \\\r\n  -c /project/gbru_fy18_rice_methane/anvio/contigs.db \\\r\n  --output-dir /project/gbru_fy18_rice_methane/anvio/profiles/FRCS_D1_R1_238 \r\n  --sample-name FRCS_D1_R1_238 \\\r\n  --skip-SNV-profiling -T 2 \r\n  --write-buffer-size 200 \\\r\n  --min-contig-length 5000\r\n```\r\nWhen I monitor the run this uses about 96GB of memory.  I cannot add more threads or I run out of memory. I've tried with a default buffer size of 500 and speeds were similar.\r\n\r\nContigs are processed at a rate of only about 59/min.  Smaller Bams process a little faster but these rates are at the limit of what I can process in 48 hours on a single 40 core Node with 128GB of RAM and the experiment has 100 samples.\r\n\r\nThe performance seems much slower than I would expect for the parsing of a bam file.  I was thinking there may be an issue with an underling library, for example htsliib or samtools don't  have their C parser compiled properly and are defaulting to a slower parser.\r\n\r\n# Installation method\r\n```\r\nconda create -y --name anvio5 python=3.6\r\nconda install -y --name anvio5 -c bioconda -c conda-forge anvio=5.1 diamond bwa\r\n```\r\nThe conda environment yml file was exported and attached.\r\n\r\n# System information\r\n* OS CentOS Linux release 7.4.1708 (Core)\r\n* Linux kernel: 3.10.0-693.el7.x86_64\r\n* scheduler: SLURM\r\n* Architecture:          x86_64\r\n* CPU op-mode(s):        32-bit, 64-bit\r\n* Byte Order:            Little Endian\r\n* CPU(s):                40\r\n* On-line CPU(s) list:   0-39\r\n* Thread(s) per core:    2\r\n* Core(s) per socket:    10\r\n* Socket(s):             2\r\n* NUMA node(s):          2\r\n* Vendor ID:             GenuineIntel\r\n*CPU family:            6\r\n* Model:                 62\r\n* Model name:            Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz\r\n* Stepping:              4\r\n* CPU MHz:               1715.136\r\n* CPU max MHz:           3300.0000\r\n* CPU min MHz:           1200.0000\r\n* BogoMIPS:              5000.09\r\n* Virtualization:        VT-x\r\n* L1d cache:             32K\r\n* L1i cache:             32K\r\n* L2 cache:              256K\r\n* L3 cache:              25600K\r\n* NUMA node0 CPU(s):     0-9,20-29\r\n* NUMA node1 CPU(s):     10-19,30-39\r\n* Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts"},"comment":{"url":"https://api.github.com/repos/merenlab/anvio/issues/comments/434844078","html_url":"https://github.com/merenlab/anvio/issues/961#issuecomment-434844078","issue_url":"https://api.github.com/repos/merenlab/anvio/issues/961","id":434844078,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNDg0NDA3OA==","user":{"login":"meren","id":197307,"node_id":"MDQ6VXNlcjE5NzMwNw==","avatar_url":"https://avatars2.githubusercontent.com/u/197307?v=4","gravatar_id":"","url":"https://api.github.com/users/meren","html_url":"https://github.com/meren","followers_url":"https://api.github.com/users/meren/followers","following_url":"https://api.github.com/users/meren/following{/other_user}","gists_url":"https://api.github.com/users/meren/gists{/gist_id}","starred_url":"https://api.github.com/users/meren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/meren/subscriptions","organizations_url":"https://api.github.com/users/meren/orgs","repos_url":"https://api.github.com/users/meren/repos","events_url":"https://api.github.com/users/meren/events{/privacy}","received_events_url":"https://api.github.com/users/meren/received_events","type":"User","site_admin":false},"created_at":"2018-10-31T20:54:39Z","updated_at":"2018-10-31T20:54:39Z","author_association":"MEMBER","body":"That is the right way to address that, @adriaaula."}}
{"actor":{"display_login":"meren"}}