{"action":"closed","issue":{"url":"https://api.github.com/repos/Illumina/Isaac3/issues/7","repository_url":"https://api.github.com/repos/Illumina/Isaac3","labels_url":"https://api.github.com/repos/Illumina/Isaac3/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/Illumina/Isaac3/issues/7/comments","events_url":"https://api.github.com/repos/Illumina/Isaac3/issues/7/events","html_url":"https://github.com/Illumina/Isaac3/issues/7","id":177368155,"number":7,"title":"isaac: reduce I/O on himem machines","user":{"login":"sklages","id":1837422,"avatar_url":"https://avatars.githubusercontent.com/u/1837422?v=3","gravatar_id":"","url":"https://api.github.com/users/sklages","html_url":"https://github.com/sklages","followers_url":"https://api.github.com/users/sklages/followers","following_url":"https://api.github.com/users/sklages/following{/other_user}","gists_url":"https://api.github.com/users/sklages/gists{/gist_id}","starred_url":"https://api.github.com/users/sklages/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sklages/subscriptions","organizations_url":"https://api.github.com/users/sklages/orgs","repos_url":"https://api.github.com/users/sklages/repos","events_url":"https://api.github.com/users/sklages/events{/privacy}","received_events_url":"https://api.github.com/users/sklages/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-16T08:13:56Z","updated_at":"2016-09-16T09:05:03Z","closed_at":"2016-09-16T09:05:03Z","body":"Hi,\r\nI want to distribute different isaac (human/mouse) mappings on different servers all writing on the same (high performance) storage system during alignment. That is our computing setup.\r\nWe do have a lot of servers with more than 500G RAM.\r\n\r\nHow do I choose the concurrency parameters to produce as little I/O as possible? \r\nI do see that multiple jobs on multiple servers dramatically kill I/O performance on shared filesystem.\r\n\r\nI also see a lot of warnings like this in isaac's output:\r\n```\r\n2016-09-16 00:17:48  [7f866ebe5700] WARNING: Holding up processing of bin: BinMetadata(200id ReferencePosition(15:40562784:0f)bs 10140696bl 708874476ds 0do 0se 1277975rs 1250093f /path/to/temp/bin-00000016-00000200.dat) until std::bad_alloc clears. Error data: 982017348\r\n\r\n2016-09-16 00:19:39  [7f88621da700] WARNING: Holding up processing of bin: BinMetadata(217id ReferencePosition(16:60844176:0f)bs 10140696bl 796554434ds 0do 0se 1434240rs 1408124f /path/to/temp/bin-00000017-00000217.dat) until std::bad_alloc clears. Error data: 1103634210\r\n\r\n2016-09-16 00:25:44  [7f86703e8700] WARNING: Holding up processing of bin: /path/to/temp/bin-00000000-00000000.dat until a load slot is available\r\n\r\n2016-09-16 00:26:03  [7f86713ea700] WARNING: Holding up processing of bin: /path/to/temp/bin-00000000-00000000.dat until a load slot is available\r\n```\r\nand runtimes more than 10h (usually max. 2h).\r\n\r\nThis happened with:\r\n```\r\n--input-concurrent-load 10 \r\n--temp-concurrent-load 10 \r\n--output-concurrent-save 4 \r\n--temp-concurrent-save 4\r\n```\r\n\r\nI tried with different concurrency setting, with similar results.\r\n\r\nSo when I run isaac with one or two jobs, everything is fine. When I try to run 20 Jobs on 10 servers writing on the same filesystem, performance descreases dramatically.\r\nSo I'd like to opimise conncurrency parameters to possibly **decrease file I/O**.\r\n\r\nAny idea what to do? I could use `/dev/shm` but I really like to avoid that.\r\n\r\n"}}
{"actor":{"display_login":"sklages"}}