{"action":"created","issue":{"url":"https://api.github.com/repos/dereneaton/ipyrad/issues/55","labels_url":"https://api.github.com/repos/dereneaton/ipyrad/issues/55/labels{/name}","comments_url":"https://api.github.com/repos/dereneaton/ipyrad/issues/55/comments","events_url":"https://api.github.com/repos/dereneaton/ipyrad/issues/55/events","html_url":"https://github.com/dereneaton/ipyrad/issues/55","id":125531674,"number":55,"title":"demultiplexing is eating RAM","user":{"login":"dereneaton","id":5395480,"avatar_url":"https://avatars.githubusercontent.com/u/5395480?v=3","gravatar_id":"","url":"https://api.github.com/users/dereneaton","html_url":"https://github.com/dereneaton","followers_url":"https://api.github.com/users/dereneaton/followers","following_url":"https://api.github.com/users/dereneaton/following{/other_user}","gists_url":"https://api.github.com/users/dereneaton/gists{/gist_id}","starred_url":"https://api.github.com/users/dereneaton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dereneaton/subscriptions","organizations_url":"https://api.github.com/users/dereneaton/orgs","repos_url":"https://api.github.com/users/dereneaton/repos","events_url":"https://api.github.com/users/dereneaton/events{/privacy}","received_events_url":"https://api.github.com/users/dereneaton/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dereneaton/ipyrad/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":{"login":"dereneaton","id":5395480,"avatar_url":"https://avatars.githubusercontent.com/u/5395480?v=3","gravatar_id":"","url":"https://api.github.com/users/dereneaton","html_url":"https://github.com/dereneaton","followers_url":"https://api.github.com/users/dereneaton/followers","following_url":"https://api.github.com/users/dereneaton/following{/other_user}","gists_url":"https://api.github.com/users/dereneaton/gists{/gist_id}","starred_url":"https://api.github.com/users/dereneaton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dereneaton/subscriptions","organizations_url":"https://api.github.com/users/dereneaton/orgs","repos_url":"https://api.github.com/users/dereneaton/repos","events_url":"https://api.github.com/users/dereneaton/events{/privacy}","received_events_url":"https://api.github.com/users/dereneaton/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2016-01-08T02:53:30Z","updated_at":"2016-01-11T23:09:44Z","closed_at":null,"body":"Ran step1 on some real data files. Lots to optimize here. \r\n\r\n+ Not sure why, but it's maxing out my RAM in a bad way.\r\n+ Parallelizing gzip was running at super low %CPU. Is it already multithreaded by default? \r\n+ Getting low %CPU running during pure Python code as well."},"comment":{"url":"https://api.github.com/repos/dereneaton/ipyrad/issues/comments/170727565","html_url":"https://github.com/dereneaton/ipyrad/issues/55#issuecomment-170727565","issue_url":"https://api.github.com/repos/dereneaton/ipyrad/issues/55","id":170727565,"user":{"login":"dereneaton","id":5395480,"avatar_url":"https://avatars.githubusercontent.com/u/5395480?v=3","gravatar_id":"","url":"https://api.github.com/users/dereneaton","html_url":"https://github.com/dereneaton","followers_url":"https://api.github.com/users/dereneaton/followers","following_url":"https://api.github.com/users/dereneaton/following{/other_user}","gists_url":"https://api.github.com/users/dereneaton/gists{/gist_id}","starred_url":"https://api.github.com/users/dereneaton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dereneaton/subscriptions","organizations_url":"https://api.github.com/users/dereneaton/orgs","repos_url":"https://api.github.com/users/dereneaton/repos","events_url":"https://api.github.com/users/dereneaton/events{/privacy}","received_events_url":"https://api.github.com/users/dereneaton/received_events","type":"User","site_admin":false},"created_at":"2016-01-11T23:09:43Z","updated_at":"2016-01-11T23:09:43Z","body":"Pretty much all of my earlier ideas were really dumb. Can't parallelize splitting/chunking files into smaller bits or collating them back together, since these functions are limited by the speed at which data can be written to disk, not by processor speed. \r\n\r\nThese functions are now non-parallel. The best speed will come from optimizing splitting to create a number of files that matches the number of processors. And not splitting if there are already more files than processors. I'm not sure yet whether writing with gzip can be parallelized since it's probably spending some computation on compressing. Anyways, still working on optimizing this. \r\n\r\n "}}