{"action":"created","issue":{"url":"https://api.github.com/repos/psathyrella/partis/issues/190","repository_url":"https://api.github.com/repos/psathyrella/partis","labels_url":"https://api.github.com/repos/psathyrella/partis/issues/190/labels{/name}","comments_url":"https://api.github.com/repos/psathyrella/partis/issues/190/comments","events_url":"https://api.github.com/repos/psathyrella/partis/issues/190/events","html_url":"https://github.com/psathyrella/partis/issues/190","id":157012301,"number":190,"title":"Parallelization on other queueing systems","user":{"login":"krdav","id":17021827,"avatar_url":"https://avatars.githubusercontent.com/u/17021827?v=3","gravatar_id":"","url":"https://api.github.com/users/krdav","html_url":"https://github.com/krdav","followers_url":"https://api.github.com/users/krdav/followers","following_url":"https://api.github.com/users/krdav/following{/other_user}","gists_url":"https://api.github.com/users/krdav/gists{/gist_id}","starred_url":"https://api.github.com/users/krdav/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krdav/subscriptions","organizations_url":"https://api.github.com/users/krdav/orgs","repos_url":"https://api.github.com/users/krdav/repos","events_url":"https://api.github.com/users/krdav/events{/privacy}","received_events_url":"https://api.github.com/users/krdav/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2016-05-26T15:32:05Z","updated_at":"2016-10-28T07:39:12Z","closed_at":null,"body":"As I know there are a lot of bioinformaticians (including myself) that are using compute clusters running the PBS scheduler like TORQUE. So it is a shame that partis does not support this using this. I know that slurm is suggested but I don't have access to an HPC running slurm.\r\n\r\nThe case is that I am running into slow runtimes when running large files, like this run I just did yesterday:\r\n500,000 seqs\r\n28 cpus with total time of 17:41:00 and 78.82% cpu efficiency.\r\n\r\nI have been thinking about how to implement it myself but haven't gotten so far yet. Any suggestions/starting point would be welcomed."},"comment":{"url":"https://api.github.com/repos/psathyrella/partis/issues/comments/256856609","html_url":"https://github.com/psathyrella/partis/issues/190#issuecomment-256856609","issue_url":"https://api.github.com/repos/psathyrella/partis/issues/190","id":256856609,"user":{"login":"krdav","id":17021827,"avatar_url":"https://avatars.githubusercontent.com/u/17021827?v=3","gravatar_id":"","url":"https://api.github.com/users/krdav","html_url":"https://github.com/krdav","followers_url":"https://api.github.com/users/krdav/followers","following_url":"https://api.github.com/users/krdav/following{/other_user}","gists_url":"https://api.github.com/users/krdav/gists{/gist_id}","starred_url":"https://api.github.com/users/krdav/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krdav/subscriptions","organizations_url":"https://api.github.com/users/krdav/orgs","repos_url":"https://api.github.com/users/krdav/repos","events_url":"https://api.github.com/users/krdav/events{/privacy}","received_events_url":"https://api.github.com/users/krdav/received_events","type":"User","site_admin":false},"created_at":"2016-10-28T07:39:12Z","updated_at":"2016-10-28T07:39:12Z","body":"As Chaim suggests Docker is not playing nice with SGE/PBS at this point \nand as Duncan experienced it is extremely easy to compile partis \nyourself. I was myself testing out the docker image when starting to use \npartis but quickly found that it takes very little to compile it \nyourself so this is what I have been doing ever since.\n\nWhat Chaim says about 1 cpu per task on the SGE system also applies for \nPBS but I think it is not uncommon to enable reservation on an entire \nmachine/node. E.g. if you submit to a system composed of multiple 8 core \nmachines then if you submit a job with a reservation on 8 cpu's the job \nwill simply be pushed to reserve a full node and then you can also get \nthe full benefit of the physical memory of the given node. For me this \nmeans I can either submit to a 28 cores/128Gb memory or a 32 \ncores/1024Gb memory node and therefore runtime has never really been a \nproblem for me when it comes to caching. My problem when starting this \ndiscussion was that the partitioning was running very slow for me and \nknow with the tremendous improvements in this procedure its not a \nproblem anymore. Right now I can submit caching/partitioning + my own \nhacks and have it finishing on 8 heavy and 8 light chain samples in less \nthan an hour.\n\nIn conclusion I think that SGE/PBS parallelization is getting less \nimportant now with the runtime improvements done on the base code.\n\n\nOn 28/10/2016 00.44, scharch wrote:\n> Near as I can tell, docker-sge is for running sge from within docker, not\n> running docker on an sge cluster. Sounds great for creating your own\n> cluster with aws, though.\n>\n> On Oct 27, 2016 6:30 PM, \"Duncan Ralph\" <notifications@github.com> wrote:\n>\n> > The docker image <https://github.com/gawbul/docker-sge> I found uses the\n> > -f option, e.g.\n> >\n> > docker run -it --rm gawbul/docker-sge login -f sgeadmin\n> >\n> > not sure if that works in practice though. I actually installed \n> partis on\n> > a new laptop running ubuntu 16.04, and was shocked how easy it was, so I\n> > added a comment here\n> > \n> <https://github.com/psathyrella/partis/blob/dev/manual.md#installation-from-scratch>\n> > with some tips. It just required a couple extra apt-get and pip installs\n> > Honestly for a lot of purposes I think it's just easier to not use \n> docker\n> > at this point, depending on your os of course.\n> >\n> > ok, thanks. I added\n> > \n> <https://github.com/psathyrella/partis/commit/6abf905513dbc0ac4aee65671a1d2961d692b835>\n> > the -e and -o options.\n> >\n> > Well, maybe it's ok how it is then. You can tweak anything you need to\n> > with --batch-options. The important thing is when you're running vsearch\n> > partitioning that you can use more than one thread, which you can -- \n> it's\n> > also good to notify the batch system that you're doing that, but \n> that's a\n> > secondary goal (as long as I'm not your sysadmin...).\n> >\n> > —\n> > You are receiving this because you were mentioned.\n> > Reply to this email directly, view it on GitHub\n> > \n> <https://github.com/psathyrella/partis/issues/190#issuecomment-256788190>,\n> > or mute the thread\n> > \n> <https://github.com/notifications/unsubscribe-auth/AGZe4MJMps3ObkShafVaeYbtFzPwznBsks5q4SXsgaJpZM4InqnO>\n> > .\n> >\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub \n> <https://github.com/psathyrella/partis/issues/190#issuecomment-256790782>, \n> or mute the thread \n> <https://github.com/notifications/unsubscribe-auth/AQO7g0vEFZCj_xrScfC5eFjcG_I1h-8pks5q4Sk_gaJpZM4InqnO>.\n>\n\n"}}
{"actor":{"display_login":"krdav"}}