{"action":"created","issue":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1276","repository_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen","labels_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1276/labels{/name}","comments_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1276/comments","events_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1276/events","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/1276","id":141360837,"number":1276,"title":"pbspro controller start fails","user":{"login":"razZ0r","id":205865,"avatar_url":"https://avatars.githubusercontent.com/u/205865?v=3","gravatar_id":"","url":"https://api.github.com/users/razZ0r","html_url":"https://github.com/razZ0r","followers_url":"https://api.github.com/users/razZ0r/followers","following_url":"https://api.github.com/users/razZ0r/following{/other_user}","gists_url":"https://api.github.com/users/razZ0r/gists{/gist_id}","starred_url":"https://api.github.com/users/razZ0r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/razZ0r/subscriptions","organizations_url":"https://api.github.com/users/razZ0r/orgs","repos_url":"https://api.github.com/users/razZ0r/repos","events_url":"https://api.github.com/users/razZ0r/events{/privacy}","received_events_url":"https://api.github.com/users/razZ0r/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":33,"created_at":"2016-03-16T18:27:09Z","updated_at":"2016-03-23T11:06:32Z","closed_at":null,"body":"I'm trying to run bcbio on a cluster with PBSPro. Among many problems I have, the first is that the controller script fails.\r\n\r\nStartup command is:\r\n\r\n<pre>\r\n/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/tool/bin/bcbio_nextgen.py ../config/single-pair-cineca-test-resources.yaml ../config/single-pair-cineca-test-interactive-samples.yaml -t ipython -s pbspro -q parallel -n 16 -r account=try16_EMUTPRED\r\n</pre>\r\n\r\nI have some output, etc and then the command is stuck, and 0 engines are running.\r\n\r\nIn the work/log/ipython/log dir, the log says:\r\n\r\n<pre>\r\n2016-03-16 19:18:03.953 [IPClusterStart] Starting ipcluster with [daemon=True]\r\n2016-03-16 19:18:03.959 [IPClusterStart] Creating pid file: /pico/scratch/userexternal/esebesty/bcbio_run/single-pair-cineca-interactive/work/log/ipython/pid/ipcluster-e71c8d9d-02d5-42f5-ae0c-a1290b5bc5b0.pid\r\n2016-03-16 19:18:03.960 [IPClusterStart] Starting Controller with cluster_helper.cluster.BcbioPBSPROControllerLauncher\r\n2016-03-16 19:18:03.960 [IPClusterStart] Starting BcbioPBSPROControllerLauncher: ['qsub', u'./pbspro_controller']\r\n2016-03-16 19:18:03.960 [IPClusterStart] adding PBS queue settings to batch script\r\n2016-03-16 19:18:03.960 [IPClusterStart] adding job array settings to batch script\r\n2016-03-16 19:18:03.961 [IPClusterStart] Writing batch script: ./pbspro_controller\r\n2016-03-16 19:18:03.968 [IPClusterStart] ERROR | Controller start failed\r\nTraceback (most recent call last):\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/site-packages/ipyparallel/apps/ipclusterapp.py\", line 506, in start_controller\r\n    self.controller_launcher.start()\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/site-packages/cluster_helper/cluster.py\", line 718, in start\r\n    return super(BcbioPBSPROControllerLauncher, self).start(1)\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/site-packages/ipyparallel/apps/launcher.py\", line 1151, in start\r\n    output = check_output(self.args, env=os.environ)\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/subprocess.py\", line 566, in check_output\r\n    process = Popen(stdout=PIPE, *popenargs, **kwargs)\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/subprocess.py\", line 710, in __init__\r\n    errread, errwrite)\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/subprocess.py\", line 1335, in _execute_child\r\n    raise child_exception\r\nOSError: [Errno 2] No such file or directory\r\n</pre>\r\n\r\nThe pbspro_controller script is there, so I'm wondering what might be the source of the \"No such file or directory\" error.\r\n\r\n<pre>\r\n#!/bin/sh\r\n\r\n#PBS -q parallel\r\n#PBS -V\r\n#PBS -N bcbio-c\r\n#PBS -l select=1:ncpus=1\r\ncd $PBS_O_WORKDIR\r\n/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/bin/python -E -c 'import resource; cur_proc, max_proc = resource.getrlimit(resource.RLIMIT_NPROC); target_proc = min(max_proc, 10240) if max_proc > 0 else 10240; resource.setrlimit(resource.RLIMIT_NPROC, (max(cur_proc, target_proc), max_proc)); cur_hdls, max_hdls = resource.getrlimit(resource.RLIMIT_NOFILE); target_hdls = min(max_hdls, 10240) if max_hdls > 0 else 10240; resource.setrlimit(resource.RLIMIT_NOFILE, (max(cur_hdls, target_hdls), max_hdls)); from cluster_helper.cluster import VMFixIPControllerApp; VMFixIPControllerApp.launch_instance()' --ip=* --log-to-file --profile-dir=\"/pico/scratch/userexternal/esebesty/bcbio_run/single-pair-cineca-interactive/work/log/ipython\" --cluster-id=\"e71c8d9d-02d5-42f5-ae0c-a1290b5bc5b0\" --nodb --hwm=1 --scheme=leastload --HeartMonitor.max_heartmonitor_misses=720 --HeartMonitor.period=5000\r\n</pre>"},"comment":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/comments/200298167","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/1276#issuecomment-200298167","issue_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1276","id":200298167,"user":{"login":"razZ0r","id":205865,"avatar_url":"https://avatars.githubusercontent.com/u/205865?v=3","gravatar_id":"","url":"https://api.github.com/users/razZ0r","html_url":"https://github.com/razZ0r","followers_url":"https://api.github.com/users/razZ0r/followers","following_url":"https://api.github.com/users/razZ0r/following{/other_user}","gists_url":"https://api.github.com/users/razZ0r/gists{/gist_id}","starred_url":"https://api.github.com/users/razZ0r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/razZ0r/subscriptions","organizations_url":"https://api.github.com/users/razZ0r/orgs","repos_url":"https://api.github.com/users/razZ0r/repos","events_url":"https://api.github.com/users/razZ0r/events{/privacy}","received_events_url":"https://api.github.com/users/razZ0r/received_events","type":"User","site_admin":false},"created_at":"2016-03-23T11:06:32Z","updated_at":"2016-03-23T11:06:32Z","body":"Hi, it turned out that this is not a local setting at the cluster I'm using, but a PBSPro restriction. The PBSPro [manual](http://www.hpc.cineca.it/sites/default/files/PBSProUserGuide13.0beta.pdf) says:\r\n\r\n> Job arrays are required to be rerunnable. PBS will not accept a job array that is marked as\r\nnot rerunnable. You can submit a job array without specifying whether it is rerunnable,\r\nand PBS will automatically mark it as rerunnable.\r\n\r\nNo idea why this is the case. So there is not a way to support bcbio with PBSPro, unless there is some mechanism for handling rerunnable array jobs. :(\r\n\r\nI'll probably write some script that generates a qsub job for my sample pairs, where I'm running bcbio with `-t local -n 8/16/whatever` options."}}