{"action":"created","issue":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1276","repository_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen","labels_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1276/labels{/name}","comments_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1276/comments","events_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1276/events","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/1276","id":141360837,"number":1276,"title":"pbspro controller start fails","user":{"login":"razZ0r","id":205865,"avatar_url":"https://avatars.githubusercontent.com/u/205865?v=3","gravatar_id":"","url":"https://api.github.com/users/razZ0r","html_url":"https://github.com/razZ0r","followers_url":"https://api.github.com/users/razZ0r/followers","following_url":"https://api.github.com/users/razZ0r/following{/other_user}","gists_url":"https://api.github.com/users/razZ0r/gists{/gist_id}","starred_url":"https://api.github.com/users/razZ0r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/razZ0r/subscriptions","organizations_url":"https://api.github.com/users/razZ0r/orgs","repos_url":"https://api.github.com/users/razZ0r/repos","events_url":"https://api.github.com/users/razZ0r/events{/privacy}","received_events_url":"https://api.github.com/users/razZ0r/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":28,"created_at":"2016-03-16T18:27:09Z","updated_at":"2016-03-18T10:30:45Z","closed_at":null,"body":"I'm trying to run bcbio on a cluster with PBSPro. Among many problems I have, the first is that the controller script fails.\r\n\r\nStartup command is:\r\n\r\n<pre>\r\n/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/tool/bin/bcbio_nextgen.py ../config/single-pair-cineca-test-resources.yaml ../config/single-pair-cineca-test-interactive-samples.yaml -t ipython -s pbspro -q parallel -n 16 -r account=try16_EMUTPRED\r\n</pre>\r\n\r\nI have some output, etc and then the command is stuck, and 0 engines are running.\r\n\r\nIn the work/log/ipython/log dir, the log says:\r\n\r\n<pre>\r\n2016-03-16 19:18:03.953 [IPClusterStart] Starting ipcluster with [daemon=True]\r\n2016-03-16 19:18:03.959 [IPClusterStart] Creating pid file: /pico/scratch/userexternal/esebesty/bcbio_run/single-pair-cineca-interactive/work/log/ipython/pid/ipcluster-e71c8d9d-02d5-42f5-ae0c-a1290b5bc5b0.pid\r\n2016-03-16 19:18:03.960 [IPClusterStart] Starting Controller with cluster_helper.cluster.BcbioPBSPROControllerLauncher\r\n2016-03-16 19:18:03.960 [IPClusterStart] Starting BcbioPBSPROControllerLauncher: ['qsub', u'./pbspro_controller']\r\n2016-03-16 19:18:03.960 [IPClusterStart] adding PBS queue settings to batch script\r\n2016-03-16 19:18:03.960 [IPClusterStart] adding job array settings to batch script\r\n2016-03-16 19:18:03.961 [IPClusterStart] Writing batch script: ./pbspro_controller\r\n2016-03-16 19:18:03.968 [IPClusterStart] ERROR | Controller start failed\r\nTraceback (most recent call last):\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/site-packages/ipyparallel/apps/ipclusterapp.py\", line 506, in start_controller\r\n    self.controller_launcher.start()\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/site-packages/cluster_helper/cluster.py\", line 718, in start\r\n    return super(BcbioPBSPROControllerLauncher, self).start(1)\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/site-packages/ipyparallel/apps/launcher.py\", line 1151, in start\r\n    output = check_output(self.args, env=os.environ)\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/subprocess.py\", line 566, in check_output\r\n    process = Popen(stdout=PIPE, *popenargs, **kwargs)\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/subprocess.py\", line 710, in __init__\r\n    errread, errwrite)\r\n  File \"/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/lib/python2.7/subprocess.py\", line 1335, in _execute_child\r\n    raise child_exception\r\nOSError: [Errno 2] No such file or directory\r\n</pre>\r\n\r\nThe pbspro_controller script is there, so I'm wondering what might be the source of the \"No such file or directory\" error.\r\n\r\n<pre>\r\n#!/bin/sh\r\n\r\n#PBS -q parallel\r\n#PBS -V\r\n#PBS -N bcbio-c\r\n#PBS -l select=1:ncpus=1\r\ncd $PBS_O_WORKDIR\r\n/pico/home/userexternal/esebesty/work/bcbio-20160314/bcbio/data/anaconda/bin/python -E -c 'import resource; cur_proc, max_proc = resource.getrlimit(resource.RLIMIT_NPROC); target_proc = min(max_proc, 10240) if max_proc > 0 else 10240; resource.setrlimit(resource.RLIMIT_NPROC, (max(cur_proc, target_proc), max_proc)); cur_hdls, max_hdls = resource.getrlimit(resource.RLIMIT_NOFILE); target_hdls = min(max_hdls, 10240) if max_hdls > 0 else 10240; resource.setrlimit(resource.RLIMIT_NOFILE, (max(cur_hdls, target_hdls), max_hdls)); from cluster_helper.cluster import VMFixIPControllerApp; VMFixIPControllerApp.launch_instance()' --ip=* --log-to-file --profile-dir=\"/pico/scratch/userexternal/esebesty/bcbio_run/single-pair-cineca-interactive/work/log/ipython\" --cluster-id=\"e71c8d9d-02d5-42f5-ae0c-a1290b5bc5b0\" --nodb --hwm=1 --scheme=leastload --HeartMonitor.max_heartmonitor_misses=720 --HeartMonitor.period=5000\r\n</pre>"},"comment":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/comments/198297985","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/1276#issuecomment-198297985","issue_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1276","id":198297985,"user":{"login":"chapmanb","id":39391,"avatar_url":"https://avatars.githubusercontent.com/u/39391?v=3","gravatar_id":"","url":"https://api.github.com/users/chapmanb","html_url":"https://github.com/chapmanb","followers_url":"https://api.github.com/users/chapmanb/followers","following_url":"https://api.github.com/users/chapmanb/following{/other_user}","gists_url":"https://api.github.com/users/chapmanb/gists{/gist_id}","starred_url":"https://api.github.com/users/chapmanb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chapmanb/subscriptions","organizations_url":"https://api.github.com/users/chapmanb/orgs","repos_url":"https://api.github.com/users/chapmanb/repos","events_url":"https://api.github.com/users/chapmanb/events{/privacy}","received_events_url":"https://api.github.com/users/chapmanb/received_events","type":"User","site_admin":false},"created_at":"2016-03-18T10:30:45Z","updated_at":"2016-03-18T10:30:45Z","body":"Endre;\r\nThanks for all the patience debugging this. All of our support for PBSPro has been via back and forth like this since we don't run those systems locally so appreciate the help. Regarding you last three posts:\r\n\r\n- I don't believe the non-rerunable flag is important. That might be leftover from someone's suggestion but Rory would know for sure.\r\n- Did you also start the controller for that engine job that died? It looks like it's complaining because it can't find the directory that contains details about connecting to the controller. If you're only testing submission, that's expected since there is no controller.\r\n- Not using job arrays would be a large change in how we submit and manage jobs. Right now we let the queuing software take care of managing multiple engines, so would have to re-write all that logic in bcbio. What concern do you have about allowing job arrays?\r\n\r\nHope this helps."}}