{"action":"created","issue":{"url":"https://api.github.com/repos/PacificBiosciences/FALCON/issues/502","repository_url":"https://api.github.com/repos/PacificBiosciences/FALCON","labels_url":"https://api.github.com/repos/PacificBiosciences/FALCON/issues/502/labels{/name}","comments_url":"https://api.github.com/repos/PacificBiosciences/FALCON/issues/502/comments","events_url":"https://api.github.com/repos/PacificBiosciences/FALCON/issues/502/events","html_url":"https://github.com/PacificBiosciences/FALCON/issues/502","id":197057096,"number":502,"title":"Process getting stuck now at fc_consensus","user":{"login":"hodgett","id":24687753,"avatar_url":"https://avatars.githubusercontent.com/u/24687753?v=3","gravatar_id":"","url":"https://api.github.com/users/hodgett","html_url":"https://github.com/hodgett","followers_url":"https://api.github.com/users/hodgett/followers","following_url":"https://api.github.com/users/hodgett/following{/other_user}","gists_url":"https://api.github.com/users/hodgett/gists{/gist_id}","starred_url":"https://api.github.com/users/hodgett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hodgett/subscriptions","organizations_url":"https://api.github.com/users/hodgett/orgs","repos_url":"https://api.github.com/users/hodgett/repos","events_url":"https://api.github.com/users/hodgett/events{/privacy}","received_events_url":"https://api.github.com/users/hodgett/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2016-12-21T23:18:58Z","updated_at":"2016-12-23T19:20:30Z","closed_at":null,"body":"We have a large cluster that we are running FALCON on (250+ nodes) and now keep running in to new problems, some due to the configuration of the carious clusters we have tried using, some due to the software. Two weeks ago we ran some mid-size samples okay and so scaled up, but now no run goes to conclusion.\r\n\r\nLast week we were getting the EXIT 256 problem during the first daligner process and so, after reading the issues register, I upgraded to the current version. Now it won't get past fc_consensus. I find that fc_consensus jobs nearly always hang on the node with 0 CPU, usually with no stderr or stdout logs, sometimes stdout says it spawned the workers for fc_consensus and that is all. Some times I get an EXIT 99 and the following is all that is in stderr;\r\n```\r\n+ python2.7 -m pwatcher.mains.fs_heartbeat --directory=/lustre/work-lustre/banana_team/Falcon_Test-20161220c/0-rawreads/preads/cns_00006 --heartbeat-file=/lustre/work-lustre/banana_team/Falcon_Test-20161220c/mypwatcher/heartbeats/heartbeat-P1d481d812852fa --exit-file=/lustre/work-lustre/banana_team/Falcon_Test-20161220c/mypwatcher/exits/exit-P1d481d812852fa --rate=10.0 /bin/bash run.sh\r\n/var/spool/PBS/mom_priv/jobs/1378550.pbs.SC: line 6: 11445 Illegal instruction     python2.7 -m pwatcher.mains.fs_heartbeat --directory=/lustre/work-lustre/banana_team/Falcon_Test-20161220c/0-rawreads/preads/cns_00006 --heartbeat-file=/lustre/work-lustre/banana_team/Falcon_Test-20161220c/mypwatcher/heartbeats/heartbeat-P1d481d812852fa --exit-file=/lustre/work-lustre/banana_team/Falcon_Test-20161220c/mypwatcher/exits/exit-P1d481d812852fa --rate=10.0 /bin/bash run.sh\r\n+ echo 99\r\n```\r\nOur system runs SUSE12 and PBS Pro, python 2.7.11. \r\n\r\nI have attached the fc_run and pbs script in configs.zip, and a successful fc_consensus job that has the environment variables in Pd7d3d71c1e5923.zip.\r\n\r\nI would appreciate some guidance as to what could be wrong, and how we could figure out what is going wrong. Let me know if I am missing some information.\r\n\r\n[configs.zip](https://github.com/PacificBiosciences/FALCON/files/667564/configs.zip)\r\n[Pd7d3d71c1e5923.zip](https://github.com/PacificBiosciences/FALCON/files/667585/Pd7d3d71c1e5923.zip)\r\n"},"comment":{"url":"https://api.github.com/repos/PacificBiosciences/FALCON/issues/comments/269034300","html_url":"https://github.com/PacificBiosciences/FALCON/issues/502#issuecomment-269034300","issue_url":"https://api.github.com/repos/PacificBiosciences/FALCON/issues/502","id":269034300,"user":{"login":"msc51","id":23461010,"avatar_url":"https://avatars.githubusercontent.com/u/23461010?v=3","gravatar_id":"","url":"https://api.github.com/users/msc51","html_url":"https://github.com/msc51","followers_url":"https://api.github.com/users/msc51/followers","following_url":"https://api.github.com/users/msc51/following{/other_user}","gists_url":"https://api.github.com/users/msc51/gists{/gist_id}","starred_url":"https://api.github.com/users/msc51/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msc51/subscriptions","organizations_url":"https://api.github.com/users/msc51/orgs","repos_url":"https://api.github.com/users/msc51/repos","events_url":"https://api.github.com/users/msc51/events{/privacy}","received_events_url":"https://api.github.com/users/msc51/received_events","type":"User","site_admin":false},"created_at":"2016-12-23T19:20:30Z","updated_at":"2016-12-23T19:20:30Z","body":"Should I interrupt my current Falcon run, change the config file (--n_core 64 to 8), remove mypwatcher and then re-run it?? "}}
{"actor":{"display_login":"msc51"}}