{"action":"closed","issue":{"url":"https://api.github.com/repos/choderalab/yank/issues/367","repository_url":"https://api.github.com/repos/choderalab/yank","labels_url":"https://api.github.com/repos/choderalab/yank/issues/367/labels{/name}","comments_url":"https://api.github.com/repos/choderalab/yank/issues/367/comments","events_url":"https://api.github.com/repos/choderalab/yank/issues/367/events","html_url":"https://github.com/choderalab/yank/issues/367","id":150103813,"number":367,"title":"MPI errors when running on more than 8 GPUs","user":{"login":"steven-albanese","id":10441641,"avatar_url":"https://avatars.githubusercontent.com/u/10441641?v=3","gravatar_id":"","url":"https://api.github.com/users/steven-albanese","html_url":"https://github.com/steven-albanese","followers_url":"https://api.github.com/users/steven-albanese/followers","following_url":"https://api.github.com/users/steven-albanese/following{/other_user}","gists_url":"https://api.github.com/users/steven-albanese/gists{/gist_id}","starred_url":"https://api.github.com/users/steven-albanese/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steven-albanese/subscriptions","organizations_url":"https://api.github.com/users/steven-albanese/orgs","repos_url":"https://api.github.com/users/steven-albanese/repos","events_url":"https://api.github.com/users/steven-albanese/events{/privacy}","received_events_url":"https://api.github.com/users/steven-albanese/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2016-04-21T15:12:29Z","updated_at":"2016-04-21T20:35:07Z","closed_at":"2016-04-21T20:35:07Z","body":"Just wanted to raise this issue because I think @andrrizzi has seen it as well. I get the following error whenever I try to run a YANK simulation in MPI mode on more than 8 GPUS: \r\n\r\n```\r\ngpu-3-8-gpu3\r\ngpu-3-8-gpu2\r\ngpu-3-8-gpu1\r\ngpu-3-8-gpu0\r\ngpu-1-10-gpu3\r\ngpu-1-10-gpu2\r\ngpu-1-10-gpu1\r\ngpu-1-10-gpu0\r\ngpu-2-11-gpu3\r\ngpu-2-11-gpu2\r\ngpu-2-11-gpu1\r\ngpu-2-11-gpu0\r\ngpu-1-13-gpu3\r\ngpu-1-13-gpu2\r\ngpu-1-13-gpu1\r\ngpu-1-13-gpu0\r\ngpu-1-17-gpu3\r\ngpu-1-17-gpu2\r\ngpu-1-17-gpu1\r\ngpu-1-17-gpu0\r\nRunning simulation via MPI...\r\n[mpiexec@gpu-3-8.local] match_arg (./utils/args/args.c:122): unrecognized argument \r\n[mpiexec@gpu-3-8.local] HYDU_parse_array (./utils/args/args.c:140): argument matching returned error\r\n[mpiexec@gpu-3-8.local] parse_args (./ui/mpich/utils.c:1387): error parsing input array\r\n[mpiexec@gpu-3-8.local] HYD_uii_mpx_get_parameters (./ui/mpich/utils.c:1475): error parsing config args\r\n\r\nUsage: ./mpiexec [global opts] [exec1 local opts] : [exec2 local opts] : ...\r\n\r\nGlobal options (passed to all executables):\r\n\r\n  Global environment options:\r\n    -genv {name} {value}             environment variable name and value\r\n    -genvlist {env1,env2,...}        environment variable list to pass\r\n    -genvnone                        do not pass any environment variables\r\n    -genvall                         pass all environment variables not managed\r\n                                          by the launcher (default)\r\n\r\n  Other global options:\r\n    -f {name}                        file containing the host names\r\n    -hosts {host list}               comma separated host list\r\n    -wdir {dirname}                  working directory to use\r\n    -configfile {name}               config file containing MPMD launch options\r\n\r\n\r\nLocal options (passed to individual executables):\r\n\r\n  Local environment options:\r\n    -env {name} {value}              environment variable name and value\r\n    -envlist {env1,env2,...}         environment variable list to pass\r\n    -envnone                         do not pass any environment variables\r\n    -envall                          pass all environment variables (default)\r\n\r\n  Other local options:\r\n    -n/-np {value}                   number of processes\r\n    {exec_name} {args}               executable name and arguments\r\n\r\n\r\nHydra specific options (treated as global):\r\n\r\n  Launch options:\r\n    -launcher                        launcher to use ( ssh rsh fork slurm ll lsf sge manual persist)\r\n    -launcher-exec                   executable to use to launch processes\r\n    -enable-x/-disable-x             enable or disable X forwarding\r\n\r\n  Resource management kernel options:\r\n    -rmk                             resource management kernel to use ( user slurm ll lsf sge pbs)\r\n\r\n  Hybrid programming options:\r\n    -ranks-per-proc                  assign so many ranks to each process\r\n\r\n  Processor topology options:\r\n    -binding                         process-to-core binding mode\r\n    -topolib                         processor topology library ( hwloc plpa)\r\n\r\n  Checkpoint/Restart options:\r\n    -ckpoint-interval                checkpoint interval\r\n    -ckpoint-prefix                  checkpoint file prefix\r\n    -ckpoint-num                     checkpoint number to restart\r\n    -ckpointlib                      checkpointing library (none)\r\n\r\n  Demux engine options:\r\n    -demux                           demux engine ( poll select)\r\n\r\n  Other Hydra options:\r\n    -verbose                         verbose mode\r\n    -info                            build information\r\n    -print-all-exitcodes             print exit codes of all processes\r\n    -iface                           network interface to use\r\n    -ppn                             processes per node\r\n    -profile                         turn on internal profiling\r\n    -prepend-rank                    prepend rank to output\r\n    -prepend-pattern                 prepend pattern to output\r\n    -outfile-pattern                 direct stdout to file\r\n    -errfile-pattern                 direct stderr to file\r\n    -nameserver                      name server information (host:port format)\r\n    -disable-auto-cleanup            don't cleanup processes on error\r\n    -disable-hostname-propagation    let MPICH2 auto-detect the hostname\r\n    -order-nodes                     order nodes as ascending/descending cores\r\n\r\nPlease see the intructions provided at\r\nhttp://wiki.mcs.anl.gov/mpich2/index.php/Using_the_Hydra_Process_Manager\r\nfor further details\r\n\r\nThu Apr 21 11:03:00 EDT 2016\r\nSet compute mode to DEFAULT for GPU 0000:84:00.0.\r\nAll done.\r\nSet compute mode to DEFAULT for GPU 0000:83:00.0.\r\nAll done.\r\nSet compute mode to DEFAULT for GPU 0000:04:00.0.\r\nAll done.\r\nSet compute mode to DEFAULT for GPU 0000:03:00.0.\r\nAll done.\r\n\r\n``` "}}