{"action":"created","issue":{"url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/625","repository_url":"https://api.github.com/repos/raffenet/mpich-migrate-test","labels_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/625/labels{/name}","comments_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/625/comments","events_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/625/events","html_url":"https://github.com/raffenet/mpich-migrate-test/issues/625","id":168446837,"number":625,"title":"running more than one mpdboot","user":{"login":"raffenet","id":1410792,"avatar_url":"https://avatars.githubusercontent.com/u/1410792?v=3","gravatar_id":"","url":"https://api.github.com/users/raffenet","html_url":"https://github.com/raffenet","followers_url":"https://api.github.com/users/raffenet/followers","following_url":"https://api.github.com/users/raffenet/following{/other_user}","gists_url":"https://api.github.com/users/raffenet/gists{/gist_id}","starred_url":"https://api.github.com/users/raffenet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raffenet/subscriptions","organizations_url":"https://api.github.com/users/raffenet/orgs","repos_url":"https://api.github.com/users/raffenet/repos","events_url":"https://api.github.com/users/raffenet/events{/privacy}","received_events_url":"https://api.github.com/users/raffenet/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-07-30T03:37:37Z","updated_at":"2016-07-30T03:37:38Z","closed_at":null,"body":"**Originally by _William Garber <garberw@bnl.gov>_ on 2009-05-29 16:22:34 -0500**\n\n___\n\n\n```\n\nI am using torque and I submit a pbs/torque script:\n=======================\n#!/bin/sh\n\n#PBS -N\nuu\n\n#PBS\n-V\n\n#PBS -l\nnodes=5:ppn=1\n\n\nexport SCRATCH=\"/scratch/$LOGNAME/$PBS_JOBID\"\nexport SRC_DIR=$MPI_O_WORKDIR\n\nMPI_NODEFILE=$PBS_O_WORKDIR/mpd_nodes.${PBS_JOBID}\nsort $PBS_NODEFILE | uniq -c | awk '{ printf(\"%s:%s\\n\", $2, $1);}' >\n$MPI_NODEFILE\n\nNUMPROCESSES=`cat $PBS_NODEFILE | wc -l`\nNUMNODES=`cat $MPI_NODEFILE | wc -l`\nmpdboot -f $MPI_NODEFILE -n $NUMNODES --maxbranch=30\ncd $PBS_O_WORKDIR\nmpiexec -1 -machinefile $MPI_NODEFILE -np $NUMPROCESSES ./uu.sh >\nuu.out.${PBS_JOBID} 2> uu.err.${PBS_JOBID}\nmpdallexit\n=========================================================\nRepeated problem:\nI run two jobs each having one process per node.\nThe jobs share at least one node.\nThe second job fails with the error:\n================================================\nmpdboot_node005 (handle_mpd_output 385): failed to handshake with mpd on\nnode001; recvd output={}\n\nBegin Fri May 29 16:02:41 EDT 2009\nEnd Fri May 29 16:02:41 EDT 2009\nmpdallexit: cannot connect to local mpd (/tmp/mpd2.console_garberw);\npossible causes:\n  1. no mpd is running on this host\n  2. an mpd is running but was started without a \"console\" (-n option)\nIn case 1, you can start an mpd on this host with:\n    mpd &\nand you will be able to run jobs just on this host.\nFor more details on starting mpds on a set of hosts, see\nthe MPICH2 Installation Guide.\n============================================#### the /etc/hosts on the nodes is==========\n::1     localhost6.localdomain6 localhost6\n127.0.0.1       localhost.localdomain   localhost\n172.16.100.2    clio.                   clio            head\n172.16.100.1    morpheus.               morpheus\n172.16.0.1      node001.        node001         n001\n172.16.0.2      node002.        node002         n002\n172.16.0.3      node003.        node003         n003\n172.16.0.4      node004.        node004         n004\n172.16.0.5      node005.        node005         n005\n172.16.0.6      node006.        node006         n006\n172.16.0.7      node007.        node007         n007\n172.16.0.8      node008.        node008         n008\n172.16.0.9      node009.        node009         n009\netc\n==========#### the /etc/hosts on the head clio is==========\n::1     localhost6.localdomain6 localhost6\n127.0.0.1       localhost.localdomain   localhost\n172.16.0.1      node001.        node001         n001\n172.16.0.2      node002.        node002         n002\n172.16.0.3      node003.        node003         n003\n172.16.0.4      node004.        node004         n004\n172.16.0.5      node005.        node005         n005\n172.16.0.6      node006.        node006         n006\n172.16.0.7      node007.        node007         n007\n172.16.0.8      node008.        node008         n008\n172.16.0.9      node009.        node009         n009\n130.199.74.52   morpheus.phy.bnl.gov    morpheus\n172.16.100.2    clio.           clio            head\n130.199.74.51   clio.phy.bnl.gov\n==============\nIf it is not a problem with the /etc/hosts, then\nI read in the troubleshooting section of the installation\nguide  that you are not supposed to\nrun two mpdboot rings at the same time.\nCan someone explain this?\nIt works if  there are two rings on nonintersecting sets of nodes.\nOne solution is to simply always use ppn=2 (two processes\nper node) in pbs.  This is not the best answer though.\n\n```"},"comment":{"url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/comments/236338540","html_url":"https://github.com/raffenet/mpich-migrate-test/issues/625#issuecomment-236338540","issue_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/625","id":236338540,"user":{"login":"raffenet","id":1410792,"avatar_url":"https://avatars.githubusercontent.com/u/1410792?v=3","gravatar_id":"","url":"https://api.github.com/users/raffenet","html_url":"https://github.com/raffenet","followers_url":"https://api.github.com/users/raffenet/followers","following_url":"https://api.github.com/users/raffenet/following{/other_user}","gists_url":"https://api.github.com/users/raffenet/gists{/gist_id}","starred_url":"https://api.github.com/users/raffenet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raffenet/subscriptions","organizations_url":"https://api.github.com/users/raffenet/orgs","repos_url":"https://api.github.com/users/raffenet/repos","events_url":"https://api.github.com/users/raffenet/events{/privacy}","received_events_url":"https://api.github.com/users/raffenet/received_events","type":"User","site_admin":false},"created_at":"2016-07-30T03:37:37Z","updated_at":"2016-07-30T03:37:37Z","body":"**Originally by _Ralph Butler_ on 2009-05-30 06:44:37 -0500**\n\n___\n```\n\nThere are a couple of relevant comments one could make here:\n\n1.\nThere is an mpiexec from Pete Wyckoff @ osc.edu that you can use\ndirectly with pbs, i.e. no mpd, and which you may prefer.\n\n2.\nBoth pbs and mpd think of themselves as process management systems,\nthus there is some natural tension there.  Further, since mpd generally\nassumes it is operating independently, i.e. without another process\nmanager trying to \"help\", it plans on a single long-lived daemon per\nnode\nand can run multiple jobs via that one daemon.  While it is possible\nto run 2\nor more daemons per node, that is an undocumented option that was\ninvented\nmostly for testing and not general use.  But, there are at least 2\nmodels for\nstarting multiple daemons on one node.  One of them involves setting the\nenv var named MPD_CON_EXT to a non-empty simple ascii string.  If you\nthen start 2 rings with different values of MPD_CON_EXT, the daemons in\neach ring use their value to uniquely name Unix sockets which they\nplace in\nthe filesystem.\n\nI would propose looking into the wyckoff option for the short-term.\n--ralph\n\nOn FriMay 29, at Fri May 29 4:22PM, mpich2 wrote:\n\n> ----------------------------------------------\n> +-----------------------------\n> Reporter:  William Garber <garberw@bnl.gov>  |        Type:  bug\n>   Status:  new                               |    Priority:  major\n> Milestone:                                    |   Component:  mpich2\n> ----------------------------------------------\n> +-----------------------------\n>\n>\n> {{{\n>\n> I am using torque and I submit a pbs/torque script:\n> =======================\n> #!/bin/sh\n>\n> #PBS -N\n> uu\n>\n> #PBS\n> -V\n>\n> #PBS -l\n> nodes=5:ppn=1\n>\n>\n> export SCRATCH=\"/scratch/$LOGNAME/$PBS_JOBID\"\n> export SRC_DIR=$MPI_O_WORKDIR\n>\n> MPI_NODEFILE=$PBS_O_WORKDIR/mpd_nodes.${PBS_JOBID}\n> sort $PBS_NODEFILE | uniq -c | awk '{ printf(\"%s:%s\\n\", $2, $1);}' >\n> $MPI_NODEFILE\n>\n> NUMPROCESSES=`cat $PBS_NODEFILE | wc -l`\n> NUMNODES=`cat $MPI_NODEFILE | wc -l`\n> mpdboot -f $MPI_NODEFILE -n $NUMNODES --maxbranch=30\n> cd $PBS_O_WORKDIR\n> mpiexec -1 -machinefile $MPI_NODEFILE -np $NUMPROCESSES ./uu.sh >\n> uu.out.${PBS_JOBID} 2> uu.err.${PBS_JOBID}\n> mpdallexit\n> =========================================================\n> Repeated problem:\n> I run two jobs each having one process per node.\n> The jobs share at least one node.\n> The second job fails with the error:\n> ================================================\n> mpdboot_node005 (handle_mpd_output 385): failed to handshake with\n> mpd on\n> node001; recvd output={}\n>\n> Begin Fri May 29 16:02:41 EDT 2009\n> End Fri May 29 16:02:41 EDT 2009\n> mpdallexit: cannot connect to local mpd (/tmp/mpd2.console_garberw);\n> possible causes:\n>   1. no mpd is running on this host\n>   2. an mpd is running but was started without a \"console\" (-n option)\n> In case 1, you can start an mpd on this host with:\n>     mpd &\n> and you will be able to run jobs just on this host.\n> For more details on starting mpds on a set of hosts, see\n> the MPICH2 Installation Guide.\n> ================================================\n> the /etc/hosts on the nodes is\n> ==============\n> ::1     localhost6.localdomain6 localhost6\n> 127.0.0.1       localhost.localdomain   localhost\n> 172.16.100.2    clio.                   clio            head\n> 172.16.100.1    morpheus.               morpheus\n> 172.16.0.1      node001.        node001         n001\n> 172.16.0.2      node002.        node002         n002\n> 172.16.0.3      node003.        node003         n003\n> 172.16.0.4      node004.        node004         n004\n> 172.16.0.5      node005.        node005         n005\n> 172.16.0.6      node006.        node006         n006\n> 172.16.0.7      node007.        node007         n007\n> 172.16.0.8      node008.        node008         n008\n> 172.16.0.9      node009.        node009         n009\n> etc\n> ==============\n> the /etc/hosts on the head clio is\n> ==============\n> ::1     localhost6.localdomain6 localhost6\n> 127.0.0.1       localhost.localdomain   localhost\n> 172.16.0.1      node001.        node001         n001\n> 172.16.0.2      node002.        node002         n002\n> 172.16.0.3      node003.        node003         n003\n> 172.16.0.4      node004.        node004         n004\n> 172.16.0.5      node005.        node005         n005\n> 172.16.0.6      node006.        node006         n006\n> 172.16.0.7      node007.        node007         n007\n> 172.16.0.8      node008.        node008         n008\n> 172.16.0.9      node009.        node009         n009\n> 130.199.74.52   morpheus.phy.bnl.gov    morpheus\n> 172.16.100.2    clio.           clio            head\n> 130.199.74.51   clio.phy.bnl.gov\n> ==============\n> If it is not a problem with the /etc/hosts, then\n> I read in the troubleshooting section of the installation\n> guide  that you are not supposed to\n> run two mpdboot rings at the same time.\n> Can someone explain this?\n> It works if  there are two rings on nonintersecting sets of nodes.\n> One solution is to simply always use ppn=2 (two processes\n> per node) in pbs.  This is not the best answer though.\n>\n> ```\n>\n>\n> --\n> Ticket URL: <https://trac.mcs.anl.gov/projects/mpich2/ticket/625>\n\n}}}"}}
{"actor":{"display_login":"raffenet"}}