{"action":"created","issue":{"url":"https://api.github.com/repos/hpcanalytics/paper-hpc-py-parallel-mdanalysis/issues/1","repository_url":"https://api.github.com/repos/hpcanalytics/paper-hpc-py-parallel-mdanalysis","labels_url":"https://api.github.com/repos/hpcanalytics/paper-hpc-py-parallel-mdanalysis/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/hpcanalytics/paper-hpc-py-parallel-mdanalysis/issues/1/comments","events_url":"https://api.github.com/repos/hpcanalytics/paper-hpc-py-parallel-mdanalysis/issues/1/events","html_url":"https://github.com/hpcanalytics/paper-hpc-py-parallel-mdanalysis/issues/1","id":324538405,"number":1,"title":"SuperMic MPI4py execution via custom OpenMPI installation","user":{"login":"iparask","id":8268665,"avatar_url":"https://avatars3.githubusercontent.com/u/8268665?v=4","gravatar_id":"","url":"https://api.github.com/users/iparask","html_url":"https://github.com/iparask","followers_url":"https://api.github.com/users/iparask/followers","following_url":"https://api.github.com/users/iparask/following{/other_user}","gists_url":"https://api.github.com/users/iparask/gists{/gist_id}","starred_url":"https://api.github.com/users/iparask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iparask/subscriptions","organizations_url":"https://api.github.com/users/iparask/orgs","repos_url":"https://api.github.com/users/iparask/repos","events_url":"https://api.github.com/users/iparask/events{/privacy}","received_events_url":"https://api.github.com/users/iparask/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"iparask","id":8268665,"avatar_url":"https://avatars3.githubusercontent.com/u/8268665?v=4","gravatar_id":"","url":"https://api.github.com/users/iparask","html_url":"https://github.com/iparask","followers_url":"https://api.github.com/users/iparask/followers","following_url":"https://api.github.com/users/iparask/following{/other_user}","gists_url":"https://api.github.com/users/iparask/gists{/gist_id}","starred_url":"https://api.github.com/users/iparask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iparask/subscriptions","organizations_url":"https://api.github.com/users/iparask/orgs","repos_url":"https://api.github.com/users/iparask/repos","events_url":"https://api.github.com/users/iparask/events{/privacy}","received_events_url":"https://api.github.com/users/iparask/received_events","type":"User","site_admin":false},"assignees":[{"login":"iparask","id":8268665,"avatar_url":"https://avatars3.githubusercontent.com/u/8268665?v=4","gravatar_id":"","url":"https://api.github.com/users/iparask","html_url":"https://github.com/iparask","followers_url":"https://api.github.com/users/iparask/followers","following_url":"https://api.github.com/users/iparask/following{/other_user}","gists_url":"https://api.github.com/users/iparask/gists{/gist_id}","starred_url":"https://api.github.com/users/iparask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iparask/subscriptions","organizations_url":"https://api.github.com/users/iparask/orgs","repos_url":"https://api.github.com/users/iparask/repos","events_url":"https://api.github.com/users/iparask/events{/privacy}","received_events_url":"https://api.github.com/users/iparask/received_events","type":"User","site_admin":false},{"login":"mkhoshle","id":21323137,"avatar_url":"https://avatars3.githubusercontent.com/u/21323137?v=4","gravatar_id":"","url":"https://api.github.com/users/mkhoshle","html_url":"https://github.com/mkhoshle","followers_url":"https://api.github.com/users/mkhoshle/followers","following_url":"https://api.github.com/users/mkhoshle/following{/other_user}","gists_url":"https://api.github.com/users/mkhoshle/gists{/gist_id}","starred_url":"https://api.github.com/users/mkhoshle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkhoshle/subscriptions","organizations_url":"https://api.github.com/users/mkhoshle/orgs","repos_url":"https://api.github.com/users/mkhoshle/repos","events_url":"https://api.github.com/users/mkhoshle/events{/privacy}","received_events_url":"https://api.github.com/users/mkhoshle/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-05-18T20:14:27Z","updated_at":"2018-05-22T22:20:07Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I create the following script and run it on SuperMic using a python module offered by SUpermic's admins and the python environment I installed based on your instructions. The script is:\r\n```python\r\n#!/usr/bin/env python\r\nimport os\r\nfrom mpi4py import MPI\r\n\r\ncomm = MPI.COMM_WORLD\r\nsize = comm.Get_size()\r\nrank = comm.Get_rank()\r\n\r\nprint(\"I am rank %d from %d procs, running on %s\"%(rank,size,os.uname()[1]))\r\n```\r\nEssentially it prints the rank of a process, the world size and the hostname of the node the process run.\r\n\r\nWhen I used the python module provided by SuperMic I got:\r\n```bash\r\n[iparaske@smic067 MPI-RMSD-Parallel-IO]$ module load python/2.7.10-mkl-mic\r\n[iparaske@smic067 MPI-RMSD-Parallel-IO]$ mpirun -n 40 python test_mpi.py\r\nI am rank 5 from 40 procs, running on smic067\r\nI am rank 26 from 40 procs, running on smic068\r\nI am rank 38 from 40 procs, running on smic068\r\nI am rank 34 from 40 procs, running on smic068\r\nI am rank 19 from 40 procs, running on smic067\r\nI am rank 20 from 40 procs, running on smic068\r\nI am rank 16 from 40 procs, running on smic067\r\nI am rank 24 from 40 procs, running on smic068\r\nI am rank 18 from 40 procs, running on smic067\r\nI am rank 13 from 40 procs, running on smic067\r\nI am rank 32 from 40 procs, running on smic068\r\nI am rank 12 from 40 procs, running on smic067\r\nI am rank 35 from 40 procs, running on smic068\r\nI am rank 23 from 40 procs, running on smic068\r\nI am rank 3 from 40 procs, running on smic067\r\nI am rank 22 from 40 procs, running on smic068\r\nI am rank 9 from 40 procs, running on smic067\r\nI am rank 6 from 40 procs, running on smic067\r\nI am rank 21 from 40 procs, running on smic068\r\nI am rank 29 from 40 procs, running on smic068\r\nI am rank 2 from 40 procs, running on smic067\r\nI am rank 14 from 40 procs, running on smic067\r\nI am rank 27 from 40 procs, running on smic068\r\nI am rank 4 from 40 procs, running on smic067\r\nI am rank 30 from 40 procs, running on smic068\r\nI am rank 31 from 40 procs, running on smic068\r\nI am rank 11 from 40 procs, running on smic067\r\nI am rank 0 from 40 procs, running on smic067\r\nI am rank 37 from 40 procs, running on smic068\r\nI am rank 17 from 40 procs, running on smic067\r\nI am rank 36 from 40 procs, running on smic068\r\nI am rank 8 from 40 procs, running on smic067\r\nI am rank 25 from 40 procs, running on smic068\r\nI am rank 10 from 40 procs, running on smic067\r\nI am rank 33 from 40 procs, running on smic068\r\nI am rank 1 from 40 procs, running on smic067\r\nI am rank 28 from 40 procs, running on smic068\r\nI am rank 39 from 40 procs, running on smic068I am rank 7 from 40 procs, running on smic067\r\n\r\nI am rank 15 from 40 procs, running on smic067\r\n```\r\n\r\nwhich is what I expected it to be.\r\n\r\nBut when I used our installation, I got:\r\n```bash\r\n[iparaske@smic067 MPI-RMSD-Parallel-IO]$ module purge\r\n[iparaske@smic067 MPI-RMSD-Parallel-IO]$ module load gcc/4.9.0\r\n[iparaske@smic067 MPI-RMSD-Parallel-IO]$ module load openmpi_custom\r\n[iparaske@smic067 MPI-RMSD-Parallel-IO]$ module load hdf5_custom\r\n[iparaske@smic067 MPI-RMSD-Parallel-IO]$\r\n[iparaske@smic067 MPI-RMSD-Parallel-IO]$ export PATH=/home/iparaske/miniconda3/bin:$PATH\r\n[iparaske@smic067 MPI-RMSD-Parallel-IO]$ export HDF5_MPI=\"ON\"\r\n[iparaske@smic067 MPI-RMSD-Parallel-IO]$\r\n[iparaske@smic067 MPI-RMSD-Parallel-IO]$ source activate mpio\r\n(mpio) [iparaske@smic067 MPI-RMSD-Parallel-IO]$\r\n(mpio) [iparaske@smic067 MPI-RMSD-Parallel-IO]$ mpirun -n 40 python test_mpi.py\r\nI am rank 16 from 40 procs, running on smic067\r\nI am rank 17 from 40 procs, running on smic067\r\nI am rank 18 from 40 procs, running on smic067\r\nI am rank 19 from 40 procs, running on smic067\r\nI am rank 22 from 40 procs, running on smic067\r\nI am rank 24 from 40 procs, running on smic067\r\nI am rank 25 from 40 procs, running on smic067\r\nI am rank 28 from 40 procs, running on smic067\r\nI am rank 29 from 40 procs, running on smic067\r\nI am rank 30 from 40 procs, running on smic067\r\nI am rank 31 from 40 procs, running on smic067\r\nI am rank 33 from 40 procs, running on smic067\r\nI am rank 34 from 40 procs, running on smic067\r\nI am rank 37 from 40 procs, running on smic067\r\nI am rank 38 from 40 procs, running on smic067\r\nI am rank 0 from 40 procs, running on smic067\r\nI am rank 1 from 40 procs, running on smic067\r\nI am rank 2 from 40 procs, running on smic067\r\nI am rank 3 from 40 procs, running on smic067\r\nI am rank 4 from 40 procs, running on smic067\r\nI am rank 5 from 40 procs, running on smic067\r\nI am rank 11 from 40 procs, running on smic067\r\nI am rank 12 from 40 procs, running on smic067\r\nI am rank 13 from 40 procs, running on smic067\r\nI am rank 15 from 40 procs, running on smic067\r\nI am rank 7 from 40 procs, running on smic067\r\nI am rank 8 from 40 procs, running on smic067\r\nI am rank 9 from 40 procs, running on smic067\r\nI am rank 36 from 40 procs, running on smic067\r\nI am rank 21 from 40 procs, running on smic067\r\nI am rank 6 from 40 procs, running on smic067\r\nI am rank 32 from 40 procs, running on smic067\r\nI am rank 35 from 40 procs, running on smic067\r\nI am rank 10 from 40 procs, running on smic067\r\nI am rank 20 from 40 procs, running on smic067\r\nI am rank 26 from 40 procs, running on smic067\r\nI am rank 39 from 40 procs, running on smic067\r\nI am rank 27 from 40 procs, running on smic067\r\nI am rank 23 from 40 procs, running on smic067\r\nI am rank 14 from 40 procs, running on smic067\r\n```\r\n\r\nClearly, the custom installation does not place the processes correctly. I will run a check with the placement command you provided to see what is going on."},"comment":{"url":"https://api.github.com/repos/hpcanalytics/paper-hpc-py-parallel-mdanalysis/issues/comments/391159115","html_url":"https://github.com/hpcanalytics/paper-hpc-py-parallel-mdanalysis/issues/1#issuecomment-391159115","issue_url":"https://api.github.com/repos/hpcanalytics/paper-hpc-py-parallel-mdanalysis/issues/1","id":391159115,"user":{"login":"iparask","id":8268665,"avatar_url":"https://avatars3.githubusercontent.com/u/8268665?v=4","gravatar_id":"","url":"https://api.github.com/users/iparask","html_url":"https://github.com/iparask","followers_url":"https://api.github.com/users/iparask/followers","following_url":"https://api.github.com/users/iparask/following{/other_user}","gists_url":"https://api.github.com/users/iparask/gists{/gist_id}","starred_url":"https://api.github.com/users/iparask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iparask/subscriptions","organizations_url":"https://api.github.com/users/iparask/orgs","repos_url":"https://api.github.com/users/iparask/repos","events_url":"https://api.github.com/users/iparask/events{/privacy}","received_events_url":"https://api.github.com/users/iparask/received_events","type":"User","site_admin":false},"created_at":"2018-05-22T22:20:07Z","updated_at":"2018-05-22T22:20:07Z","author_association":"CONTRIBUTOR","body":"There are two issues that did not allow the execution to distribute correctly on SuperMic. Apparenlty, none of those had to do with the exact distribution of the processes.\r\n\r\nFirst of all, SuperMic is a PBS cluster and not SLURM. That means that information about the node list and environment is a bit different compared to Comet and Bridges (both SLURM clusters).\r\n\r\nInitially, I had to reinstall OpenMPI and include `--enable-orterun-prefix-by-default` flag during configuration. Secondly, I had to include the path and library path to my bashrc. It seems that `mpirun` did not see the correct configuration although I was passing `PATH` and `LD_LIBRARY_PATH` with the `-x` flag. I was told by the lead developer in the lab that `-x` option has to do with the workload and not with `mpirun` and suggested to include the path to the executable and libraries to my `.bashrc`. This is was the first issue.\r\n\r\nThe second issue was that `mpirun` did not see the host list correctly and assumed a single node. In order to distribute the processes correctly to the acquired nodes the host list had to provided manually.\r\n\r\nBased on all this:\r\n 1. The path should be setup in the `.bashrc` so that all nodes see the correct `mpirun` and the correct libraries, and\r\n 2. The hostlist should be discovered by the user and passed to `mpirun`. \r\n\r\nSo the command on SuperMic look like:\r\n```\r\n(mpio) [iparaske@smic025 MPI-RMSD-Parallel-IO]$ mpirun -x PATH -x LD_LIBRARY_PATH -host smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic025,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027,smic027 -np 40 python test_mpi.py\r\n```\r\nAnd the output is:\r\n```\r\nI am rank 33 from 40 procs, running on smic027.\r\nI am rank 34 from 40 procs, running on smic027.\r\nI am rank 35 from 40 procs, running on smic027.\r\nI am rank 36 from 40 procs, running on smic027.\r\nI am rank 37 from 40 procs, running on smic027.\r\nI am rank 38 from 40 procs, running on smic027.\r\nI am rank 39 from 40 procs, running on smic027.\r\nI am rank 20 from 40 procs, running on smic027.\r\nI am rank 21 from 40 procs, running on smic027.\r\nI am rank 23 from 40 procs, running on smic027.\r\nI am rank 24 from 40 procs, running on smic027.\r\nI am rank 25 from 40 procs, running on smic027.\r\nI am rank 26 from 40 procs, running on smic027.\r\nI am rank 27 from 40 procs, running on smic027.\r\nI am rank 28 from 40 procs, running on smic027.\r\nI am rank 29 from 40 procs, running on smic027.\r\nI am rank 30 from 40 procs, running on smic027.\r\nI am rank 31 from 40 procs, running on smic027.\r\nI am rank 32 from 40 procs, running on smic027.\r\nI am rank 14 from 40 procs, running on smic025.\r\nI am rank 15 from 40 procs, running on smic025.\r\nI am rank 16 from 40 procs, running on smic025.\r\nI am rank 17 from 40 procs, running on smic025.\r\nI am rank 18 from 40 procs, running on smic025.\r\nI am rank 19 from 40 procs, running on smic025.\r\nI am rank 0 from 40 procs, running on smic025.\r\nI am rank 1 from 40 procs, running on smic025.\r\nI am rank 2 from 40 procs, running on smic025.\r\nI am rank 3 from 40 procs, running on smic025.\r\nI am rank 4 from 40 procs, running on smic025.\r\nI am rank 5 from 40 procs, running on smic025.\r\nI am rank 6 from 40 procs, running on smic025.\r\nI am rank 7 from 40 procs, running on smic025.\r\nI am rank 8 from 40 procs, running on smic025.\r\nI am rank 9 from 40 procs, running on smic025.\r\nI am rank 10 from 40 procs, running on smic025.\r\nI am rank 11 from 40 procs, running on smic025.\r\nI am rank 12 from 40 procs, running on smic025.\r\nI am rank 13 from 40 procs, running on smic025.\r\n```\r\n\r\nConcerning the experiments. Unless you have to propose a specific distribution, I will put the first 20 processes to the first node, the second 20 to the second node and so on. When, they finish I will provide you with the data."}}
{"actor":{"display_login":"iparask"}}