{"action":"created","issue":{"url":"https://api.github.com/repos/nest/nest-simulator/issues/238","repository_url":"https://api.github.com/repos/nest/nest-simulator","labels_url":"https://api.github.com/repos/nest/nest-simulator/issues/238/labels{/name}","comments_url":"https://api.github.com/repos/nest/nest-simulator/issues/238/comments","events_url":"https://api.github.com/repos/nest/nest-simulator/issues/238/events","html_url":"https://github.com/nest/nest-simulator/issues/238","id":136728592,"number":238,"title":"No speedup from mpi parallelization (SLI and PyNEST)","user":{"login":"antolikjan","id":1029740,"avatar_url":"https://avatars.githubusercontent.com/u/1029740?v=3","gravatar_id":"","url":"https://api.github.com/users/antolikjan","html_url":"https://github.com/antolikjan","followers_url":"https://api.github.com/users/antolikjan/followers","following_url":"https://api.github.com/users/antolikjan/following{/other_user}","gists_url":"https://api.github.com/users/antolikjan/gists{/gist_id}","starred_url":"https://api.github.com/users/antolikjan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antolikjan/subscriptions","organizations_url":"https://api.github.com/users/antolikjan/orgs","repos_url":"https://api.github.com/users/antolikjan/repos","events_url":"https://api.github.com/users/antolikjan/events{/privacy}","received_events_url":"https://api.github.com/users/antolikjan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-02-26T15:34:42Z","updated_at":"2016-02-29T18:15:32Z","closed_at":null,"body":"Nest 2.10.0, intel compiler suite, RedHat version 6.7\r\n\r\nCompiled with following options:\r\nconfigure CC=mpiicc CFLAGS=\"-openmp -mt_mpi -O3\" CXX=mpiicpc CXXFLAGS=\"-openmp -mt_mpi -O3\" LDFLAGS=\"-L/.../mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -L/.../compiler/lib/intel64 -liomp5\"\r\n\r\nTo test parallelization we use the hpc_benchmark.sli and slightly modified version of the brunel_alpha_nest.py example script.\r\n\r\nThe parallelization with threads works perfectly, achieving nearly linear speedup with up to 32 threads (maximum cores on a single node).\r\n\r\nHowever, if I schedule a job with multiple mpi processes (single thread per mpi process) I observe no speedup, if anything the simulation runs slightly slower than in single thread/process condition, both for the sli and pynest scripts.\r\n\r\nThe outputs of the hpc_benchmark.sli when run with 1 and 16 threads:\r\n[hpc_benchmark_mpiproc=16_out.tar.gz](https://github.com/nest/nest-simulator/files/148454/hpc_benchmark_mpiproc.16_out.tar.gz)\r\n[hpc_benchmark_mpiproc=1_out.tar.gz](https://github.com/nest/nest-simulator/files/148455/hpc_benchmark_mpiproc.1_out.tar.gz)\r\n\r\nThe LoadLeveler submit script with which I submit the jobs:\r\n[sub.ll.tar.gz](https://github.com/nest/nest-simulator/files/148459/sub.ll.tar.gz)"},"comment":{"url":"https://api.github.com/repos/nest/nest-simulator/issues/comments/190317164","html_url":"https://github.com/nest/nest-simulator/issues/238#issuecomment-190317164","issue_url":"https://api.github.com/repos/nest/nest-simulator/issues/238","id":190317164,"user":{"login":"antolikjan","id":1029740,"avatar_url":"https://avatars.githubusercontent.com/u/1029740?v=3","gravatar_id":"","url":"https://api.github.com/users/antolikjan","html_url":"https://github.com/antolikjan","followers_url":"https://api.github.com/users/antolikjan/followers","following_url":"https://api.github.com/users/antolikjan/following{/other_user}","gists_url":"https://api.github.com/users/antolikjan/gists{/gist_id}","starred_url":"https://api.github.com/users/antolikjan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antolikjan/subscriptions","organizations_url":"https://api.github.com/users/antolikjan/orgs","repos_url":"https://api.github.com/users/antolikjan/repos","events_url":"https://api.github.com/users/antolikjan/events{/privacy}","received_events_url":"https://api.github.com/users/antolikjan/received_events","type":"User","site_admin":false},"created_at":"2016-02-29T18:15:32Z","updated_at":"2016-02-29T18:15:32Z","body":"@heplesser yes that is strange but that is the behavior I observe. To be clear I am now attaching the exact combination of files that I use to run the hpc_benchmark.sli with 32 processes:\r\n\r\n[hpc_benchmark.sli.tar.gz](https://github.com/nest/nest-simulator/files/151484/hpc_benchmark.sli.tar.gz)\r\n[hpc_b_output.tar.gz](https://github.com/nest/nest-simulator/files/151483/hpc_b_output.tar.gz) (the output of the simulation)\r\n[sub.ll.tar.gz](https://github.com/nest/nest-simulator/files/151482/sub.ll.tar.gz) (the loadlevel submit script)\r\n\r\nAs you can see I am setting the number of virtual processes in the sli script (and accordingly in the loadleveler submit script), and from the output you can see that in fact 32 instances of nest simulator are executed, but as you point out the number of local nodes remains 2253. Any idea what can cause this specific combinations of errors? To me it seems as if some MPI variables are not correctly propagated?"}}