{"action":"opened","issue":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/5810","repository_url":"https://api.github.com/repos/open-mpi/ompi","labels_url":"https://api.github.com/repos/open-mpi/ompi/issues/5810/labels{/name}","comments_url":"https://api.github.com/repos/open-mpi/ompi/issues/5810/comments","events_url":"https://api.github.com/repos/open-mpi/ompi/issues/5810/events","html_url":"https://github.com/open-mpi/ompi/issues/5810","id":365153017,"node_id":"MDU6SXNzdWUzNjUxNTMwMTc=","number":5810,"title":"Component (openib) errors mlx4_0 device ","user":{"login":"abeltre1","id":11620908,"node_id":"MDQ6VXNlcjExNjIwOTA4","avatar_url":"https://avatars0.githubusercontent.com/u/11620908?v=4","gravatar_id":"","url":"https://api.github.com/users/abeltre1","html_url":"https://github.com/abeltre1","followers_url":"https://api.github.com/users/abeltre1/followers","following_url":"https://api.github.com/users/abeltre1/following{/other_user}","gists_url":"https://api.github.com/users/abeltre1/gists{/gist_id}","starred_url":"https://api.github.com/users/abeltre1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abeltre1/subscriptions","organizations_url":"https://api.github.com/users/abeltre1/orgs","repos_url":"https://api.github.com/users/abeltre1/repos","events_url":"https://api.github.com/users/abeltre1/events{/privacy}","received_events_url":"https://api.github.com/users/abeltre1/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-29T17:52:07Z","updated_at":"2018-09-29T17:52:07Z","closed_at":null,"author_association":"NONE","body":"Thank you for taking the time to submit an issue!\r\n\r\n## Background information\r\nIn this experiment, I am using Mellanox devices and drivers to run an MPI application (osu). I keep getting the errors below when I run with `openib` since the driver does not load for a specific host. When I take the particular host that `mpirun` throws as an error from the hostfile, it proceeds to throw another host, and another etc.  \r\n\r\n\r\n\r\n### What version of Open MPI are you using? \r\n\r\n- mpirun (Open MPI) 3.1.1\r\n\r\n### Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)\r\n\r\n- from Mellanox OFED software system distribution package \r\n\r\n### Please describe the system on which you are running\r\n\r\n* Operating system/version: Centos 7.5\r\n* Computer hardware: Intel Xion \r\n\r\n## Things are normal for TCP \r\n```shell\r\nmpirun --mca mtl_mxm_np 0 --mca btl tcp,self --mca plm_rsh_no_tree_spawn 1 --map-by node -hostfile ~/hostfile -np 72 ./osu_alltoallv \r\n# OSU MPI All-to-Allv Personalized Exchange Latency Test v5.3.2\r\n# Size       Avg Latency(us)\r\n1                      43.25\r\n2                      44.03\r\n4                      44.47\r\n8                      45.48\r\n16                     47.70\r\n32                     53.13\r\n64                     64.61\r\n128                   104.95\r\n256                   178.48\r\n512                   317.46\r\n1024                  607.22\r\n2048                  334.65\r\n4096                  641.85\r\n8192                 1624.47\r\n16384                3257.20\r\n32768                6246.35\r\n65536               12574.22\r\n131072              23627.27\r\n262144              38249.50\r\n524288              72622.47\r\n1048576            139280.63\r\n```\r\n## Things are not normal for openib \r\n```bash \r\nmpirun --mca mtl_mxm_np 0 --mca btl self,openib --mca plm_rsh_no_tree_spawn 1 --map-by node -hostfile ~/hostfile -np 24 ./osu_alltoallv \r\n\r\n\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-4][[6526,1],3][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-5][[6526,1],22][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-5][[6526,1],4][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\n--------------------------------------------------------------------------\r\nWARNING: There was an error initializing an OpenFabrics device.\r\n\r\n  Local host:   kube-infi-4\r\n  Local device: mlx4_0\r\n--------------------------------------------------------------------------\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-5][[6526,1],16][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-2][[6526,1],7][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-5][[6526,1],10][btl_openib_component.c:1670:init_one_device] ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-4][[6526,1],15][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nerror obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-2][[6526,1],1][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-2][[6526,1],13][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-6][[6526,1],17][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\n[kube-infi-2][[6526,1],19][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-1][[6526,1],6][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-1][[6526,1],12][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-1][[6526,1],0][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-6][[6526,1],5][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-4][[6526,1],9][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-1][[6526,1],18][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-6][[6526,1],11][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-4][[6526,1],21][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-6][[6526,1],23][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-3][[6526,1],8][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-3][[6526,1],14][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-3][[6526,1],20][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\nibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0xffffffffffffffff valid_mask = 0x1)\r\n[kube-infi-3][[6526,1],2][btl_openib_component.c:1670:init_one_device] error obtaining device attributes for mlx4_0 errno says Invalid argument\r\n\r\n# OSU MPI All-to-Allv Personalized Exchange Latency Test v5.3.2\r\n# Size       Avg Latency(us)\r\n1                      11.66\r\n2                      11.40\r\n4                      11.45\r\n8                      11.43\r\n16                     11.43\r\n32                     11.61\r\n64                     12.23\r\n128                    13.16\r\n256                    12.77\r\n512                    14.50\r\n1024                   21.37\r\n2048                   38.99\r\n4096                   69.41\r\n8192                  158.51\r\n16384                 296.76\r\n32768                 566.62\r\n65536                1176.81\r\n131072               2343.93\r\n262144               5253.02\r\n524288              10401.06\r\n\r\n\r\n\r\n```\r\n\r\n"}}
{"actor":{"display_login":"abeltre1"}}