{"action":"created","issue":{"url":"https://api.github.com/repos/xcat2/xcat-core/issues/505","repository_url":"https://api.github.com/repos/xcat2/xcat-core","labels_url":"https://api.github.com/repos/xcat2/xcat-core/issues/505/labels{/name}","comments_url":"https://api.github.com/repos/xcat2/xcat-core/issues/505/comments","events_url":"https://api.github.com/repos/xcat2/xcat-core/issues/505/events","html_url":"https://github.com/xcat2/xcat-core/issues/505","id":119868278,"number":505,"title":"［Customer］xcat 2.11 provision diskfull RHEL7.2LE and InfiniBand MLNX OFED may need a reboot(?)","user":{"login":"lxguang","id":15136852,"avatar_url":"https://avatars.githubusercontent.com/u/15136852?v=3","gravatar_id":"","url":"https://api.github.com/users/lxguang","html_url":"https://github.com/lxguang","followers_url":"https://api.github.com/users/lxguang/followers","following_url":"https://api.github.com/users/lxguang/following{/other_user}","gists_url":"https://api.github.com/users/lxguang/gists{/gist_id}","starred_url":"https://api.github.com/users/lxguang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lxguang/subscriptions","organizations_url":"https://api.github.com/users/lxguang/orgs","repos_url":"https://api.github.com/users/lxguang/repos","events_url":"https://api.github.com/users/lxguang/events{/privacy}","received_events_url":"https://api.github.com/users/lxguang/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/xcat2/xcat-core/labels/component:infiniband","name":"component:infiniband","color":"fbca04"},{"url":"https://api.github.com/repos/xcat2/xcat-core/labels/priority:normal","name":"priority:normal","color":"009800"},{"url":"https://api.github.com/repos/xcat2/xcat-core/labels/type:feature","name":"type:feature","color":"c7def8"}],"state":"open","locked":false,"assignee":{"login":"hu-weihua","id":7370626,"avatar_url":"https://avatars.githubusercontent.com/u/7370626?v=3","gravatar_id":"","url":"https://api.github.com/users/hu-weihua","html_url":"https://github.com/hu-weihua","followers_url":"https://api.github.com/users/hu-weihua/followers","following_url":"https://api.github.com/users/hu-weihua/following{/other_user}","gists_url":"https://api.github.com/users/hu-weihua/gists{/gist_id}","starred_url":"https://api.github.com/users/hu-weihua/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hu-weihua/subscriptions","organizations_url":"https://api.github.com/users/hu-weihua/orgs","repos_url":"https://api.github.com/users/hu-weihua/repos","events_url":"https://api.github.com/users/hu-weihua/events{/privacy}","received_events_url":"https://api.github.com/users/hu-weihua/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/xcat2/xcat-core/milestones/6","html_url":"https://github.com/xcat2/xcat-core/milestones/2.12","labels_url":"https://api.github.com/repos/xcat2/xcat-core/milestones/6/labels","id":1369794,"number":6,"title":"2.12","description":"","creator":{"login":"zet809","id":8222144,"avatar_url":"https://avatars.githubusercontent.com/u/8222144?v=3","gravatar_id":"","url":"https://api.github.com/users/zet809","html_url":"https://github.com/zet809","followers_url":"https://api.github.com/users/zet809/followers","following_url":"https://api.github.com/users/zet809/following{/other_user}","gists_url":"https://api.github.com/users/zet809/gists{/gist_id}","starred_url":"https://api.github.com/users/zet809/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zet809/subscriptions","organizations_url":"https://api.github.com/users/zet809/orgs","repos_url":"https://api.github.com/users/zet809/repos","events_url":"https://api.github.com/users/zet809/events{/privacy}","received_events_url":"https://api.github.com/users/zet809/received_events","type":"User","site_admin":false},"open_issues":102,"closed_issues":279,"state":"open","created_at":"2015-10-22T07:51:37Z","updated_at":"2016-04-19T07:36:27Z","due_on":"2016-05-20T04:00:00Z","closed_at":null},"comments":3,"created_at":"2015-12-02T05:02:42Z","updated_at":"2016-04-19T09:04:58Z","closed_at":null,"body":"Hi,  I have been following the URL https://xcat-docs.readthedocs.org/en/latest/advanced/networks/infiniband/mlnxofed_ib_install_v2_preparation.html to provision diskfull RHEL7.2 LE with MLNX OFED by xCAT 2.11 running Ubuntu 14.04.2 LE on Power 8.\r\n```\r\n# cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install.v2    /install/postscripts/mlnxofed_ib_install\r\n# chdef compute -p postbootscripts=\"mlnxofed_ib_install -p /install/rhels7.2/ppc64le/MLNX_OFED_LINUX-3.1-1.0.3.1-rhel7.2-ppc64le.iso -m --without-32bit --without-fw-update --add-kernel-support --force \"\r\n# chmod +x /install/postscripts/mlnxofed_ib_install\r\n```\r\nWhen the FireStone node is provisioned, it looks like the InfiniBand is up with the specified IP address. Ping other IB ip addresses works.  \r\n```\r\n[root@fs04 ~]# ibstat\r\nCA 'mlx4_0'\r\n        CA type: MT4099\r\n        Number of ports: 2\r\n        Firmware version: 2.35.5100\r\n        Hardware version: 0\r\n        Node GUID: 0xe41d2d0300233ea0\r\n        System image GUID: 0xe41d2d0300233ea0\r\n        Port 1:\r\n                State: Down\r\n                Physical state: Disabled\r\n                Rate: 40\r\n                Base lid: 0\r\n                LMC: 0\r\n                SM lid: 0\r\n                Capability mask: 0x04010000\r\n                Port GUID: 0xe61d2dfffe233ea0\r\n                Link layer: Ethernet\r\n        Port 2:\r\n                State: Down\r\n                Physical state: Disabled\r\n                Rate: 40\r\n                Base lid: 0\r\n                LMC: 0\r\n                SM lid: 0\r\n                Capability mask: 0x04010000\r\n                Port GUID: 0xe61d2dfffe233ea1\r\n                Link layer: Ethernet\r\nCA 'mlx5_0'\r\n        CA type: MT4113\r\n        Number of ports: 2\r\n        Firmware version: 10.12.1100\r\n        Hardware version: 0\r\n        Node GUID: 0xf4521403006d5010\r\n        System image GUID: 0xf4521403006d5010\r\n        Port 1:\r\n                State: Active\r\n                Physical state: LinkUp\r\n                Rate: 56\r\n                Base lid: 14\r\n                LMC: 0\r\n                SM lid: 1\r\n                Capability mask: 0x26596848\r\n                Port GUID: 0xf4521403006d5010\r\n                Link layer: InfiniBand\r\n        Port 2:\r\n                State: Active\r\n                Physical state: LinkUp\r\n                Rate: 56\r\n                Base lid: 29\r\n                LMC: 0\r\n                SM lid: 1\r\n                Capability mask: 0x26596848\r\n                Port GUID: 0xf4521403006d5018\r\n                Link layer: InfiniBand\r\n[root@fs04 ~]# ibstatus\r\nInfiniband device 'mlx4_0' port 1 status:\r\n        default gid:     fe80:0000:0000:0000:e61d:2dff:fe23:3ea0\r\n        base lid:        0x0\r\n        sm lid:          0x0\r\n        state:           1: DOWN\r\n        phys state:      3: Disabled\r\n        rate:            40 Gb/sec (4X QDR)\r\n        link_layer:      Ethernet\r\n\r\nInfiniband device 'mlx4_0' port 2 status:\r\n        default gid:     fe80:0000:0000:0000:e61d:2dff:fe23:3ea1\r\n        base lid:        0x0\r\n        sm lid:          0x0\r\n        state:           1: DOWN\r\n        phys state:      3: Disabled\r\n        rate:            40 Gb/sec (4X QDR)\r\n        link_layer:      Ethernet\r\n\r\nInfiniband device 'mlx5_0' port 1 status:\r\n        default gid:     fe80:0000:0000:0000:f452:1403:006d:5010\r\n        base lid:        0xe\r\n        sm lid:          0x1\r\n        state:           4: ACTIVE\r\n        phys state:      5: LinkUp\r\n        rate:            56 Gb/sec (4X FDR)\r\n        link_layer:      InfiniBand\r\n\r\nInfiniband device 'mlx5_0' port 2 status:\r\n        default gid:     fe80:0000:0000:0000:f452:1403:006d:5018\r\n        base lid:        0x1d\r\n        sm lid:          0x1\r\n        state:           4: ACTIVE\r\n        phys state:      5: LinkUp\r\n        rate:            56 Gb/sec (4X FDR)\r\n        link_layer:      InfiniBand\r\n\r\n[root@fs04 ~]# ping gssio1-ib0 -c5\r\nPING gssio1-ib0.gpfs.net (172.18.30.21) 56(84) bytes of data.\r\n64 bytes from gssio1-ib0.gpfs.net (172.18.30.21): icmp_seq=1 ttl=64 time=3.11 ms\r\n64 bytes from gssio1-ib0.gpfs.net (172.18.30.21): icmp_seq=2 ttl=64 time=0.051 ms\r\n64 bytes from gssio1-ib0.gpfs.net (172.18.30.21): icmp_seq=3 ttl=64 time=0.043 ms\r\n64 bytes from gssio1-ib0.gpfs.net (172.18.30.21): icmp_seq=4 ttl=64 time=0.049 ms\r\n64 bytes from gssio1-ib0.gpfs.net (172.18.30.21): icmp_seq=5 ttl=64 time=0.040 ms\r\n\r\n--- gssio1-ib0.gpfs.net ping statistics ---\r\n5 packets transmitted, 5 received, 0% packet loss, time 3998ms\r\nrtt min/avg/max/mdev = 0.040/0.658/3.110/1.226 ms\r\n```\r\nYet, when trying to log into the ESS server by ssh, it takes long time to log onto the GPFS quorum server. And on the provisioned node, it failed to re-startup the openibd service.\r\n```\r\n[root@fs04 ~]# /etc/init.d/openibd restart\r\nCannot unload driver while ib_isert is loaded...\r\nPlease close all isert sessions and unload ib_isert module first.\r\n```\r\nA reboot can solve the problem.\r\n\r\nHere is a workaround without reboot.\r\nOn the node, unload the ib_isert, xprtrdma, ib_srpt module, and then restart openibd, it works.\r\n```\r\n# modprobe -r ib_isert xprtrdma ib_srpt\r\n# /etc/init.d/openibd restart\r\n```\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/xcat2/xcat-core/issues/comments/211813932","html_url":"https://github.com/xcat2/xcat-core/issues/505#issuecomment-211813932","issue_url":"https://api.github.com/repos/xcat2/xcat-core/issues/505","id":211813932,"user":{"login":"hu-weihua","id":7370626,"avatar_url":"https://avatars.githubusercontent.com/u/7370626?v=3","gravatar_id":"","url":"https://api.github.com/users/hu-weihua","html_url":"https://github.com/hu-weihua","followers_url":"https://api.github.com/users/hu-weihua/followers","following_url":"https://api.github.com/users/hu-weihua/following{/other_user}","gists_url":"https://api.github.com/users/hu-weihua/gists{/gist_id}","starred_url":"https://api.github.com/users/hu-weihua/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hu-weihua/subscriptions","organizations_url":"https://api.github.com/users/hu-weihua/orgs","repos_url":"https://api.github.com/users/hu-weihua/repos","events_url":"https://api.github.com/users/hu-weihua/events{/privacy}","received_events_url":"https://api.github.com/users/hu-weihua/received_events","type":"User","site_admin":false},"created_at":"2016-04-19T09:04:58Z","updated_at":"2016-04-19T09:04:58Z","body":"Now ``mlnxofed_ib_install.v2`` can run during ``postscripts`` phase to avoid reboot server twice in the scenarios we have verified, which include ``rhels7.2+ppc64le`` scenario.  please refer to the latest document. "}}