{"action":"created","issue":{"url":"https://api.github.com/repos/gc3-uzh-ch/elasticluster/issues/357","repository_url":"https://api.github.com/repos/gc3-uzh-ch/elasticluster","labels_url":"https://api.github.com/repos/gc3-uzh-ch/elasticluster/issues/357/labels{/name}","comments_url":"https://api.github.com/repos/gc3-uzh-ch/elasticluster/issues/357/comments","events_url":"https://api.github.com/repos/gc3-uzh-ch/elasticluster/issues/357/events","html_url":"https://github.com/gc3-uzh-ch/elasticluster/issues/357","id":192726104,"number":357,"title":"NFS client mounts fail","user":{"login":"mbookman","id":5818294,"avatar_url":"https://avatars.githubusercontent.com/u/5818294?v=3","gravatar_id":"","url":"https://api.github.com/users/mbookman","html_url":"https://github.com/mbookman","followers_url":"https://api.github.com/users/mbookman/followers","following_url":"https://api.github.com/users/mbookman/following{/other_user}","gists_url":"https://api.github.com/users/mbookman/gists{/gist_id}","starred_url":"https://api.github.com/users/mbookman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbookman/subscriptions","organizations_url":"https://api.github.com/users/mbookman/orgs","repos_url":"https://api.github.com/users/mbookman/repos","events_url":"https://api.github.com/users/mbookman/events{/privacy}","received_events_url":"https://api.github.com/users/mbookman/received_events","type":"User","site_admin":false},"labels":[{"id":446040522,"url":"https://api.github.com/repos/gc3-uzh-ch/elasticluster/labels/debian","name":"debian","color":"c70036","default":false},{"id":418239467,"url":"https://api.github.com/repos/gc3-uzh-ch/elasticluster/labels/gridengine","name":"gridengine","color":"1d76db","default":false}],"state":"open","locked":false,"assignee":{"login":"riccardomurri","id":447228,"avatar_url":"https://avatars.githubusercontent.com/u/447228?v=3","gravatar_id":"","url":"https://api.github.com/users/riccardomurri","html_url":"https://github.com/riccardomurri","followers_url":"https://api.github.com/users/riccardomurri/followers","following_url":"https://api.github.com/users/riccardomurri/following{/other_user}","gists_url":"https://api.github.com/users/riccardomurri/gists{/gist_id}","starred_url":"https://api.github.com/users/riccardomurri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/riccardomurri/subscriptions","organizations_url":"https://api.github.com/users/riccardomurri/orgs","repos_url":"https://api.github.com/users/riccardomurri/repos","events_url":"https://api.github.com/users/riccardomurri/events{/privacy}","received_events_url":"https://api.github.com/users/riccardomurri/received_events","type":"User","site_admin":false},"assignees":[{"login":"riccardomurri","id":447228,"avatar_url":"https://avatars.githubusercontent.com/u/447228?v=3","gravatar_id":"","url":"https://api.github.com/users/riccardomurri","html_url":"https://github.com/riccardomurri","followers_url":"https://api.github.com/users/riccardomurri/followers","following_url":"https://api.github.com/users/riccardomurri/following{/other_user}","gists_url":"https://api.github.com/users/riccardomurri/gists{/gist_id}","starred_url":"https://api.github.com/users/riccardomurri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/riccardomurri/subscriptions","organizations_url":"https://api.github.com/users/riccardomurri/orgs","repos_url":"https://api.github.com/users/riccardomurri/repos","events_url":"https://api.github.com/users/riccardomurri/events{/privacy}","received_events_url":"https://api.github.com/users/riccardomurri/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/gc3-uzh-ch/elasticluster/milestones/7","html_url":"https://github.com/gc3-uzh-ch/elasticluster/milestones/1.4","labels_url":"https://api.github.com/repos/gc3-uzh-ch/elasticluster/milestones/7/labels","id":1444935,"number":7,"title":"1.4","description":"Goals:\r\n\r\n- support Azure\r\n- ensure that AWS EC2 support is on par with OpenStack and GCE\r\n- playbook to install Hadoop2 and Spark/YARN\r\n","creator":{"login":"riccardomurri","id":447228,"avatar_url":"https://avatars.githubusercontent.com/u/447228?v=3","gravatar_id":"","url":"https://api.github.com/users/riccardomurri","html_url":"https://github.com/riccardomurri","followers_url":"https://api.github.com/users/riccardomurri/followers","following_url":"https://api.github.com/users/riccardomurri/following{/other_user}","gists_url":"https://api.github.com/users/riccardomurri/gists{/gist_id}","starred_url":"https://api.github.com/users/riccardomurri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/riccardomurri/subscriptions","organizations_url":"https://api.github.com/users/riccardomurri/orgs","repos_url":"https://api.github.com/users/riccardomurri/repos","events_url":"https://api.github.com/users/riccardomurri/events{/privacy}","received_events_url":"https://api.github.com/users/riccardomurri/received_events","type":"User","site_admin":false},"open_issues":17,"closed_issues":5,"state":"open","created_at":"2015-12-04T21:06:32Z","updated_at":"2016-12-05T06:03:53Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2016-12-01T00:57:34Z","updated_at":"2016-12-06T03:24:49Z","closed_at":null,"body":"This is a re-occurrence of Issue #253.\r\n\r\nMy gridengine cluster on Debian 8 had failed to initialize - I noted this in my comment on commit 806eef7 (\"Job type reload is not applicable for unit netfilter-persistent.service\").\r\n\r\nI looked to ignore that issue and proceed, so I then ran:\r\n\r\n```\r\n$ elasticluster -v -v -v setup gridengine\r\n```\r\n\r\nwhich failed with:\r\n\r\n```\r\nTASK [nfs-client : add to /etc/fstab] ******************************************\r\ntask path: /usr/local/google/home/mbookman/.python-eggs/elasticluster-1.3.dev0-py2.7.egg-tmp/elasticluster/share/playbooks/roles/roles/nfs-client/tasks/nfsmount.yml:8\r\nfatal: [compute002]: FAILED! => {\"changed\": false, \"failed\": true, \"msg\": \"Error mounting /home: mount.nfs: Connection timed out\\n\"}\r\nfatal: [compute001]: FAILED! => {\"changed\": false, \"failed\": true, \"msg\": \"Error mounting /home: mount.nfs: Connection timed out\\n\"}\r\n\tto retry, use: --limit @/usr/local/google/home/mbookman/.python-eggs/elasticluster-1.3.dev0-py2.7.egg-tmp/elasticluster/share/playbooks/site.retry\r\n```\r\n\r\nI connected to the frontend instance and could see that `/etc/exports` seemed to be properly updated:\r\n\r\n```\r\nmbookman@frontend001:~$ tail -n 2 /etc/exports \r\n/home compute001(rw,no_root_squash,async) compute002(rw,no_root_squash,async)\r\n/usr/share/gridengine/default/common compute001(rw,no_root_squash) compute002(rw,no_root_squash) frontend001(rw,no_root_squash)\r\n```\r\n\r\nBut it doesn't seem like the nfs server start completed or picked up the exports:\r\n\r\n```\r\nmbookman@frontend001:~$ sudo showmount -e localhost\r\nclnt_create: RPC: Program not registered\r\n\r\nmbookman@frontend001:~$ sudo /etc/init.d/nfs-kernel-server status\r\n● nfs-kernel-server.service - LSB: Kernel NFS server support\r\n   Loaded: loaded (/etc/init.d/nfs-kernel-server)\r\n   Active: active (exited) since Thu 2016-12-01 00:22:13 UTC; 23min ago\r\n\r\nDec 01 00:22:13 frontend001 nfs-kernel-server[21774]: Not starting NFS kernel...\r\nDec 01 00:22:13 frontend001 systemd[1]: Started LSB: Kernel NFS server support.\r\nHint: Some lines were ellipsized, use -l to show in full.\r\n```\r\n\r\nI could see that nfs-server/tasks/main.yml tries to reload the NFS exports:\r\n\r\n```\r\n- name: Reload NFS exports file\r\n  shell:\r\n    exportfs -r\r\n```\r\n\r\nBut this had generated a bunch of errors. More clear here running it directly:\r\n\r\n```\r\nmbookman@frontend001:~$ sudo exportfs -r\r\nexportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export \"compute001:/home\".\r\n  Assuming default behaviour ('no_subtree_check').\r\n  NOTE: this default has changed since nfs-utils version 1.0.x\r\n\r\nexportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export \"compute002:/home\".\r\n  Assuming default behaviour ('no_subtree_check').\r\n  NOTE: this default has changed since nfs-utils version 1.0.x\r\n\r\nexportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export \"compute001:/usr/share/gridengine/default/common\".\r\n  Assuming default behaviour ('no_subtree_check').\r\n  NOTE: this default has changed since nfs-utils version 1.0.x\r\n\r\nexportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export \"compute002:/usr/share/gridengine/default/common\".\r\n  Assuming default behaviour ('no_subtree_check').\r\n  NOTE: this default has changed since nfs-utils version 1.0.x\r\n\r\nexportfs: /etc/exports [2]: Neither 'subtree_check' or 'no_subtree_check' specified for export \"frontend001:/usr/share/gridengine/default/common\".\r\n  Assuming default behaviour ('no_subtree_check').\r\n  NOTE: this default has changed since nfs-utils version 1.0.x\r\n\r\nexportfs: gridengine-compute001.c.project.internal:/usr/share/gridengine/default/common: Function not implemented\r\nexportfs: gridengine-compute002.c.project.internal:/usr/share/gridengine/default/common: Function not implemented\r\nexportfs: gridengine-frontend001.c.project.internal:/usr/share/gridengine/default/common: Function not implemented\r\nexportfs: gridengine-compute001.c.project.internal:/home: Function not implemented\r\nexportfs: gridengine-compute002.c.project.internal:/home: Function not implemented\r\n```\r\n\r\nI manually restarted the nfs-kernel-server:\r\n```\r\nmbookman@frontend001:~$ sudo /etc/init.d/nfs-kernel-server restart\r\n[ ok ] Restarting nfs-kernel-server (via systemctl): nfs-kernel-server.service.\r\n```\r\n\r\nand the exports look good.\r\n\r\n```\r\nmbookman@frontend001:~$ sudo showmount -e localhost\r\nExport list for localhost:\r\n/usr/share/gridengine/default/common gridengine-frontend001.c.project.internal,gridengine-compute002.c.project.internal,gridengine-compute001.c.project.internal\r\n/home                                gridengine-compute002.c.project.internal,gridengine-compute001.c.project.internal\r\n```\r\n\r\nWhen I then re-run `elasticluster setup gridengine`, the cluster comes up."},"comment":{"url":"https://api.github.com/repos/gc3-uzh-ch/elasticluster/issues/comments/265050523","html_url":"https://github.com/gc3-uzh-ch/elasticluster/issues/357#issuecomment-265050523","issue_url":"https://api.github.com/repos/gc3-uzh-ch/elasticluster/issues/357","id":265050523,"user":{"login":"riccardomurri","id":447228,"avatar_url":"https://avatars.githubusercontent.com/u/447228?v=3","gravatar_id":"","url":"https://api.github.com/users/riccardomurri","html_url":"https://github.com/riccardomurri","followers_url":"https://api.github.com/users/riccardomurri/followers","following_url":"https://api.github.com/users/riccardomurri/following{/other_user}","gists_url":"https://api.github.com/users/riccardomurri/gists{/gist_id}","starred_url":"https://api.github.com/users/riccardomurri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/riccardomurri/subscriptions","organizations_url":"https://api.github.com/users/riccardomurri/orgs","repos_url":"https://api.github.com/users/riccardomurri/repos","events_url":"https://api.github.com/users/riccardomurri/events{/privacy}","received_events_url":"https://api.github.com/users/riccardomurri/received_events","type":"User","site_admin":false},"created_at":"2016-12-06T03:24:49Z","updated_at":"2016-12-06T03:24:49Z","body":"I think I have a working playbook in\nbranch `issues/#357` on my fork\nhttps://github.com/riccardomurri/elasticluster\n\nCan you please try it and tell me whether it also works for you?\n\nThe root cause of the issue seems to be a malfunctioning in Debian's\n`nfs-kernel-server` systemd service: it is marked with \"Active\" state\nalthough the server is *not* started, and everything just snowballs from\nthere.  Forcing the server to *restart* apparently fixes the issue.\nSome more investigation will be needed to understand whether there is\na better/minimal fix.\n"}}
{"actor":{"display_login":"riccardomurri"}}