{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/pai/issues/1665","repository_url":"https://api.github.com/repos/Microsoft/pai","labels_url":"https://api.github.com/repos/Microsoft/pai/issues/1665/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/pai/issues/1665/comments","events_url":"https://api.github.com/repos/Microsoft/pai/issues/1665/events","html_url":"https://github.com/Microsoft/pai/issues/1665","id":377354324,"node_id":"MDU6SXNzdWUzNzczNTQzMjQ=","number":1665,"title":"sed: cannot rename /etc/sedLNnnDc: Device or resource busy","user":{"login":"leonobrien","id":25426571,"node_id":"MDQ6VXNlcjI1NDI2NTcx","avatar_url":"https://avatars1.githubusercontent.com/u/25426571?v=4","gravatar_id":"","url":"https://api.github.com/users/leonobrien","html_url":"https://github.com/leonobrien","followers_url":"https://api.github.com/users/leonobrien/followers","following_url":"https://api.github.com/users/leonobrien/following{/other_user}","gists_url":"https://api.github.com/users/leonobrien/gists{/gist_id}","starred_url":"https://api.github.com/users/leonobrien/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leonobrien/subscriptions","organizations_url":"https://api.github.com/users/leonobrien/orgs","repos_url":"https://api.github.com/users/leonobrien/repos","events_url":"https://api.github.com/users/leonobrien/events{/privacy}","received_events_url":"https://api.github.com/users/leonobrien/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"ydye","id":8675883,"node_id":"MDQ6VXNlcjg2NzU4ODM=","avatar_url":"https://avatars2.githubusercontent.com/u/8675883?v=4","gravatar_id":"","url":"https://api.github.com/users/ydye","html_url":"https://github.com/ydye","followers_url":"https://api.github.com/users/ydye/followers","following_url":"https://api.github.com/users/ydye/following{/other_user}","gists_url":"https://api.github.com/users/ydye/gists{/gist_id}","starred_url":"https://api.github.com/users/ydye/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ydye/subscriptions","organizations_url":"https://api.github.com/users/ydye/orgs","repos_url":"https://api.github.com/users/ydye/repos","events_url":"https://api.github.com/users/ydye/events{/privacy}","received_events_url":"https://api.github.com/users/ydye/received_events","type":"User","site_admin":false},"assignees":[{"login":"ydye","id":8675883,"node_id":"MDQ6VXNlcjg2NzU4ODM=","avatar_url":"https://avatars2.githubusercontent.com/u/8675883?v=4","gravatar_id":"","url":"https://api.github.com/users/ydye","html_url":"https://github.com/ydye","followers_url":"https://api.github.com/users/ydye/followers","following_url":"https://api.github.com/users/ydye/following{/other_user}","gists_url":"https://api.github.com/users/ydye/gists{/gist_id}","starred_url":"https://api.github.com/users/ydye/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ydye/subscriptions","organizations_url":"https://api.github.com/users/ydye/orgs","repos_url":"https://api.github.com/users/ydye/repos","events_url":"https://api.github.com/users/ydye/events{/privacy}","received_events_url":"https://api.github.com/users/ydye/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-11-05T11:17:36Z","updated_at":"2018-11-09T09:35:07Z","closed_at":null,"author_association":"NONE","body":"Hi, i have big issues trying to complete cluster configuration on docker.\r\n\r\nUsing pai release v0.8.y\r\ndocker version 17.06.2-ce\r\n\r\nI have 5 different docker containers, Dev-Box, Master and three Member nodes. \r\n\r\nMy quick-start.yaml looks like;\r\n```\r\nmachines:\r\n  - 172.18.0.3\r\n  - 172.18.0.4\r\n  - 172.18.0.5\r\n  - 172.18.0.6\r\n\r\nssh-username: pai\r\nssh-password: pai-password\r\n\r\nssh-keyfile-path: /root/.ssh/id_rsa\r\nssh-secret-name: pai-password\r\n```\r\n\r\nGenerating configuration works as expected, the Dev box can access all nodes via ssh\r\n```\r\npython paictl.py config generate -i ./deployment/quick-start/quick-start-example.yaml -o /cluster-configuration -f\r\n/usr/local/lib/python2.7/dist-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown. warnings.warn(warning, RequestsDependencyWarning)\r\n2018-11-05 10:38:56,366 [INFO] - paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2) /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: FutureWarning: CTR mode needs counter parameter, not IV self._cipher = factory.new(key, *args, **kwargs)\r\n2018-11-05 10:38:56,562 [INFO] - paramiko.transport : Authentication (publickey) successful!\r\n2018-11-05 10:38:56,649 [INFO] - deployment.k8sPaiLibrary.maintainlib.common : Executing the command on host [172.18.0.3]: cat /etc/resolv.conf | grep nameserver | cut -d ' ' -f 2 | head -n 1 127.0.0.11\r\n2018-11-05 10:38:56,664 [INFO] - paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2)\r\n2018-11-05 10:38:56,860 [INFO] - paramiko.transport : Authentication (publickey) successful!\r\n2018-11-05 10:38:56,947 [INFO] - deployment.k8sPaiLibrary.maintainlib.common : Executing the command on host [172.18.0.3]: hostname 48cff661a0ae\r\n2018-11-05 10:38:56,962 [INFO] - paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2)\r\n2018-11-05 10:38:57,156 [INFO] - paramiko.transport : Authentication (publickey) successful!\r\n2018-11-05 10:38:57,242 [INFO] - deployment.k8sPaiLibrary.maintainlib.common : Executing the command on host [172.18.0.4]: hostname 9ca63035a91b\r\n2018-11-05 10:38:57,257 [INFO] - paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2)\r\n2018-11-05 10:38:57,528 [INFO] - paramiko.transport : Authentication (publickey) successful!\r\n2018-11-05 10:38:57,604 [INFO] - deployment.k8sPaiLibrary.maintainlib.common : Executing the command on host [172.18.0.5]: hostname 67807992a189\r\n2018-11-05 10:38:57,641 [INFO] - paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2)\r\n2018-11-05 10:38:57,884 [INFO] - paramiko.transport : Authentication (publickey) successful!\r\n2018-11-05 10:38:57,958 [INFO] - deployment.k8sPaiLibrary.maintainlib.common : Executing the command on host [172.18.0.6]: hostname f8baef49f2c8\r\n```\r\nBut starting the cluster causes an error;\r\n```\r\npython paictl.py cluster k8s-bootup -p /cluster-configuration\r\n\r\n/usr/local/lib/python2.7/dist-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.\r\n  warnings.warn(warning, RequestsDependencyWarning)\r\n2018-11-05 10:47:36,517 [INFO] - __main__ : Begin to initialize PAI k8s cluster.\r\n2018-11-05 10:47:36,574 [WARNING] - deployment.k8sPaiLibrary.maintainlib.deploy : Begin to deploy a new cluster to your machine or vm.\r\n2018-11-05 10:47:36,575 [INFO] - deployment.k8sPaiLibrary.maintainlib.deploy : Begin to deploy k8s on host 172.18.0.3, the node role is [ master ]\r\n2018-11-05 10:47:36,644 [INFO] - paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2)\r\n/usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: FutureWarning: CTR mode needs counter parameter, not IV\r\n  self._cipher = factory.new(key, *args, **kwargs)\r\n2018-11-05 10:47:36,842 [INFO] - paramiko.transport : Authentication (publickey) successful!\r\n2018-11-05 10:47:36,928 [INFO] - deployment.k8sPaiLibrary.maintainlib.common : Executing the command on host [172.18.0.3]: getent passwd pai | cut -d: -f6\r\n/home/pai\r\n2018-11-05 10:47:36,963 [INFO] - paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2)\r\n2018-11-05 10:47:37,161 [INFO] - paramiko.transport : Authentication (publickey) successful!\r\npai-password\r\n2018-11-05 10:47:37,277 [INFO] - paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2)\r\n2018-11-05 10:47:37,446 [INFO] - paramiko.transport : Authentication (password) successful!\r\n2018-11-05 10:47:37,547 [INFO] - paramiko.transport.sftp : [chan 0] Opened sftp connection (server version 3)\r\n2018-11-05 10:47:37,552 [INFO] - paramiko.transport.sftp : [chan 0] sftp session closed.\r\n2018-11-05 10:47:37,564 [INFO] - paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2)\r\n2018-11-05 10:47:37,760 [INFO] - paramiko.transport : Authentication (publickey) successful!\r\n2018-11-05 10:47:37,847 [INFO] - deployment.k8sPaiLibrary.maintainlib.common : Executing the command on host [172.18.0.3]: tar -xvf master-deployment.tar\r\nmaster-deployment/docker-daemon.json\r\nmaster-deployment/docker-ce-install.sh\r\nmaster-deployment/docker-config-update.py\r\nmaster-deployment/kubelet.sh\r\nmaster-deployment/hosts-check.sh\r\nmaster-deployment/kubelet-start.sh\r\nmaster-deployment/etc/kubernetes/config\r\nmaster-deployment/etc/kubernetes/manifests/scheduler.yaml\r\nmaster-deployment/etc/kubernetes/manifests/controller-manager.yaml\r\nmaster-deployment/etc/kubernetes/manifests/apiserver.yaml\r\nmaster-deployment/etc/kubernetes/manifests/etcd.yaml\r\n2018-11-05 10:47:37,880 [INFO] - paramiko.transport : Connected (version 2.0, client OpenSSH_7.2p2)\r\n2018-11-05 10:47:38,075 [INFO] - paramiko.transport : Authentication (publickey) successful!\r\n2018-11-05 10:47:38,169 [INFO] - deployment.k8sPaiLibrary.maintainlib.common : Executing the command on host [172.18.0.3]: sudo ./master-deployment/hosts-check.sh 172.18.0.3\r\npai-password\r\nsed: cannot rename /etc/sedlXim0b: Device or resource busy\r\nsed: cannot rename /etc/sedPAzZKb: Device or resource busy\r\nsed: cannot rename /etc/sedAC3fMc: Device or resource busy\r\nsed: cannot rename /etc/sedLNnnDc: Device or resource busy\r\n\r\n```\r\n\r\nSo looking at the script on the cluster master it is trying to replace the localhost address in /etc/hosts file to the hostname of the container.\r\n\r\n```\r\n# set host ip\r\nhostip=$1\r\n\r\n# Change 127.0.0.1 line to \"127.0.0.1 localhost $hostname\"\r\ngrep -q \"127.0.0.1\" /etc/hosts && \\\r\n    sed -i \"/127.0.0.1/c\\127.0.0.1 localhost $(hostname)\" /etc/hosts || \\\r\n    sed -i \"\\$a127.0.0.1 localhost $(hostname)\" /etc/hosts\r\n\r\n# Comment 127.0.1.1 line\r\nsed -i \"/127.0.1.1/s/^/# /\" /etc/hosts\r\n\r\n# Comment hostip line\r\nsed -i \"/$hostip/s/^/# /\" /etc/hosts\r\n\r\n```\r\n\r\nSo I have tried many times and search widely to try to sort this help. I am assuming of course that each node of the cluster (Master and Member nodes) can be docker containers too.\r\n\r\nAny help would be very appreciated.\r\n"},"comment":{"url":"https://api.github.com/repos/Microsoft/pai/issues/comments/437303447","html_url":"https://github.com/Microsoft/pai/issues/1665#issuecomment-437303447","issue_url":"https://api.github.com/repos/Microsoft/pai/issues/1665","id":437303447,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzMwMzQ0Nw==","user":{"login":"ydye","id":8675883,"node_id":"MDQ6VXNlcjg2NzU4ODM=","avatar_url":"https://avatars2.githubusercontent.com/u/8675883?v=4","gravatar_id":"","url":"https://api.github.com/users/ydye","html_url":"https://github.com/ydye","followers_url":"https://api.github.com/users/ydye/followers","following_url":"https://api.github.com/users/ydye/following{/other_user}","gists_url":"https://api.github.com/users/ydye/gists{/gist_id}","starred_url":"https://api.github.com/users/ydye/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ydye/subscriptions","organizations_url":"https://api.github.com/users/ydye/orgs","repos_url":"https://api.github.com/users/ydye/repos","events_url":"https://api.github.com/users/ydye/events{/privacy}","received_events_url":"https://api.github.com/users/ydye/received_events","type":"User","site_admin":false},"created_at":"2018-11-09T09:35:07Z","updated_at":"2018-11-09T09:35:07Z","author_association":"CONTRIBUTOR","body":"Hi @leonobrien \r\n\r\nYou'd better not run the ```dev-box``` container on the physical machine where you wanna deploy pai. And [here](#dev_box_k8s) is the answer. [Following](#deploy_pai) are 2 way to deploy OpenPai. And if you wanna achieve the DinD (Docker in Docker), that means running container in a container. Maybe you should refer to this [link](https://hub.docker.com/_/docker/) first. But we haven't test this scenario before.\r\n\r\n\r\n#### Why dev-box shouldn't be on the same physical machine  (Or the node runs docker-daemon?) of kubernetes, when deploy kubernetes by OpenPai. <a name=\"dev_box_k8s\"></a>\r\nThis [script](https://github.com/Microsoft/pai/blob/master/deployment/k8sPaiLibrary/maintaintool/docker-ce-install.sh) is responsible for installing docker-ce and enable log-rotation. So the docker-daemon will be restart. If the ```dev-box``` container is running on one of the node \r\nThe reason to enable log-ration is because that hadoop will ouput a large amount of log which may run out of the disk space.\r\n\r\n#### Two different way. <a name=\"deploy_pai\"></a>\r\n###### Single-box\r\nIn this mode, you only need to deploy pai on the same machine. Those machine should be labeled with ```\r\npai-master = \"true\"\r\npai-worker = \"true\"\r\nk8s-role = master\r\n```\r\n\r\nWhen deploying pai, please install the following dependency package on your host environment (Not in the container) and then instal OpenPai on the host environment.\r\n\r\n```\r\nRUN apt-get -y update && \\\r\n    apt-get -y install \\\r\n      nano \\\r\n      vim \\\r\n      joe \\\r\n      wget \\\r\n      curl \\\r\n      jq \\\r\n      gawk \\\r\n      psmisc \\\r\n      python \\\r\n      python-yaml \\\r\n      python-jinja2 \\\r\n      python-paramiko \\\r\n      python-urllib3 \\\r\n      python-tz \\\r\n      python-nose \\\r\n      python-prettytable \\\r\n      python-netifaces \\\r\n      python-dev \\\r\n      python-pip \\\r\n      python-mysqldb \\\r\n      openjdk-8-jre \\\r\n      openjdk-8-jdk \\\r\n      openssh-server \\\r\n      openssh-client \\\r\n      git \\\r\n      bash-completion \\\r\n      inotify-tools \\\r\n      rsync \\\r\n      realpath \\\r\n      net-tools && \\\r\n    mkdir -p /cluster-configuration &&\\\r\n    git clone https://github.com/Microsoft/pai.git &&\\\r\n    pip install python-etcd docker kubernetes GitPython\r\n``` \r\n\r\nMore detailed information, please refer to the [link](https://github.com/Microsoft/pai/blob/master/docs/pai-management/doc/cluster-bootup.md#singlebox)\r\n\r\n###### distributed mode.\r\n\r\nIn this mode, you should have at least 2 node. And more detailed information, please refer to this [Link](https://github.com/Microsoft/pai/blob/master/docs/pai-management/doc/cluster-bootup.md#customizeddeploy)\r\n\r\n\r\n\r\n\r\n\r\n\r\n \r\n\r\n"}}
{"actor":{"display_login":"ydye"}}