{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt-bootstrap/issues/930","repository_url":"https://api.github.com/repos/saltstack/salt-bootstrap","labels_url":"https://api.github.com/repos/saltstack/salt-bootstrap/issues/930/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt-bootstrap/issues/930/comments","events_url":"https://api.github.com/repos/saltstack/salt-bootstrap/issues/930/events","html_url":"https://github.com/saltstack/salt-bootstrap/issues/930","id":170947766,"number":930,"title":"Fresh install, Salt seems to be working only half the time","user":{"login":"Ch3LL","id":6886822,"avatar_url":"https://avatars.githubusercontent.com/u/6886822?v=3","gravatar_id":"","url":"https://api.github.com/users/Ch3LL","html_url":"https://github.com/Ch3LL","followers_url":"https://api.github.com/users/Ch3LL/followers","following_url":"https://api.github.com/users/Ch3LL/following{/other_user}","gists_url":"https://api.github.com/users/Ch3LL/gists{/gist_id}","starred_url":"https://api.github.com/users/Ch3LL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ch3LL/subscriptions","organizations_url":"https://api.github.com/users/Ch3LL/orgs","repos_url":"https://api.github.com/users/Ch3LL/repos","events_url":"https://api.github.com/users/Ch3LL/events{/privacy}","received_events_url":"https://api.github.com/users/Ch3LL/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-08-12T19:58:15Z","updated_at":"2016-08-12T21:23:01Z","closed_at":null,"body":"@SingleMalt commented on [Thu Aug 11 2016](https://github.com/saltstack/salt/issues/35405)\n\n\r\nHi, so I'm following the Salt setup instructions (https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-salt-master-and-minion-servers-on-ubuntu-14-04), and have the Salt master and minion running on the same EC2 instance (m4.large). Salt is working only half the time and I can't figure out why.\r\n\r\n```shell\r\nroot@ip-10------:~# salt '*' test.ping\r\nip-10------:\r\n    True\r\nroot@ip-10------:~# salt '*' test.ping\r\nip-10------:\r\n    True\r\nroot@ip-10------:~# salt '*' test.ping\r\nip-10------:\r\n    True\r\nroot@ip-10------:~# salt '*' test.ping\r\nSalt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.\r\nroot@ip-10------:~# salt '*' test.ping\r\nSalt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.\r\n```\r\n\r\nHere is the Salt version reports:\r\n\r\n```shell\r\nroot@ip-10------:~# salt-minion --versions-report\r\nSalt Version:\r\n           Salt: 2016.3.2\r\n\r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: Not Installed\r\n       dateutil: 1.5\r\n          gitdb: 0.5.4\r\n      gitpython: 0.3.2 RC1\r\n          ioflo: Not Installed\r\n         Jinja2: 2.7.2\r\n        libgit2: Not Installed\r\n        libnacl: Not Installed\r\n       M2Crypto: Not Installed\r\n           Mako: 0.9.1\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: 1.2.3\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.10\r\n          PyZMQ: 14.0.1\r\n           RAET: Not Installed\r\n          smmap: 0.8.2\r\n        timelib: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n\r\nSystem Versions:\r\n           dist: Ubuntu 14.04 trusty\r\n        machine: x86_64\r\n        release: 3.13.0-91-generic\r\n         system: Linux\r\n        version: Ubuntu 14.04 trusty\r\n\r\nroot@ip-10------:~# salt --versions-report\r\nSalt Version:\r\n           Salt: 2016.3.2\r\n\r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: Not Installed\r\n       dateutil: 1.5\r\n          gitdb: 0.5.4\r\n      gitpython: 0.3.2 RC1\r\n          ioflo: Not Installed\r\n         Jinja2: 2.7.2\r\n        libgit2: Not Installed\r\n        libnacl: Not Installed\r\n       M2Crypto: Not Installed\r\n           Mako: 0.9.1\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: 1.2.3\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.10\r\n          PyZMQ: 14.0.1\r\n           RAET: Not Installed\r\n          smmap: 0.8.2\r\n        timelib: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n\r\nSystem Versions:\r\n           dist: Ubuntu 14.04 trusty\r\n        machine: x86_64\r\n        release: 3.13.0-91-generic\r\n         system: Linux\r\n        version: Ubuntu 14.04 trusty\r\n\r\nroot@ip-10------:~# salt-master --versions-report\r\nSalt Version:\r\n           Salt: 2016.3.2\r\n\r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: Not Installed\r\n       dateutil: 1.5\r\n          gitdb: 0.5.4\r\n      gitpython: 0.3.2 RC1\r\n          ioflo: Not Installed\r\n         Jinja2: 2.7.2\r\n        libgit2: Not Installed\r\n        libnacl: Not Installed\r\n       M2Crypto: Not Installed\r\n           Mako: 0.9.1\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: 1.2.3\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.10\r\n          PyZMQ: 14.0.1\r\n           RAET: Not Installed\r\n          smmap: 0.8.2\r\n        timelib: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n\r\nSystem Versions:\r\n           dist: Ubuntu 14.04 trusty\r\n        machine: x86_64\r\n        release: 3.13.0-91-generic\r\n         system: Linux\r\n        version: Ubuntu 14.04 trusty\r\n```\r\n\r\nHere is some basic information about the Ubuntu instance:\r\n\r\n```shell\r\nroot@ip-10------:~# df -h\r\nFilesystem      Size  Used Avail Use% Mounted on\r\nudev            3.9G  8.0K  3.9G   1% /dev\r\ntmpfs           799M  348K  799M   1% /run\r\n/dev/xvda1      7.8G  1.7G  5.7G  23% /\r\nnone            4.0K     0  4.0K   0% /sys/fs/cgroup\r\nnone            5.0M     0  5.0M   0% /run/lock\r\nnone            3.9G   60K  3.9G   1% /run/shm\r\nnone            100M     0  100M   0% /run/user\r\n\r\nroot@ip-10------:~# df -i\r\nFilesystem      Inodes IUsed   IFree IUse% Mounted on\r\nudev           1020715   386 1020329    1% /dev\r\ntmpfs          1021952   313 1021639    1% /run\r\n/dev/xvda1      524288 78135  446153   15% /\r\nnone           1021952     2 1021950    1% /sys/fs/cgroup\r\nnone           1021952     1 1021951    1% /run/lock\r\nnone           1021952    16 1021936    1% /run/shm\r\nnone           1021952     2 1021950    1% /run/user\r\n\r\nroot@ip-10------:~# free -m\r\n             total       used       free     shared    buffers     cached\r\nMem:          7984        728       7255          0         14         95\r\n-/+ buffers/cache:        617       7366\r\nSwap:            0          0          0\r\n\r\nroot@ip-10------:~# host salt\r\nsalt.-------.com has address 10.-.-.-\r\n```\r\n\r\nAny help would be great. Thanks!\n\n---\n\n@SingleMalt commented on [Fri Aug 12 2016](https://github.com/saltstack/salt/issues/35405#issuecomment-239512339)\n\nRunning both ```salt-master -l debug``` and ```salt-minion -l debug``` shows nothing on the calls that fail. It's like Salt just isn't getting those requests.\n\n---\n\n@SingleMalt commented on [Fri Aug 12 2016](https://github.com/saltstack/salt/issues/35405#issuecomment-239515545)\n\nVerbose output from the salt command:\r\n\r\n```shell\r\nroot@ip-10------:~# salt -v -l trace '*' test.ping\r\n[DEBUG   ] Reading configuration from /etc/salt/master\r\n[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: ip-10------\r\n[DEBUG   ] Missing configuration file: /root/.saltrc\r\n[TRACE   ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.\r\n[TRACE   ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.\r\n[DEBUG   ] Configuration file path: /etc/salt/master\r\n[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.\r\n[DEBUG   ] Reading configuration from /etc/salt/master\r\n[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: ip-10------\r\n[DEBUG   ] Missing configuration file: /root/.saltrc\r\n[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc\r\n[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc\r\n[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'ip-10------_master', 'tcp://127.0.0.1:4506', 'clear')\r\n[TRACE   ] Inserted key into loop_instance_map id 140096462065392 for key ('/etc/salt/pki/master', 'ip-10------_master', 'tcp://127.0.0.1:4506', 'clear') and process 6227\r\n[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc\r\n[TRACE   ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc\r\n[DEBUG   ] SaltReqTimeoutError, retrying. (1/3)\r\n[DEBUG   ] SaltReqTimeoutError, retrying. (2/3)\r\n[DEBUG   ] SaltReqTimeoutError, retrying. (3/3)\r\n[DEBUG   ] LazyLoaded nested.output\r\n[TRACE   ] data = Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.\r\nSalt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.```\n\n---\n\n@SingleMalt commented on [Fri Aug 12 2016](https://github.com/saltstack/salt/issues/35405#issuecomment-239520244)\n\nSeems like a ZMQ issue, seeing this on startup:\r\n\r\n```shell\r\n[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'ip-10-------', 'tcp://10.-.-.-:4506', 'clear')\r\n[DEBUG   ] SaltReqTimeoutError, retrying. (1/7)\r\n[DEBUG   ] SaltReqTimeoutError, retrying. (2/7)\r\n[DEBUG   ] SaltReqTimeoutError, retrying. (3/7)\r\n[DEBUG   ] SaltReqTimeoutError, retrying. (4/7)\r\n[DEBUG   ] SaltReqTimeoutError, retrying. (5/7)\r\n[DEBUG   ] SaltReqTimeoutError, retrying. (6/7)\r\n[DEBUG   ] SaltReqTimeoutError, retrying. (7/7)\r\n[WARNING ] Fatal functionality error caught by minion handler:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/dist-packages/salt/scripts.py\", line 85, in minion_process\r\n    minion.start()\r\n  File \"/usr/lib/python2.7/dist-packages/salt/cli/daemons.py\", line 349, in start\r\n    self.minion.tune_in()\r\n  File \"/usr/lib/python2.7/dist-packages/salt/minion.py\", line 1964, in tune_in\r\n    self.sync_connect_master()\r\n  File \"/usr/lib/python2.7/dist-packages/salt/minion.py\", line 898, in sync_connect_master\r\n    raise six.reraise(*future_exception)\r\n  File \"/usr/lib/python2.7/dist-packages/tornado/gen.py\", line 876, in run\r\n    yielded = self.gen.throw(*exc_info)\r\n  File \"/usr/lib/python2.7/dist-packages/salt/minion.py\", line 907, in connect_master\r\n    master, self.pub_channel = yield self.eval_master(self.opts, self.timeout, self.safe)\r\n  File \"/usr/lib/python2.7/dist-packages/tornado/gen.py\", line 870, in run\r\n    value = future.result()\r\n  File \"/usr/lib/python2.7/dist-packages/tornado/concurrent.py\", line 215, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"/usr/lib/python2.7/dist-packages/tornado/gen.py\", line 876, in run\r\n    yielded = self.gen.throw(*exc_info)\r\n  File \"/usr/lib/python2.7/dist-packages/salt/minion.py\", line 570, in eval_master\r\n    raise exc\r\nSaltClientError: Attempt to authenticate with the salt master failed with timeout error\r\n[WARNING ] ** Restarting minion **```\n\n---\n\n@Ch3LL commented on [Fri Aug 12 2016](https://github.com/saltstack/salt/issues/35405#issuecomment-239545282)\n\n@SingleMalt Okay I think i've pinpointed a bit of what is going on here. I can only replicate this when using the saltstack bootstrap options `sudo sh install_salt.sh -P -M`. \r\n\r\nIf I do the following it works just fine:\r\n\r\n1. sudo sh install_salt.sh -M\r\n2. Install using the official saltstack packages at repo.saltstack.com\r\n\r\nSo the only added option in this situation that is causing this issue is the `-P` argument which allows a user to install packages using the pip package manager if needed. So I'm thinking there is a pip package that is causing this issue. But i'm not entirely sure and this will need to be tracked own more. I'm going to move this issue over to the salt bootstrap repo so we can track it there. As a workaround you can just run `sudo sh install_salt.sh -M` or install salt using the instructions at repo.saltstack.com\n\n"},"comment":{"url":"https://api.github.com/repos/saltstack/salt-bootstrap/issues/comments/239563149","html_url":"https://github.com/saltstack/salt-bootstrap/issues/930#issuecomment-239563149","issue_url":"https://api.github.com/repos/saltstack/salt-bootstrap/issues/930","id":239563149,"user":{"login":"SingleMalt","id":787338,"avatar_url":"https://avatars.githubusercontent.com/u/787338?v=3","gravatar_id":"","url":"https://api.github.com/users/SingleMalt","html_url":"https://github.com/SingleMalt","followers_url":"https://api.github.com/users/SingleMalt/followers","following_url":"https://api.github.com/users/SingleMalt/following{/other_user}","gists_url":"https://api.github.com/users/SingleMalt/gists{/gist_id}","starred_url":"https://api.github.com/users/SingleMalt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SingleMalt/subscriptions","organizations_url":"https://api.github.com/users/SingleMalt/orgs","repos_url":"https://api.github.com/users/SingleMalt/repos","events_url":"https://api.github.com/users/SingleMalt/events{/privacy}","received_events_url":"https://api.github.com/users/SingleMalt/received_events","type":"User","site_admin":false},"created_at":"2016-08-12T21:23:01Z","updated_at":"2016-08-12T21:23:01Z","body":"Installed from repo.saltstack.com and everything is working. Thanks again!"}}
{"actor":{"display_login":"SingleMalt"}}