{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/45955","repository_url":"https://api.github.com/repos/saltstack/salt","labels_url":"https://api.github.com/repos/saltstack/salt/issues/45955/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/45955/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/45955/events","html_url":"https://github.com/saltstack/salt/issues/45955","id":295959436,"number":45955,"title":"get_minion_data returns None because cache file missing","user":{"login":"grobinson-blockchain","id":13755880,"avatar_url":"https://avatars2.githubusercontent.com/u/13755880?v=4","gravatar_id":"","url":"https://api.github.com/users/grobinson-blockchain","html_url":"https://github.com/grobinson-blockchain","followers_url":"https://api.github.com/users/grobinson-blockchain/followers","following_url":"https://api.github.com/users/grobinson-blockchain/following{/other_user}","gists_url":"https://api.github.com/users/grobinson-blockchain/gists{/gist_id}","starred_url":"https://api.github.com/users/grobinson-blockchain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grobinson-blockchain/subscriptions","organizations_url":"https://api.github.com/users/grobinson-blockchain/orgs","repos_url":"https://api.github.com/users/grobinson-blockchain/repos","events_url":"https://api.github.com/users/grobinson-blockchain/events{/privacy}","received_events_url":"https://api.github.com/users/grobinson-blockchain/received_events","type":"User","site_admin":false},"labels":[{"id":71248262,"url":"https://api.github.com/repos/saltstack/salt/labels/Info%20Needed","name":"Info Needed","color":"c7def8","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/37","html_url":"https://github.com/saltstack/salt/milestone/37","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/37/labels","id":343741,"number":37,"title":"Blocked","description":"Issues which are blocked in some way, whether they need further discussion, clarification, justification, or are blocked for some other reason.","creator":{"login":"basepi","id":702318,"avatar_url":"https://avatars2.githubusercontent.com/u/702318?v=4","gravatar_id":"","url":"https://api.github.com/users/basepi","html_url":"https://github.com/basepi","followers_url":"https://api.github.com/users/basepi/followers","following_url":"https://api.github.com/users/basepi/following{/other_user}","gists_url":"https://api.github.com/users/basepi/gists{/gist_id}","starred_url":"https://api.github.com/users/basepi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basepi/subscriptions","organizations_url":"https://api.github.com/users/basepi/orgs","repos_url":"https://api.github.com/users/basepi/repos","events_url":"https://api.github.com/users/basepi/events{/privacy}","received_events_url":"https://api.github.com/users/basepi/received_events","type":"User","site_admin":false},"open_issues":1028,"closed_issues":1974,"state":"open","created_at":"2013-05-28T22:24:12Z","updated_at":"2018-02-12T15:29:02Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2018-02-09T18:07:23Z","updated_at":"2018-02-12T15:31:28Z","closed_at":null,"author_association":"NONE","body":"### Description of Issue/Question\r\n\r\nThe function [`get_minion_data`](https://github.com/saltstack/salt/blob/9a0ec9c30f65338dda31613516ac8a6121dfb0cb/salt/utils/minions.py#L69) returns `None` when the cache file for the minion is missing:\r\n\r\n    [salt.loaded.int.cache.localfs][DEBUG   ][17503] Cache file \"/var/cache/salt/master/minions/$MINION_ID.p\" does not exist\r\n\r\nwhich causes other functions, such as [`get_policies`](https://github.com/saltstack/salt/blob/5795201ad60fe33540c6f14efddeacc405bfdc57/salt/runners/vault.py#L134), to fail with the following warning:\r\n\r\n    [salt.loaded.int.runners.vault][WARNING ][17503] Could not resolve policy pattern saltstack/roles/{grains[roles]}\r\n\r\n`get_policies` fails to expand the pattern `saltstack/roles/{grains[roles]}` because \r\n [`salt.utils.minions.get_minion_data(minion_id, opts)`](https://github.com/saltstack/salt/blob/5795201ad60fe33540c6f14efddeacc405bfdc57/salt/runners/vault.py#L138) can return None.\r\n\r\nThis means that the vault module cannot create a token with all the correct policies as the expansion fails and therefore all pillars with `sdb://` calls fail to render.\r\n\r\nThis happens when running `state.apply` from either the master and the minion, however, it is not easily reproducible. We can have 2-3 successive runs of `state.apply` that run without failure, and then the next run will fail because of the missing cache file.\r\n\r\n### Questions\r\n\r\n1. Under what circumstances can the cache file be missing? What would cause the cache file to be removed between successive runs of `state.apply`?\r\n\r\n2. Why is it not possible to load grains if [`minion_data_cache`](https://github.com/saltstack/salt/blob/9a0ec9c30f65338dda31613516ac8a6121dfb0cb/salt/utils/minions.py#L78) is False?\r\n\r\n3. How come [`cache.fetch`](https://github.com/saltstack/salt/blob/9a0ec9c30f65338dda31613516ac8a6121dfb0cb/salt/utils/minions.py#L86) doesn't throw an exception if the cache file is missing?\r\n\r\n### Important information\r\n\r\nWe are running a multi-master set up in our production env. This issue does not happen in our staging or dev env where we have single masters. \r\n\r\n### Versions Report\r\n\r\n    Salt Version:\r\n               Salt: 2017.7.2\r\n\r\n    Dependency Versions:\r\n               cffi: 1.7.0\r\n           cherrypy: Not Installed\r\n           dateutil: 2.5.3\r\n          docker-py: Not Installed\r\n              gitdb: 2.0.0\r\n          gitpython: 2.1.1\r\n              ioflo: Not Installed\r\n             Jinja2: 2.9.4\r\n            libgit2: 0.24.0\r\n            libnacl: 1.4.5\r\n           M2Crypto: Not Installed\r\n               Mako: Not Installed\r\n       msgpack-pure: Not Installed\r\n     msgpack-python: 0.4.8\r\n       mysql-python: 1.3.7\r\n          pycparser: 2.14\r\n           pycrypto: 2.6.1\r\n       pycryptodome: Not Installed\r\n             pygit2: 0.24.1\r\n             Python: 2.7.13 (default, Nov 24 2017, 17:33:09)\r\n       python-gnupg: Not Installed\r\n             PyYAML: 3.12\r\n              PyZMQ: 16.0.2\r\n               RAET: Not Installed\r\n              smmap: 2.0.1\r\n            timelib: Not Installed\r\n            Tornado: 4.4.3\r\n                ZMQ: 4.2.1\r\n\r\n    System Versions:\r\n               dist: debian 9.3\r\n             locale: UTF-8\r\n            machine: x86_64\r\n            release: 4.9.0-5-amd64\r\n             system: Linux\r\n            version: debian 9.3\r\n"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/364957976","html_url":"https://github.com/saltstack/salt/issues/45955#issuecomment-364957976","issue_url":"https://api.github.com/repos/saltstack/salt/issues/45955","id":364957976,"user":{"login":"grobinson-blockchain","id":13755880,"avatar_url":"https://avatars2.githubusercontent.com/u/13755880?v=4","gravatar_id":"","url":"https://api.github.com/users/grobinson-blockchain","html_url":"https://github.com/grobinson-blockchain","followers_url":"https://api.github.com/users/grobinson-blockchain/followers","following_url":"https://api.github.com/users/grobinson-blockchain/following{/other_user}","gists_url":"https://api.github.com/users/grobinson-blockchain/gists{/gist_id}","starred_url":"https://api.github.com/users/grobinson-blockchain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grobinson-blockchain/subscriptions","organizations_url":"https://api.github.com/users/grobinson-blockchain/orgs","repos_url":"https://api.github.com/users/grobinson-blockchain/repos","events_url":"https://api.github.com/users/grobinson-blockchain/events{/privacy}","received_events_url":"https://api.github.com/users/grobinson-blockchain/received_events","type":"User","site_admin":false},"created_at":"2018-02-12T15:31:28Z","updated_at":"2018-02-12T15:31:28Z","author_association":"NONE","body":"We have a multi-master set up as explained here: https://docs.saltstack.com/en/latest/topics/tutorials/multimaster.html\r\n\r\n> If the minion_data_cache is false, then the grains are not cached, so it can't fetch them\r\n\r\nI thought so too. However, in all cases `minion_data_cache` was True, as I inserted additional logging while debugging the issue.\r\n\r\nWe don't use an external, shared cache, rather each master has its own cache. Could this be the issue?"}}
{"actor":{"display_login":"grobinson-blockchain"}}