{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/29239","repository_url":"https://api.github.com/repos/saltstack/salt","labels_url":"https://api.github.com/repos/saltstack/salt/issues/29239/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/29239/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/29239/events","html_url":"https://github.com/saltstack/salt/issues/29239","id":119115263,"number":29239,"title":"Occasionaly git_pillar pull fails causing incorrect results of highstate (when running highstate for multiple minions)","user":{"login":"timwsuqld","id":6201617,"avatar_url":"https://avatars.githubusercontent.com/u/6201617?v=3","gravatar_id":"","url":"https://api.github.com/users/timwsuqld","html_url":"https://github.com/timwsuqld","followers_url":"https://api.github.com/users/timwsuqld/followers","following_url":"https://api.github.com/users/timwsuqld/following{/other_user}","gists_url":"https://api.github.com/users/timwsuqld/gists{/gist_id}","starred_url":"https://api.github.com/users/timwsuqld/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timwsuqld/subscriptions","organizations_url":"https://api.github.com/users/timwsuqld/orgs","repos_url":"https://api.github.com/users/timwsuqld/repos","events_url":"https://api.github.com/users/timwsuqld/events{/privacy}","received_events_url":"https://api.github.com/users/timwsuqld/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Bug","name":"Bug","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/High%20Severity","name":"High Severity","color":"ff9143"},{"url":"https://api.github.com/repos/saltstack/salt/labels/P1","name":"P1","color":"2181ee"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Pillar","name":"Pillar","color":"006b75"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Platform","name":"Platform","color":"fef2c0"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/13","html_url":"https://github.com/saltstack/salt/milestones/Approved","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/13/labels","id":9265,"number":13,"title":"Approved","description":"All issues that are ready to be worked on, both bugs and features.","creator":{"login":"thatch45","id":507599,"avatar_url":"https://avatars.githubusercontent.com/u/507599?v=3","gravatar_id":"","url":"https://api.github.com/users/thatch45","html_url":"https://github.com/thatch45","followers_url":"https://api.github.com/users/thatch45/followers","following_url":"https://api.github.com/users/thatch45/following{/other_user}","gists_url":"https://api.github.com/users/thatch45/gists{/gist_id}","starred_url":"https://api.github.com/users/thatch45/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatch45/subscriptions","organizations_url":"https://api.github.com/users/thatch45/orgs","repos_url":"https://api.github.com/users/thatch45/repos","events_url":"https://api.github.com/users/thatch45/events{/privacy}","received_events_url":"https://api.github.com/users/thatch45/received_events","type":"User","site_admin":false},"open_issues":2931,"closed_issues":3413,"state":"open","created_at":"2011-05-14T04:00:56Z","updated_at":"2016-03-08T18:32:42Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2015-11-27T01:02:45Z","updated_at":"2016-03-08T19:38:55Z","closed_at":null,"body":"When running state.highstate for a single minion, everything works fine.\r\nWhen running state.highstate for all minions (5), it sometimes gives incorrect results. (All highstate commands are being run with test=True)\r\nDigging down, it appears that for some minions, the git_pillar fails to update, and so the pillar data for that minion is empty, causing the states to give the wrong output. Ideally if the git_pillar (ext_pillar) fails it shouldn't try and compile states for the minion, as the data is incorrect. I'm also not sure why the pillar data appears to be empty instead of using the last successful pull.\r\n\r\nSome of the workarounds that I've seen involve just using Cron to pull the pillars repo, and then pointing at that. This would probably speed things up, but I'd expect salt to already do that.\r\n\r\nLines such as the following appear in the logs when this occurs\r\n```\r\n2015-11-27 10:12:18,883 [salt.utils.gitfs ][ERROR   ][26560] Failed to checkout master from git_pillar remote 'master git@version-control:it-internal/saltstack-config.git': aabb80c2f754bdcb6a9100c16445a05c4858c309: The index is locked. This might be due to a concurrent or crashed process\r\n2015-11-27 10:38:09,410 [salt.utils.gitfs ][ERROR   ][26556] Failed to checkout master from git_pillar remote 'master git@version-control:it-internal/saltstack-config.git': Failed to create locked file '/var/cache/salt/master/git_pillar/5e3205db799031016a50dbe438df411c/.git/index.lock': File exists\r\n```\r\n\r\nMy understanding of https://github.com/saltstack/salt/pull/22962 and https://github.com/saltstack/salt/issues/19994 is that this should have been fixed in 2015.8.0. Maybe this is related, maybe not.\r\n\r\nRunning on Centos 7 with 4Gb of RAM\r\n```\r\n$ salt --version\r\nsalt 2015.8.1 (Beryllium)\r\n```"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/193936757","html_url":"https://github.com/saltstack/salt/issues/29239#issuecomment-193936757","issue_url":"https://api.github.com/repos/saltstack/salt/issues/29239","id":193936757,"user":{"login":"aabognah","id":3789988,"avatar_url":"https://avatars.githubusercontent.com/u/3789988?v=3","gravatar_id":"","url":"https://api.github.com/users/aabognah","html_url":"https://github.com/aabognah","followers_url":"https://api.github.com/users/aabognah/followers","following_url":"https://api.github.com/users/aabognah/following{/other_user}","gists_url":"https://api.github.com/users/aabognah/gists{/gist_id}","starred_url":"https://api.github.com/users/aabognah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aabognah/subscriptions","organizations_url":"https://api.github.com/users/aabognah/orgs","repos_url":"https://api.github.com/users/aabognah/repos","events_url":"https://api.github.com/users/aabognah/events{/privacy}","received_events_url":"https://api.github.com/users/aabognah/received_events","type":"User","site_admin":false},"created_at":"2016-03-08T19:38:54Z","updated_at":"2016-03-08T19:38:54Z","body":"Hi,\r\n\r\nI am seeing the same problem where the wrong set of data (old data or no data at all) is returned by git_pillar. I am using pillar data to template sudoers file and this is resulting is corrupted sudoers files. \r\n\r\nI traced the error in the log file with debug level, and I see the following error:\r\n\r\n    2016-03-08 14:08:54,109 [salt.utils.gitfs ][ERROR   ][5582] Failed to checkout master from git_pillar remote 'master gitlab@don.private.uwaterloo.ca:ist-tis-sas/salt-root.git': remote ref does not exist\r\n    2016-03-08 14:08:54,119 [salt.loaded.int.pillar.git_pillar][DEBUG   ][5591] git_pillar is processing pillar SLS from /var/cache/salt/master/git_pillar/0bac491499545b545cd9d407aa125c19/pillar/base for pillar env 'base'\r\n\r\nThe remote exists and the error seems random (maybe caused by multiple attempts to checkout the repo at the same time!).\r\n\r\nMoreover, the file in the cache is fine and the returned pillar should not be corrupted, but they are. \r\n\r\nsalt '*' pillar.item returnes the correct set of data so the problem is happening when highstate is tempting the files.\r\n\r\nsalt --versions-report\r\n```\r\nSalt Version:\r\n           Salt: 2015.8.7\r\n \r\nDependency Versions:\r\n         Jinja2: unknown\r\n       M2Crypto: 0.20.2\r\n           Mako: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 14.5.0\r\n         Python: 2.6.6 (r266:84292, May 22 2015, 08:34:51)\r\n           RAET: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n           cffi: Not Installed\r\n       cherrypy: 3.2.2\r\n       dateutil: 1.4.1\r\n          gitdb: 0.5.4\r\n      gitpython: 0.3.2 RC1\r\n          ioflo: Not Installed\r\n        libgit2: 0.20.0\r\n        libnacl: Not Installed\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: Not Installed\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: 0.20.3\r\n   python-gnupg: Not Installed\r\n          smmap: 0.8.1\r\n        timelib: Not Installed\r\n \r\nSystem Versions:\r\n           dist: redhat 6.7 Santiago\r\n        machine: x86_64\r\n        release: 2.6.32-573.12.1.el6.x86_64\r\n         system: Red Hat Enterprise Linux Server 6.7 Santiago\r\n```\r\n\r\n\r\n\r\n\r\n"}}