{"action":"closed","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/29239","repository_url":"https://api.github.com/repos/saltstack/salt","labels_url":"https://api.github.com/repos/saltstack/salt/issues/29239/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/29239/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/29239/events","html_url":"https://github.com/saltstack/salt/issues/29239","id":119115263,"number":29239,"title":"Occasionaly git_pillar pull fails causing incorrect results of highstate (when running highstate for multiple minions)","user":{"login":"timwsuqld","id":6201617,"avatar_url":"https://avatars.githubusercontent.com/u/6201617?v=3","gravatar_id":"","url":"https://api.github.com/users/timwsuqld","html_url":"https://github.com/timwsuqld","followers_url":"https://api.github.com/users/timwsuqld/followers","following_url":"https://api.github.com/users/timwsuqld/following{/other_user}","gists_url":"https://api.github.com/users/timwsuqld/gists{/gist_id}","starred_url":"https://api.github.com/users/timwsuqld/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timwsuqld/subscriptions","organizations_url":"https://api.github.com/users/timwsuqld/orgs","repos_url":"https://api.github.com/users/timwsuqld/repos","events_url":"https://api.github.com/users/timwsuqld/events{/privacy}","received_events_url":"https://api.github.com/users/timwsuqld/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Bug","name":"Bug","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Fixed%20Pending%20Verification","name":"Fixed Pending Verification","color":"d4c5f9"},{"url":"https://api.github.com/repos/saltstack/salt/labels/High%20Severity","name":"High Severity","color":"ff9143"},{"url":"https://api.github.com/repos/saltstack/salt/labels/P1","name":"P1","color":"2181ee"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Pillar","name":"Pillar","color":"006b75"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Platform","name":"Platform","color":"fef2c0"}],"state":"closed","locked":false,"assignee":{"login":"terminalmage","id":328598,"avatar_url":"https://avatars.githubusercontent.com/u/328598?v=3","gravatar_id":"","url":"https://api.github.com/users/terminalmage","html_url":"https://github.com/terminalmage","followers_url":"https://api.github.com/users/terminalmage/followers","following_url":"https://api.github.com/users/terminalmage/following{/other_user}","gists_url":"https://api.github.com/users/terminalmage/gists{/gist_id}","starred_url":"https://api.github.com/users/terminalmage/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/terminalmage/subscriptions","organizations_url":"https://api.github.com/users/terminalmage/orgs","repos_url":"https://api.github.com/users/terminalmage/repos","events_url":"https://api.github.com/users/terminalmage/events{/privacy}","received_events_url":"https://api.github.com/users/terminalmage/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/13","html_url":"https://github.com/saltstack/salt/milestones/Approved","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/13/labels","id":9265,"number":13,"title":"Approved","description":"All issues that are ready to be worked on, both bugs and features.","creator":{"login":"thatch45","id":507599,"avatar_url":"https://avatars.githubusercontent.com/u/507599?v=3","gravatar_id":"","url":"https://api.github.com/users/thatch45","html_url":"https://github.com/thatch45","followers_url":"https://api.github.com/users/thatch45/followers","following_url":"https://api.github.com/users/thatch45/following{/other_user}","gists_url":"https://api.github.com/users/thatch45/gists{/gist_id}","starred_url":"https://api.github.com/users/thatch45/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatch45/subscriptions","organizations_url":"https://api.github.com/users/thatch45/orgs","repos_url":"https://api.github.com/users/thatch45/repos","events_url":"https://api.github.com/users/thatch45/events{/privacy}","received_events_url":"https://api.github.com/users/thatch45/received_events","type":"User","site_admin":false},"open_issues":3034,"closed_issues":3545,"state":"open","created_at":"2011-05-14T04:00:56Z","updated_at":"2016-04-22T06:39:25Z","due_on":null,"closed_at":null},"comments":8,"created_at":"2015-11-27T01:02:45Z","updated_at":"2016-04-22T06:39:26Z","closed_at":"2016-04-22T06:39:25Z","body":"When running state.highstate for a single minion, everything works fine.\r\nWhen running state.highstate for all minions (5), it sometimes gives incorrect results. (All highstate commands are being run with test=True)\r\nDigging down, it appears that for some minions, the git_pillar fails to update, and so the pillar data for that minion is empty, causing the states to give the wrong output. Ideally if the git_pillar (ext_pillar) fails it shouldn't try and compile states for the minion, as the data is incorrect. I'm also not sure why the pillar data appears to be empty instead of using the last successful pull.\r\n\r\nSome of the workarounds that I've seen involve just using Cron to pull the pillars repo, and then pointing at that. This would probably speed things up, but I'd expect salt to already do that.\r\n\r\nLines such as the following appear in the logs when this occurs\r\n```\r\n2015-11-27 10:12:18,883 [salt.utils.gitfs ][ERROR   ][26560] Failed to checkout master from git_pillar remote 'master git@version-control:it-internal/saltstack-config.git': aabb80c2f754bdcb6a9100c16445a05c4858c309: The index is locked. This might be due to a concurrent or crashed process\r\n2015-11-27 10:38:09,410 [salt.utils.gitfs ][ERROR   ][26556] Failed to checkout master from git_pillar remote 'master git@version-control:it-internal/saltstack-config.git': Failed to create locked file '/var/cache/salt/master/git_pillar/5e3205db799031016a50dbe438df411c/.git/index.lock': File exists\r\n```\r\n\r\nMy understanding of https://github.com/saltstack/salt/pull/22962 and https://github.com/saltstack/salt/issues/19994 is that this should have been fixed in 2015.8.0. Maybe this is related, maybe not.\r\n\r\nRunning on Centos 7 with 4Gb of RAM\r\n```\r\n$ salt --version\r\nsalt 2015.8.1 (Beryllium)\r\n```"}}