{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27004","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27004/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27004/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27004/events","html_url":"https://github.com/kubernetes/kubernetes/issues/27004","id":159040838,"number":27004,"title":"Race condition of Informer and DeltaFIFO ","user":{"login":"hongchaodeng","id":920884,"avatar_url":"https://avatars.githubusercontent.com/u/920884?v=3","gravatar_id":"","url":"https://api.github.com/users/hongchaodeng","html_url":"https://github.com/hongchaodeng","followers_url":"https://api.github.com/users/hongchaodeng/followers","following_url":"https://api.github.com/users/hongchaodeng/following{/other_user}","gists_url":"https://api.github.com/users/hongchaodeng/gists{/gist_id}","starred_url":"https://api.github.com/users/hongchaodeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hongchaodeng/subscriptions","organizations_url":"https://api.github.com/users/hongchaodeng/orgs","repos_url":"https://api.github.com/users/hongchaodeng/repos","events_url":"https://api.github.com/users/hongchaodeng/events{/privacy}","received_events_url":"https://api.github.com/users/hongchaodeng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-06-07T22:37:47Z","updated_at":"2016-06-07T22:39:13Z","closed_at":null,"body":"### What's the issue?\r\nInformer could end up in wrong state.\r\n\r\n### Example?\r\nI have been stress testing RC of 100 replicas. I kept deleting the pod and let RC recover it to 100. The program will do informer.store.List() to check if there are 100 pods. But at the end of day it returned more than 100 (e.g. 131), while listing directly from apiserver returned 100.\r\n\r\n### Why?\r\nInformer has a queue and a store (aka indexer) internally.\r\n- The store relies on the queue to do blocking [get/pop](1) and and then [updates itself](2) in [Process()](1).\r\n- The queue (delta fifo) is used to queue events from watcher. For [each operation in reflector](3), e.g. Delete, we will [enqueue an event to the queue](4).\r\n- However, in DeltaFIFO.Delete(), it will [check the store](5) first to see if it needs to send a delete event. **This logic is wrong because the queue and store are run concurrently. There could be an ADD event not processed by store yet thus queue won't enqueue DELETE event.**\r\n\r\n[1]: https://github.com/kubernetes/kubernetes/blob/0d3be6a316b39a71b62d45272d339f7fa9162165/pkg/controller/framework/controller.go#L127-L128\r\n[2]: https://github.com/kubernetes/kubernetes/blob/0d3be6a316b39a71b62d45272d339f7fa9162165/pkg/controller/framework/shared_informer.go#L200-L224\r\n[3]: https://github.com/kubernetes/kubernetes/blob/0d3be6a316b39a71b62d45272d339f7fa9162165/pkg/client/cache/reflector.go#L384-L392\r\n[4]: https://github.com/kubernetes/kubernetes/blob/0d3be6a316b39a71b62d45272d339f7fa9162165/pkg/client/cache/delta_fifo.go#L200\r\n[5]: https://github.com/kubernetes/kubernetes/blob/0d3be6a316b39a71b62d45272d339f7fa9162165/pkg/client/cache/delta_fifo.go#L192-L198"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/224436014","html_url":"https://github.com/kubernetes/kubernetes/issues/27004#issuecomment-224436014","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27004","id":224436014,"user":{"login":"hongchaodeng","id":920884,"avatar_url":"https://avatars.githubusercontent.com/u/920884?v=3","gravatar_id":"","url":"https://api.github.com/users/hongchaodeng","html_url":"https://github.com/hongchaodeng","followers_url":"https://api.github.com/users/hongchaodeng/followers","following_url":"https://api.github.com/users/hongchaodeng/following{/other_user}","gists_url":"https://api.github.com/users/hongchaodeng/gists{/gist_id}","starred_url":"https://api.github.com/users/hongchaodeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hongchaodeng/subscriptions","organizations_url":"https://api.github.com/users/hongchaodeng/orgs","repos_url":"https://api.github.com/users/hongchaodeng/repos","events_url":"https://api.github.com/users/hongchaodeng/events{/privacy}","received_events_url":"https://api.github.com/users/hongchaodeng/received_events","type":"User","site_admin":false},"created_at":"2016-06-07T22:39:13Z","updated_at":"2016-06-07T22:39:13Z","body":"@kubernetes/sig-api-machinery @kubernetes/sig-scalability @xiang90 @philips "}}
{"actor":{"display_login":"hongchaodeng"}}