{"action":"created","issue":{"url":"https://api.github.com/repos/istio/istio/issues/10379","repository_url":"https://api.github.com/repos/istio/istio","labels_url":"https://api.github.com/repos/istio/istio/issues/10379/labels{/name}","comments_url":"https://api.github.com/repos/istio/istio/issues/10379/comments","events_url":"https://api.github.com/repos/istio/istio/issues/10379/events","html_url":"https://github.com/istio/istio/pull/10379","id":389448457,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM3NDM2Mjk1","number":10379,"title":"First round or mutex fixing","user":{"login":"costinm","id":84202,"node_id":"MDQ6VXNlcjg0MjAy","avatar_url":"https://avatars1.githubusercontent.com/u/84202?v=4","gravatar_id":"","url":"https://api.github.com/users/costinm","html_url":"https://github.com/costinm","followers_url":"https://api.github.com/users/costinm/followers","following_url":"https://api.github.com/users/costinm/following{/other_user}","gists_url":"https://api.github.com/users/costinm/gists{/gist_id}","starred_url":"https://api.github.com/users/costinm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/costinm/subscriptions","organizations_url":"https://api.github.com/users/costinm/orgs","repos_url":"https://api.github.com/users/costinm/repos","events_url":"https://api.github.com/users/costinm/events{/privacy}","received_events_url":"https://api.github.com/users/costinm/received_events","type":"User","site_admin":false},"labels":[{"id":659652351,"node_id":"MDU6TGFiZWw2NTk2NTIzNTE=","url":"https://api.github.com/repos/istio/istio/labels/approved","name":"approved","color":"bfdadc","default":false},{"id":491318011,"node_id":"MDU6TGFiZWw0OTEzMTgwMTE=","url":"https://api.github.com/repos/istio/istio/labels/cla:%20yes","name":"cla: yes","color":"bfdadc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-12-10T19:20:46Z","updated_at":"2018-12-10T19:56:49Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/istio/istio/pulls/10379","html_url":"https://github.com/istio/istio/pull/10379","diff_url":"https://github.com/istio/istio/pull/10379.diff","patch_url":"https://github.com/istio/istio/pull/10379.patch"},"body":"Smoking gun:\r\n```\r\ngoroutine 107 [semacquire, 182 minutes]:\r\nsync.(*Mutex).Lock(0x2503008)\r\n\t/usr/local/go/src/sync/mutex.go:134 +0x108\r\nistio.io/istio/pilot/pkg/proxy/envoy.(*DiscoveryService).ConfigUpdate(0xc42078c580, 0x18d4c01)\r\n\t/workspace/go/src/istio.io/istio/pilot/pkg/proxy/envoy/discovery.go:480 +0x3f\r\nistio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).WorkloadUpdate(0xc4200e8510, 0xc43390d2a0, 0xb, 0xc43d16dda0, 0xc43d16de00)\r\n\t/workspace/go/src/istio.io/istio/pilot/pkg/proxy/envoy/v2/eds.go:491 +0x1c9\r\nistio.io/istio/pilot/pkg/serviceregistry/kube.(*PodCache).event(0xc42078c500, 0x189b2c0, 0xc425850000, 0x1, 0x0, 0x0)\r\n\t/workspace/go/src/istio.io/istio/pilot/pkg/serviceregistry/kube/cache.go:97 +0x61f\r\n```\r\n\r\nThe root cause: we frequently use the pattern 'mutex.Lock()/defer mutex.Unlock(), and in many cases\r\nthis holds the lock longer than needed to protect the structures. \r\n\r\nI went over some of the other mutex uses and reduced the scope of the mutexes - there are others to review/fix, but this first PR should fix  #9043\r\n\r\nTesting wise - we need more coverage for the 'deleted pods' cases, the stability tests are focused on stable config under load, only few pods are 'flipping' (bad health). Since this bug affects k8s registry it won't be caught by the tests using the fake registry, and we moved the local apiserver unit tests to integration/e2e where we don't have the racetest.\r\n\r\n \r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/istio/istio/issues/comments/445950878","html_url":"https://github.com/istio/istio/pull/10379#issuecomment-445950878","issue_url":"https://api.github.com/repos/istio/istio/issues/10379","id":445950878,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTk1MDg3OA==","user":{"login":"codecov[bot]","id":22429695,"node_id":"MDM6Qm90MjI0Mjk2OTU=","avatar_url":"https://avatars2.githubusercontent.com/in/254?v=4","gravatar_id":"","url":"https://api.github.com/users/codecov%5Bbot%5D","html_url":"https://github.com/apps/codecov","followers_url":"https://api.github.com/users/codecov%5Bbot%5D/followers","following_url":"https://api.github.com/users/codecov%5Bbot%5D/following{/other_user}","gists_url":"https://api.github.com/users/codecov%5Bbot%5D/gists{/gist_id}","starred_url":"https://api.github.com/users/codecov%5Bbot%5D/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/codecov%5Bbot%5D/subscriptions","organizations_url":"https://api.github.com/users/codecov%5Bbot%5D/orgs","repos_url":"https://api.github.com/users/codecov%5Bbot%5D/repos","events_url":"https://api.github.com/users/codecov%5Bbot%5D/events{/privacy}","received_events_url":"https://api.github.com/users/codecov%5Bbot%5D/received_events","type":"Bot","site_admin":false},"created_at":"2018-12-10T19:56:49Z","updated_at":"2018-12-10T19:56:49Z","author_association":"NONE","body":"# [Codecov](https://codecov.io/gh/istio/istio/pull/10379?src=pr&el=h1) Report\n> Merging [#10379](https://codecov.io/gh/istio/istio/pull/10379?src=pr&el=desc) into [release-1.0](https://codecov.io/gh/istio/istio/commit/b6d33a3eda7169456c6ebc19d0432c1633666c23?src=pr&el=desc) will **decrease** coverage by `1%`.\n> The diff coverage is `72%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/istio/istio/pull/10379/graphs/tree.svg?width=650&token=nYtepj7fDQ&height=150&src=pr)](https://codecov.io/gh/istio/istio/pull/10379?src=pr&el=tree)\n\n```diff\n@@              Coverage Diff              @@\n##           release-1.0   #10379    +/-   ##\n=============================================\n- Coverage           72%      72%   -<1%     \n=============================================\n  Files              366      366            \n  Lines            32394    32109   -285     \n=============================================\n- Hits             23210    22971   -239     \n+ Misses            8195     8159    -36     \n+ Partials           989      979    -10\n```\n\n\n| [Impacted Files](https://codecov.io/gh/istio/istio/pull/10379?src=pr&el=tree) | Coverage Δ | |\n|---|---|---|\n| [pilot/pkg/proxy/envoy/discovery.go](https://codecov.io/gh/istio/istio/pull/10379/diff?src=pr&el=tree#diff-cGlsb3QvcGtnL3Byb3h5L2Vudm95L2Rpc2NvdmVyeS5nbw==) | `4% <0%> (ø)` | :arrow_down: |\n| [pilot/pkg/serviceregistry/kube/cache.go](https://codecov.io/gh/istio/istio/pull/10379/diff?src=pr&el=tree#diff-cGlsb3QvcGtnL3NlcnZpY2VyZWdpc3RyeS9rdWJlL2NhY2hlLmdv) | `88% <100%> (+2%)` | :arrow_up: |\n| [pilot/pkg/proxy/envoy/v2/eds.go](https://codecov.io/gh/istio/istio/pull/10379/diff?src=pr&el=tree#diff-cGlsb3QvcGtnL3Byb3h5L2Vudm95L3YyL2Vkcy5nbw==) | `79% <87%> (ø)` | :arrow_down: |\n| [...er/adapter/solarwinds/internal/appoptics/client.go](https://codecov.io/gh/istio/istio/pull/10379/diff?src=pr&el=tree#diff-bWl4ZXIvYWRhcHRlci9zb2xhcndpbmRzL2ludGVybmFsL2FwcG9wdGljcy9jbGllbnQuZ28=) | `40% <0%> (-28%)` | :arrow_down: |\n| [mixer/adapter/rbac/controller.go](https://codecov.io/gh/istio/istio/pull/10379/diff?src=pr&el=tree#diff-bWl4ZXIvYWRhcHRlci9yYmFjL2NvbnRyb2xsZXIuZ28=) | `29% <0%> (-24%)` | :arrow_down: |\n| [mixer/adapter/cloudwatch/cloudwatch.go](https://codecov.io/gh/istio/istio/pull/10379/diff?src=pr&el=tree#diff-bWl4ZXIvYWRhcHRlci9jbG91ZHdhdGNoL2Nsb3Vkd2F0Y2guZ28=) | `58% <0%> (-19%)` | :arrow_down: |\n| [pkg/probe/client.go](https://codecov.io/gh/istio/istio/pull/10379/diff?src=pr&el=tree#diff-cGtnL3Byb2JlL2NsaWVudC5nbw==) | `75% <0%> (-6%)` | :arrow_down: |\n| [...r/adapter/stackdriver/contextgraph/contextgraph.go](https://codecov.io/gh/istio/istio/pull/10379/diff?src=pr&el=tree#diff-bWl4ZXIvYWRhcHRlci9zdGFja2RyaXZlci9jb250ZXh0Z3JhcGgvY29udGV4dGdyYXBoLmdv) | `83% <0%> (-3%)` | :arrow_down: |\n| [mixer/adapter/redisquota/redisquota.go](https://codecov.io/gh/istio/istio/pull/10379/diff?src=pr&el=tree#diff-bWl4ZXIvYWRhcHRlci9yZWRpc3F1b3RhL3JlZGlzcXVvdGEuZ28=) | `87% <0%> (-3%)` | :arrow_down: |\n| [mixer/adapter/bypass/bypass.go](https://codecov.io/gh/istio/istio/pull/10379/diff?src=pr&el=tree#diff-bWl4ZXIvYWRhcHRlci9ieXBhc3MvYnlwYXNzLmdv) | `62% <0%> (-1%)` | :arrow_down: |\n| ... and [14 more](https://codecov.io/gh/istio/istio/pull/10379/diff?src=pr&el=tree-more) | |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/istio/istio/pull/10379?src=pr&el=continue).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)\n> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/istio/istio/pull/10379?src=pr&el=footer). Last update [b6d33a3...7d57da2](https://codecov.io/gh/istio/istio/pull/10379?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).\n"}}
{"actor":{"display_login":"codecov"}}