{"action":"created","issue":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-dotnet-client/issues/288","repository_url":"https://api.github.com/repos/rabbitmq/rabbitmq-dotnet-client","labels_url":"https://api.github.com/repos/rabbitmq/rabbitmq-dotnet-client/issues/288/labels{/name}","comments_url":"https://api.github.com/repos/rabbitmq/rabbitmq-dotnet-client/issues/288/comments","events_url":"https://api.github.com/repos/rabbitmq/rabbitmq-dotnet-client/issues/288/events","html_url":"https://github.com/rabbitmq/rabbitmq-dotnet-client/issues/288","id":188285876,"number":288,"title":"AutorecoveringConnection does not synchronize access to all recorded entity collections","user":{"login":"Mattish","id":2098550,"avatar_url":"https://avatars.githubusercontent.com/u/2098550?v=3","gravatar_id":"","url":"https://api.github.com/users/Mattish","html_url":"https://github.com/Mattish","followers_url":"https://api.github.com/users/Mattish/followers","following_url":"https://api.github.com/users/Mattish/following{/other_user}","gists_url":"https://api.github.com/users/Mattish/gists{/gist_id}","starred_url":"https://api.github.com/users/Mattish/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mattish/subscriptions","organizations_url":"https://api.github.com/users/Mattish/orgs","repos_url":"https://api.github.com/users/Mattish/repos","events_url":"https://api.github.com/users/Mattish/events{/privacy}","received_events_url":"https://api.github.com/users/Mattish/received_events","type":"User","site_admin":false},"labels":[{"id":178241404,"url":"https://api.github.com/repos/rabbitmq/rabbitmq-dotnet-client/labels/bug","name":"bug","color":"fc2929","default":true},{"id":178242227,"url":"https://api.github.com/repos/rabbitmq/rabbitmq-dotnet-client/labels/effort-low","name":"effort-low","color":"88b3e4","default":false}],"state":"open","locked":false,"assignee":{"login":"kjnilsson","id":1180564,"avatar_url":"https://avatars.githubusercontent.com/u/1180564?v=3","gravatar_id":"","url":"https://api.github.com/users/kjnilsson","html_url":"https://github.com/kjnilsson","followers_url":"https://api.github.com/users/kjnilsson/followers","following_url":"https://api.github.com/users/kjnilsson/following{/other_user}","gists_url":"https://api.github.com/users/kjnilsson/gists{/gist_id}","starred_url":"https://api.github.com/users/kjnilsson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kjnilsson/subscriptions","organizations_url":"https://api.github.com/users/kjnilsson/orgs","repos_url":"https://api.github.com/users/kjnilsson/repos","events_url":"https://api.github.com/users/kjnilsson/events{/privacy}","received_events_url":"https://api.github.com/users/kjnilsson/received_events","type":"User","site_admin":false},"assignees":[{"login":"kjnilsson","id":1180564,"avatar_url":"https://avatars.githubusercontent.com/u/1180564?v=3","gravatar_id":"","url":"https://api.github.com/users/kjnilsson","html_url":"https://github.com/kjnilsson","followers_url":"https://api.github.com/users/kjnilsson/followers","following_url":"https://api.github.com/users/kjnilsson/following{/other_user}","gists_url":"https://api.github.com/users/kjnilsson/gists{/gist_id}","starred_url":"https://api.github.com/users/kjnilsson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kjnilsson/subscriptions","organizations_url":"https://api.github.com/users/kjnilsson/orgs","repos_url":"https://api.github.com/users/kjnilsson/repos","events_url":"https://api.github.com/users/kjnilsson/events{/privacy}","received_events_url":"https://api.github.com/users/kjnilsson/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-dotnet-client/milestones/12","html_url":"https://github.com/rabbitmq/rabbitmq-dotnet-client/milestones/3.6.x","labels_url":"https://api.github.com/repos/rabbitmq/rabbitmq-dotnet-client/milestones/12/labels","id":1465450,"number":12,"title":"3.6.x","description":null,"creator":{"login":"dumbbell","id":159804,"avatar_url":"https://avatars.githubusercontent.com/u/159804?v=3","gravatar_id":"","url":"https://api.github.com/users/dumbbell","html_url":"https://github.com/dumbbell","followers_url":"https://api.github.com/users/dumbbell/followers","following_url":"https://api.github.com/users/dumbbell/following{/other_user}","gists_url":"https://api.github.com/users/dumbbell/gists{/gist_id}","starred_url":"https://api.github.com/users/dumbbell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dumbbell/subscriptions","organizations_url":"https://api.github.com/users/dumbbell/orgs","repos_url":"https://api.github.com/users/dumbbell/repos","events_url":"https://api.github.com/users/dumbbell/events{/privacy}","received_events_url":"https://api.github.com/users/dumbbell/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":2,"state":"open","created_at":"2015-12-17T18:27:09Z","updated_at":"2016-11-09T16:32:00Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2016-11-09T16:11:39Z","updated_at":"2016-11-16T15:54:58Z","closed_at":null,"body":"We've run into some Unobserved task exceptions being thrown from AutorecoveringConnection.RecoverExchanges() due to the collection being modified.\r\n\r\n\r\n```\r\nUnobserved task exception occurred \r\nSystem.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute. \r\nat System.Collections.Generic.Dictionary`2.ValueCollection.Enumerator.MoveNext() \r\nat RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.RecoverExchanges() in d:\\work\\RabbitMQ\\projects\\client\\RabbitMQ.Client\\src\\client\\impl\\AutorecoveringConnection.cs:line 911 \r\nat RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.PerformAutomaticRecovery() in d:\\work\\RabbitMQ\\projects\\client\\RabbitMQ.Client\\src\\client\\impl\\AutorecoveringConnection.cs:line 374 \r\nat RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.<>c__DisplayClass10.<BeginAutomaticRecovery>b__f() in d:\\work\\RabbitMQ\\projects\\client\\RabbitMQ.Client\\src\\client\\impl\\AutorecoveringConnection.cs:line 352 \r\nat System.Threading.Tasks.Task.Execute() \r\n--- End of inner exception stack trace --- \r\n---> (Inner Exception #0) System.InvalidOperationException: Collection was modified; enumeration operation may not execute. \r\nat System.Collections.Generic.Dictionary`2.ValueCollection.Enumerator.MoveNext() \r\nat RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.RecoverExchanges() in d:\\work\\RabbitMQ\\projects\\client\\RabbitMQ.Client\\src\\client\\impl\\AutorecoveringConnection.cs:line 911 \r\nat RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.PerformAutomaticRecovery() in d:\\work\\RabbitMQ\\projects\\client\\RabbitMQ.Client\\src\\client\\impl\\AutorecoveringConnection.cs:line 374 \r\nat RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.<>c__DisplayClass10.<BeginAutomaticRecovery>b__f() in d:\\work\\RabbitMQ\\projects\\client\\RabbitMQ.Client\\src\\client\\impl\\AutorecoveringConnection.cs:line 352 \r\nat System.Threading.Tasks.Task.Execute()<--- \r\n```\r\n\r\nIn `RecoverQueues()` we lock on `m_recordedQueues` but elsewhere in the class `m_recordedQueues` is locked by `m_recordedEntitiesLock` instead. As there is no synchronization in `RecoverQueues()`, the `Dictionary` constructor uses `foreach` so the dictionary clone isn't protected and can throw if elsewhere manipulated.\r\n\r\nWe believe this problem will also occur for `RecoverExchanges()` due to no locking around the `foreach` but there is elsewhere in the class, although we have no seen this thrown.\r\n\r\nWe've only had 2 single cases of this happening in the past year+ of using rabbit, but it does cause the service to fall over. We are running on 3.6.5, but I can see this set of locking still exists on master"},"comment":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-dotnet-client/issues/comments/260983075","html_url":"https://github.com/rabbitmq/rabbitmq-dotnet-client/issues/288#issuecomment-260983075","issue_url":"https://api.github.com/repos/rabbitmq/rabbitmq-dotnet-client/issues/288","id":260983075,"user":{"login":"michaelklishin","id":1090,"avatar_url":"https://avatars.githubusercontent.com/u/1090?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelklishin","html_url":"https://github.com/michaelklishin","followers_url":"https://api.github.com/users/michaelklishin/followers","following_url":"https://api.github.com/users/michaelklishin/following{/other_user}","gists_url":"https://api.github.com/users/michaelklishin/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelklishin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelklishin/subscriptions","organizations_url":"https://api.github.com/users/michaelklishin/orgs","repos_url":"https://api.github.com/users/michaelklishin/repos","events_url":"https://api.github.com/users/michaelklishin/events{/privacy}","received_events_url":"https://api.github.com/users/michaelklishin/received_events","type":"User","site_admin":false},"created_at":"2016-11-16T15:54:58Z","updated_at":"2016-11-16T15:54:58Z","body":"A good candidate for `3.6.7`."}}
{"actor":{"display_login":"michaelklishin"}}