{"action":"opened","issue":{"url":"https://api.github.com/repos/Azure/azure-event-hubs-dotnet/issues/76","repository_url":"https://api.github.com/repos/Azure/azure-event-hubs-dotnet","labels_url":"https://api.github.com/repos/Azure/azure-event-hubs-dotnet/issues/76/labels{/name}","comments_url":"https://api.github.com/repos/Azure/azure-event-hubs-dotnet/issues/76/comments","events_url":"https://api.github.com/repos/Azure/azure-event-hubs-dotnet/issues/76/events","html_url":"https://github.com/Azure/azure-event-hubs-dotnet/issues/76","id":193028543,"number":76,"title":"Event hub partitions cannot be shut down, which is causing the UnregisterEventProcessorAsync cannot be returned","user":{"login":"xingzhougmu","id":2773009,"avatar_url":"https://avatars.githubusercontent.com/u/2773009?v=3","gravatar_id":"","url":"https://api.github.com/users/xingzhougmu","html_url":"https://github.com/xingzhougmu","followers_url":"https://api.github.com/users/xingzhougmu/followers","following_url":"https://api.github.com/users/xingzhougmu/following{/other_user}","gists_url":"https://api.github.com/users/xingzhougmu/gists{/gist_id}","starred_url":"https://api.github.com/users/xingzhougmu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xingzhougmu/subscriptions","organizations_url":"https://api.github.com/users/xingzhougmu/orgs","repos_url":"https://api.github.com/users/xingzhougmu/repos","events_url":"https://api.github.com/users/xingzhougmu/events{/privacy}","received_events_url":"https://api.github.com/users/xingzhougmu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-12-02T03:55:41Z","updated_at":"2016-12-02T03:55:41Z","closed_at":null,"body":"Hi,\r\n\r\nWe found an issue with EventProcessorHost. It's complicated and I will try my best to articulate clearly.\r\n\r\n## Environment Set up\r\n+ Azure Cloud Service Role\r\nThe cloud service role has three role instances, and we are using EventProcessHost to consume events from Azure Event Hub.\r\n+ Event Hub\r\nConfigured to have 32 partitions\r\n\r\n## Issue Description\r\nFor some reason, we are receiving the following exception indicating that the connection to event hub was lost.\r\n```\r\n 2016-11-29T07:24:51.338Z,Ford.Cevdm.EventHubListener_IN_1,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>eventProcessorOptions_ExceptionReceived</methodName><label>e4952157-f876-442f-ba9b-05a091de70c2</label><message>***** Lost connection to event hub *****New receiver with higher epoch of '155' is created hence current receiver with epoch '154' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:f45285ca000431930012f6ca583d2c57_G6_B3, SystemTracker:fcnedogprod:eventhub:dataoffloadeventhub~19455|chev, Timestamp:11/29/2016 7:23:28 AM</message><stackTrace>   at ***.Cevdm.EventHubListener.EventHubListenerWorkerRole.eventProcessorOptions_ExceptionReceived(Object sender, ExceptionReceivedEventArgs e)\r\n```\r\nAfter the connection lost, we will unregister the EventProcessHost. However, the unregistration cannot be completed, and UnregisterEventProcessorAsync never returned.\r\n```c#\r\nprivate void shutdownEventHub(int uMS)\r\n        {\r\n            var label = CevdmTrace.TraceIn();\r\n            //Unregister the event processor so other instances will handle the partitions\r\n            for (var i = 0; i < _eventProcessorHosts.Count(); i++)\r\n            {\r\n                CevdmTrace.TraceInformation(label, \"Unregistering event processor \" + i);\r\n                try\r\n                {\r\n                    _eventProcessorHosts[i].UnregisterEventProcessorAsync().Wait();\r\n                    CevdmTrace.TraceInformation(label, \"Event processor unregistered \" + i);\r\n                }\r\n                catch (Exception ex)\r\n                {\r\n                    CevdmTrace.TraceError(label, ex);\r\n                    //Unregister the event processor so other instances will handle the partitions\r\n                    CevdmTrace.TraceInformation(label, \"Try again unregistering event processor.\");\r\n                    if (_eventProcessorHosts[i] != null)\r\n                        _eventProcessorHosts[i].UnregisterEventProcessorAsync().Wait();\r\n                    CevdmTrace.TraceInformation(label, \"Event processor unregistered.\");\r\n                }\r\n            }\r\n\r\n            if (uMS > 0) Thread.Sleep(uMS);\r\n            _runCompleteEvent.Set();\r\n            CevdmTrace.TraceOut(label);\r\n        }\r\n```\r\n## Investigation\r\nI checked the Event Hub Partition shut down process in role instance \"EventHubListener_IN_1\", and noticed that there is one partition hasn't been shutdown gracefully - **Partition 18**. This is causing the UnregisterEventProcessorAsync stuck. \r\n\r\nLogs shown as below:\r\n+ 10 partitions **lost lease** since we have three role instances, and EventProcessHost will load balance the processing among these three role instances.\r\n```\r\n<message>CevdmEventProcessor Shutting Down. Partition '0', Reason: 'LeaseLost'.</message>\r\n<message>CevdmEventProcessor Shutting Down. Partition '1', Reason: 'LeaseLost'.</message>\r\n<message>CevdmEventProcessor Shutting Down. Partition '10', Reason: 'LeaseLost'.</message>\r\n<message>CevdmEventProcessor Shutting Down. Partition '11', Reason: 'LeaseLost'.</message>\r\n<message>CevdmEventProcessor Shutting Down. Partition '12', Reason: 'LeaseLost'.</message>\r\n<message>CevdmEventProcessor Shutting Down. Partition '13', Reason: 'LeaseLost'.</message>\r\n<message>CevdmEventProcessor Shutting Down. Partition '14', Reason: 'LeaseLost'.</message>\r\n<message>CevdmEventProcessor Shutting Down. Partition '15', Reason: 'LeaseLost'.</message>\r\n<message>CevdmEventProcessor Shutting Down. Partition '16', Reason: 'LeaseLost'.</message>\r\n<message>CevdmEventProcessor Shutting Down. Partition '17', Reason: 'LeaseLost'.</message>\r\n```\r\n+ 21 partitions **shut down** by CloseAsync\r\n```\r\nCevdmEventProcessor Shutting Down. Partition '27', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '5', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '28', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '4', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '31', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '3', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '7', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '9', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '23', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '6', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '20', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '30', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '19', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '24', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '26', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '2', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '21', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '8', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '29', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '25', Reason: 'Shutdown'.</message>\r\nCevdmEventProcessor Shutting Down. Partition '22', Reason: 'Shutdown'.</message>\r\n```\r\n## Why Partition 18 cannot be shutdown?\r\nThe life cycle of Partition 18 is as below.\r\n1. 2016-11-29T07:14:01.354Z initialized in EventHubListener_IN_**0**\r\n2. 2016-11-29T07:20:55.314Z initialized in EventHubListener_IN_**1**\r\n3. 2016-11-29T07:23:24.069Z initialized in EventHubListener_IN_**2**\r\n4. 2016-11-29T07:23:28.366Z \"Lost connection to event hub\" in EventHubListener_IN_1, and Partition 18 cannot be shut down. I guess this is because Partition 18 is owned by  EventHubListener_IN_**2** now. However, I didn't find any trace saying EventHubListener_IN_**1** lost lease on Partition 18.\r\n```\r\n2016-11-29T07:14:01.354Z,***.Cevdm.EventHubListener_IN_0,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>OpenAsync</methodName><label>668de595-e949-402b-8abe-0b9c5a059bff</label><message>CevdmEventProcessor **initialized**.  Partition: '18', Offset: '98810264328'</message><stackTrace></stackTrace>\"\" TraceSource=\"\"WaWorkerHost.exe\"\"\"\r\n2016-11-29T07:20:55.314Z,***.Cevdm.EventHubListener_IN_1,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>OpenAsync</methodName><label>03f7e914-8062-4a66-af04-61be78f3fe9b</label><message>CevdmEventProcessor **initialized**.  Partition: '18', Offset: '103086931440'</message><stackTrace></stackTrace>\"\" TraceSource=\"\"WaWorkerHost.exe\"\"\"\r\n2016-11-29T07:23:24.069Z,***.Cevdm.EventHubListener_IN_2,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>OpenAsync</methodName><label>e1e3f23c-840d-4142-bb5d-cfc81fb5d46e</label><message>CevdmEventProcessor **initialized**.  Partition: '18', Offset: '103086931440'</message><stackTrace></stackTrace>\"\" TraceSource=\"\"WaWorkerHost.exe\"\"\"\r\n```\r\n\r\n**Examples on other partitions:**\r\n+ Partition 19:\r\n```\r\n2016-11-29T07:14:01.338Z,***.Cevdm.EventHubListener_IN_0,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>OpenAsync</methodName><label>08c32451-d1d0-4130-bc41-0d36c000e80e</label><message>CevdmEventProcessor **initialized**.  Partition: '19', Offset: '98810074928'</message><stackTrace></stackTrace>\"\" TraceSource=\"\"WaWorkerHost.exe\"\"\"\r\n2016-11-29T07:21:06.076Z,***.Cevdm.EventHubListener_IN_1,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>OpenAsync</methodName><label>4f1205e0-6f85-47ad-993e-a7242f788407</label><message>CevdmEventProcessor **initialized**.  Partition: '19', Offset: '103086913832'</message><stackTrace></stackTrace>\"\" TraceSource=\"\"WaWorkerHost.exe\"\"\"\r\n2016-11-29T07:23:28.624Z,***.Cevdm.EventHubListener_IN_1,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>CloseAsync</methodName><label>82e0d93a-e435-47ad-b622-b96f866aa536</label><message>CevdmEventProcessor **Shutting Down. Partition '19', Reason: 'Shutdown'**.</message><stackTrace></stackTrace>\"\" TraceSource=\"\"WaWorkerHost.exe\"\"\"\r\n2016-11-29T07:23:35.501Z,***.Cevdm.EventHubListener_IN_2,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>OpenAsync</methodName><label>b9de887c-1811-4099-bb9c-4b5641086a32</label><message>CevdmEventProcessor **initialized**.  Partition: '19', Offset: '103086951816'</message><stackTrace></stackTrace>\"\" TraceSource=\"\"WaWorkerHost.exe\"\"\"\r\n```\r\n\r\n+ Partition 17:\r\n```\r\n2016-11-29T07:14:01.330Z,***.Cevdm.EventHubListener_IN_0,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>OpenAsync</methodName><label>f15d52a8-52b9-4570-a41e-4425437b2374</label><message>CevdmEventProcessor **initialized**.  Partition: '17', Offset: '98809856608'</message><stackTrace></stackTrace>\"\" TraceSource=\"\"WaWorkerHost.exe\"\"\"\r\n2016-11-29T07:20:44.776Z,***.Cevdm.EventHubListener_IN_1,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>OpenAsync</methodName><label>e9373755-8ede-433c-b009-6dec4736a67e</label><message>CevdmEventProcessor **initialized**.  Partition: '17', Offset: '103086653704'</message><stackTrace></stackTrace>\"\" TraceSource=\"\"WaWorkerHost.exe\"\"\"\r\n2016-11-29T07:23:12.726Z,***.Cevdm.EventHubListener_IN_1,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>CloseAsync</methodName><label>0072577c-c4f4-425a-8d4c-482265537f78</label><message>CevdmEventProcessor **Shutting Down. Partition '17', Reason: 'LeaseLost'**.</message><stackTrace></stackTrace>\"\" TraceSource=\"\"WaWorkerHost.exe\"\"\"\r\n2016-11-29T07:23:13.418Z,***.Cevdm.EventHubListener_IN_2,\"EventName=\"\"MessageEvent\"\" Message=\"\"<methodName>OpenAsync</methodName><label>5a78af17-2fbb-4aeb-983c-ee6d4cbdf207</label><message>CevdmEventProcessor **initialized**.  Partition: '17', Offset: '103086653704'</message><stackTrace></stackTrace>\"\" TraceSource=\"\"WaWorkerHost.exe\"\"\"\r\n```\r\n## Requests:\r\n+ Please help me take a look at this issue and check if there was any issue in the SDK?\r\n+ Any suggestions to troubleshooting this issue further?"}}
{"actor":{"display_login":"xingzhougmu"}}