{"action":"opened","issue":{"url":"https://api.github.com/repos/Azure/azure-functions-durable-extension/issues/295","repository_url":"https://api.github.com/repos/Azure/azure-functions-durable-extension","labels_url":"https://api.github.com/repos/Azure/azure-functions-durable-extension/issues/295/labels{/name}","comments_url":"https://api.github.com/repos/Azure/azure-functions-durable-extension/issues/295/comments","events_url":"https://api.github.com/repos/Azure/azure-functions-durable-extension/issues/295/events","html_url":"https://github.com/Azure/azure-functions-durable-extension/issues/295","id":319999213,"number":295,"title":"Task hub listeners can fail to start repeatedly and endlessly","user":{"login":"cgillum","id":2704139,"avatar_url":"https://avatars2.githubusercontent.com/u/2704139?v=4","gravatar_id":"","url":"https://api.github.com/users/cgillum","html_url":"https://github.com/cgillum","followers_url":"https://api.github.com/users/cgillum/followers","following_url":"https://api.github.com/users/cgillum/following{/other_user}","gists_url":"https://api.github.com/users/cgillum/gists{/gist_id}","starred_url":"https://api.github.com/users/cgillum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgillum/subscriptions","organizations_url":"https://api.github.com/users/cgillum/orgs","repos_url":"https://api.github.com/users/cgillum/repos","events_url":"https://api.github.com/users/cgillum/events{/privacy}","received_events_url":"https://api.github.com/users/cgillum/received_events","type":"User","site_admin":false},"labels":[{"id":618529384,"url":"https://api.github.com/repos/Azure/azure-functions-durable-extension/labels/bug","name":"bug","color":"ee0701","default":true},{"id":735546911,"url":"https://api.github.com/repos/Azure/azure-functions-durable-extension/labels/dtfx","name":"dtfx","color":"52c7f9","default":false}],"state":"open","locked":false,"assignee":{"login":"cgillum","id":2704139,"avatar_url":"https://avatars2.githubusercontent.com/u/2704139?v=4","gravatar_id":"","url":"https://api.github.com/users/cgillum","html_url":"https://github.com/cgillum","followers_url":"https://api.github.com/users/cgillum/followers","following_url":"https://api.github.com/users/cgillum/following{/other_user}","gists_url":"https://api.github.com/users/cgillum/gists{/gist_id}","starred_url":"https://api.github.com/users/cgillum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgillum/subscriptions","organizations_url":"https://api.github.com/users/cgillum/orgs","repos_url":"https://api.github.com/users/cgillum/repos","events_url":"https://api.github.com/users/cgillum/events{/privacy}","received_events_url":"https://api.github.com/users/cgillum/received_events","type":"User","site_admin":false},"assignees":[{"login":"cgillum","id":2704139,"avatar_url":"https://avatars2.githubusercontent.com/u/2704139?v=4","gravatar_id":"","url":"https://api.github.com/users/cgillum","html_url":"https://github.com/cgillum","followers_url":"https://api.github.com/users/cgillum/followers","following_url":"https://api.github.com/users/cgillum/following{/other_user}","gists_url":"https://api.github.com/users/cgillum/gists{/gist_id}","starred_url":"https://api.github.com/users/cgillum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgillum/subscriptions","organizations_url":"https://api.github.com/users/cgillum/orgs","repos_url":"https://api.github.com/users/cgillum/repos","events_url":"https://api.github.com/users/cgillum/events{/privacy}","received_events_url":"https://api.github.com/users/cgillum/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/Azure/azure-functions-durable-extension/milestones/3","html_url":"https://github.com/Azure/azure-functions-durable-extension/milestone/3","labels_url":"https://api.github.com/repos/Azure/azure-functions-durable-extension/milestones/3/labels","id":3015683,"number":3,"title":"General Availability (GA)","description":"This will be the first \"go-live\" version that will be fully supported in production. In addition to all planned features being completed, documentation and samples should be fully up-to-date as well.","creator":{"login":"cgillum","id":2704139,"avatar_url":"https://avatars2.githubusercontent.com/u/2704139?v=4","gravatar_id":"","url":"https://api.github.com/users/cgillum","html_url":"https://github.com/cgillum","followers_url":"https://api.github.com/users/cgillum/followers","following_url":"https://api.github.com/users/cgillum/following{/other_user}","gists_url":"https://api.github.com/users/cgillum/gists{/gist_id}","starred_url":"https://api.github.com/users/cgillum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgillum/subscriptions","organizations_url":"https://api.github.com/users/cgillum/orgs","repos_url":"https://api.github.com/users/cgillum/repos","events_url":"https://api.github.com/users/cgillum/events{/privacy}","received_events_url":"https://api.github.com/users/cgillum/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":29,"state":"open","created_at":"2018-01-05T18:36:41Z","updated_at":"2018-05-03T16:49:08Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2018-05-03T16:49:08Z","updated_at":"2018-05-03T16:49:08Z","closed_at":null,"author_association":"COLLABORATOR","body":"### Issue\r\nAs part of investigating https://github.com/Azure/azure-functions-durable-extension/issues/288, I discovered that task hub creation failures during the functions host startup repeat continuously and endlessly. Here is an example error message:\r\n\r\n```\r\nMicrosoft.Azure.WebJobs.Host.Listeners.FunctionListenerException : The listener for function 'XXX' was unable to start. ---> Microsoft.WindowsAzure.Storage.StorageException : The remote server returned an error: (409) Conflict. ---> System.Net.WebException : The remote server returned an error: (409) Conflict.\r\n   at Microsoft.WindowsAzure.Storage.Table.Protocol.TableOperationHttpResponseParsers.TableOperationPreProcess(TableResult result,TableOperation operation,HttpWebResponse resp,Exception ex) at c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Table\\Protocol\\TableOperationHttpResponseParsers.cs : 61\r\n   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult) at c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Core\\Executor\\Executor.cs : 299 \r\n   End of inner exception\r\n   at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End() at c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Core\\Util\\StorageAsyncResult.cs : 77\r\n   at Microsoft.WindowsAzure.Storage.Table.CloudTable.EndCreateIfNotExists(IAsyncResult asyncResult) at c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Table\\CloudTable.cs : 1183\r\n   at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass1`1.<CreateCallback>b__0(IAsyncResult ar) at c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Core\\Util\\AsyncExtensions.cs : 66\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at async DurableTask.AzureStorage.AzureStorageOrchestrationService.GetTaskHubCreatorTask()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at async Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskExtension.StartTaskHubWorkerIfNotStartedAsync()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at async Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(??) \r\n   End of inner exception\r\n```\r\n\r\n### Investigation\r\nThe root cause appears to be because of how a `Task` object is being cached in the **DurableTask.AzureStorage** code, which runs when the functions trigger listeners are starting. Basically, the code awaits on the same failed `Task` object every time the listeners try to start up again. A fix is needed which either removes this caching or makes it smarter about handling failures.\r\n\r\n### Workaround\r\nThe workaround is to recycle the functions host to clear the cache."}}
{"actor":{"display_login":"cgillum"}}