{"action":"created","issue":{"url":"https://api.github.com/repos/IdentityServer/IdentityServer3.EntityFramework/issues/101","repository_url":"https://api.github.com/repos/IdentityServer/IdentityServer3.EntityFramework","labels_url":"https://api.github.com/repos/IdentityServer/IdentityServer3.EntityFramework/issues/101/labels{/name}","comments_url":"https://api.github.com/repos/IdentityServer/IdentityServer3.EntityFramework/issues/101/comments","events_url":"https://api.github.com/repos/IdentityServer/IdentityServer3.EntityFramework/issues/101/events","html_url":"https://github.com/IdentityServer/IdentityServer3.EntityFramework/issues/101","id":153130286,"number":101,"title":"Possible deadlock in access token retrieval?","user":{"login":"sixten","id":7528,"avatar_url":"https://avatars.githubusercontent.com/u/7528?v=3","gravatar_id":"","url":"https://api.github.com/users/sixten","html_url":"https://github.com/sixten","followers_url":"https://api.github.com/users/sixten/followers","following_url":"https://api.github.com/users/sixten/following{/other_user}","gists_url":"https://api.github.com/users/sixten/gists{/gist_id}","starred_url":"https://api.github.com/users/sixten/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sixten/subscriptions","organizations_url":"https://api.github.com/users/sixten/orgs","repos_url":"https://api.github.com/users/sixten/repos","events_url":"https://api.github.com/users/sixten/events{/privacy}","received_events_url":"https://api.github.com/users/sixten/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-04T22:34:13Z","updated_at":"2016-05-05T00:39:23Z","closed_at":null,"body":"We moved our IdentityServer-based user management service into production a couple of months ago. All was well until a few weeks ago, when we started to see a strange situation: the site will stop responding, and start throwing lots of exception as it exhausts its database connection pool. The web server and the SQL Server meantime are performing close to normally. After some period of time in this state (10-90 minutes), it will suddenly unblock, and start working normally again.\r\n\r\nWe've grabbed thread dumps of the app pool a couple of times when this has happened, and the stack traces end up overwhelmingly stuck waiting for the results of the client lookup that happens in the middle of processing an access token validation request. In the dump we took the other day, 638 of 674 threads had this same following trace. (Note that we're still running v1.6.1: we were deep in final testing of this stuff when you released v2. But none of this functionality seems to have changed much.)\r\n\r\n```\r\n[[GCFrame]] \r\n[[HelperMethodFrame_1OBJ] (System.Threading.Monitor.ObjWait)] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object) \r\nmscorlib_ni!System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)+3ec \r\nmscorlib_ni!System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)+db \r\nmscorlib_ni!System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)+24a \r\nmscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)+d6fd50 \r\nmscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter`1[[System.__Canon, mscorlib]].GetResult()+31 \r\nThinktecture.IdentityServer.EntityFramework.Serialization.AsyncHelper.RunSync[[System.__Canon, mscorlib]](System.Func`1>)+ef \r\nThinktecture.IdentityServer.EntityFramework.Serialization.ClientConverter.ReadJson(Newtonsoft.Json.JsonReader, System.Type, System.Object, Newtonsoft.Json.JsonSerializer)+7b \r\nNewtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(Newtonsoft.Json.JsonConverter, Newtonsoft.Json.JsonReader, System.Type, System.Object)+18c \r\nNewtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.JsonConverter, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.JsonReader, System.Object)+131 \r\nNewtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(System.Object, Newtonsoft.Json.JsonReader, Newtonsoft.Json.Serialization.JsonObjectContract, Newtonsoft.Json.Serialization.JsonProperty, System.String)+5ff \r\nNewtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(Newtonsoft.Json.JsonReader, System.Type, Newtonsoft.Json.Serialization.JsonContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty, System.Object)+357 \r\nNewtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(Newtonsoft.Json.JsonReader, System.Type, Newtonsoft.Json.Serialization.JsonContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty, System.Object)+bc \r\nNewtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.JsonConverter, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.JsonReader, System.Object)+174 \r\nNewtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(System.Object, Newtonsoft.Json.JsonReader, Newtonsoft.Json.Serialization.JsonObjectContract, Newtonsoft.Json.Serialization.JsonProperty, System.String)+5ff \r\nNewtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(Newtonsoft.Json.JsonReader, System.Type, Newtonsoft.Json.Serialization.JsonContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty, System.Object)+357 \r\nNewtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(Newtonsoft.Json.JsonReader, System.Type, Newtonsoft.Json.Serialization.JsonContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty, System.Object)+bc \r\nNewtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(Newtonsoft.Json.JsonReader, System.Type, Boolean)+1a8 \r\nNewtonsoft.Json.JsonSerializer.DeserializeInternal(Newtonsoft.Json.JsonReader, System.Type)+125 \r\nNewtonsoft.Json.JsonConvert.DeserializeObject(System.String, System.Type, Newtonsoft.Json.JsonSerializerSettings)+112 \r\nNewtonsoft.Json.JsonConvert.DeserializeObject[[System.__Canon, mscorlib]](System.String, Newtonsoft.Json.JsonSerializerSettings)+39 \r\nThinktecture.IdentityServer.EntityFramework.BaseTokenStore`1+d__0[[System.__Canon, mscorlib]].MoveNext()+2d5 \r\nmscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+285 \r\nmscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+9 \r\nmscorlib_ni!System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()+a4 \r\nSystem_Web_ni!System.Web.Util.SynchronizationHelper.SafeWrapCallback(System.Action)+b3 \r\nmscorlib_ni!System.Threading.Tasks.Task.Execute()+6e \r\nmscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+285 \r\nmscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+9 \r\nmscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)+2c5 \r\nmscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry(Boolean)+85 \r\nmscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch()+1ea \r\n[[DebuggerU2MCatchHandlerFrame]] \r\n[[ContextTransitionFrame]] \r\n[[DebuggerU2MCatchHandlerFrame]] \r\n```\r\n\r\nHave you seen this behavior before from other customers / installations of IdentityServer?\r\n"},"comment":{"url":"https://api.github.com/repos/IdentityServer/IdentityServer3.EntityFramework/issues/comments/217048112","html_url":"https://github.com/IdentityServer/IdentityServer3.EntityFramework/issues/101#issuecomment-217048112","issue_url":"https://api.github.com/repos/IdentityServer/IdentityServer3.EntityFramework/issues/101","id":217048112,"user":{"login":"brockallen","id":649492,"avatar_url":"https://avatars.githubusercontent.com/u/649492?v=3","gravatar_id":"","url":"https://api.github.com/users/brockallen","html_url":"https://github.com/brockallen","followers_url":"https://api.github.com/users/brockallen/followers","following_url":"https://api.github.com/users/brockallen/following{/other_user}","gists_url":"https://api.github.com/users/brockallen/gists{/gist_id}","starred_url":"https://api.github.com/users/brockallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brockallen/subscriptions","organizations_url":"https://api.github.com/users/brockallen/orgs","repos_url":"https://api.github.com/users/brockallen/repos","events_url":"https://api.github.com/users/brockallen/events{/privacy}","received_events_url":"https://api.github.com/users/brockallen/received_events","type":"User","site_admin":false},"created_at":"2016-05-05T00:39:23Z","updated_at":"2016-05-05T00:39:23Z","body":"Are you using reference tokens?"}}