{"action":"created","issue":{"url":"https://api.github.com/repos/aspnet/KestrelHttpServer/issues/2936","repository_url":"https://api.github.com/repos/aspnet/KestrelHttpServer","labels_url":"https://api.github.com/repos/aspnet/KestrelHttpServer/issues/2936/labels{/name}","comments_url":"https://api.github.com/repos/aspnet/KestrelHttpServer/issues/2936/comments","events_url":"https://api.github.com/repos/aspnet/KestrelHttpServer/issues/2936/events","html_url":"https://github.com/aspnet/KestrelHttpServer/issues/2936","id":360630669,"node_id":"MDU6SXNzdWUzNjA2MzA2Njk=","number":2936,"title":"The connection was closed because the response was not read by the client at the specified minimum data rate","user":{"login":"lorenzos70","id":10574993,"node_id":"MDQ6VXNlcjEwNTc0OTkz","avatar_url":"https://avatars3.githubusercontent.com/u/10574993?v=4","gravatar_id":"","url":"https://api.github.com/users/lorenzos70","html_url":"https://github.com/lorenzos70","followers_url":"https://api.github.com/users/lorenzos70/followers","following_url":"https://api.github.com/users/lorenzos70/following{/other_user}","gists_url":"https://api.github.com/users/lorenzos70/gists{/gist_id}","starred_url":"https://api.github.com/users/lorenzos70/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lorenzos70/subscriptions","organizations_url":"https://api.github.com/users/lorenzos70/orgs","repos_url":"https://api.github.com/users/lorenzos70/repos","events_url":"https://api.github.com/users/lorenzos70/events{/privacy}","received_events_url":"https://api.github.com/users/lorenzos70/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2018-09-16T11:35:40Z","updated_at":"2018-09-19T18:26:37Z","closed_at":null,"author_association":"NONE","body":"Hello,\r\n\r\nI have a .net 4.6.2 project (for my web apis) that is referencing, among other libraries, Microsoft.AspNetCore 2.1.3. I recently upgraded it from version 1.1. Since the upgrade I experienced the following issues on my production win 2016 server:\r\n \r\n- my api logs often show the following messages: \r\n    1. Connection id \"\"xxxxxx\"\", Request id \"\"yyyyyy\"\": the connection was closed because the response was not read by the client at the specified minimum data rate\r\n            \r\n   2. the request timed out because it was not sent by the client at a minimum of 240 bytes/second\".\r\n           \r\n   3. [Warning] Connection processing ended abnormally.\r\n            Microsoft.AspNetCore.Connections.ConnectionAbortedException: The connection was aborted\r\n            at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()\r\n           at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)\r\n           at System.IO.Pipelines.Pipe.GetReadAsyncResult()\r\n           at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)\r\n           at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol. \r\n           <ProcessRequests>d__188`1.MoveNext()--- End of stack trace from previous location where \r\n          exception was thrown ---   at \r\n         System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at \r\n         System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task \r\n         task)\r\n         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol. \r\n        <ProcessRequestsAsync>d__187`1.MoveNext()\r\n\r\n- The apis become unresponsive (they are hosted on iis 10 on a windows 2016 server)\r\n- IIS logs show, at the same time, 502 errors while trying to call the api endoints \r\n- I also noticed heavy peaks of the CPU.\r\n\r\nRemarks: the web api load has not increased and there haven't been any other major changes, apart from the upgrade of the  Microsoft.AspNetCore that I mentioned above, so I am still trying to figure out what the problem could be. Any advice that could point me to the right direction?\r\nThanks\r\nLorenzo"},"comment":{"url":"https://api.github.com/repos/aspnet/KestrelHttpServer/issues/comments/422909244","html_url":"https://github.com/aspnet/KestrelHttpServer/issues/2936#issuecomment-422909244","issue_url":"https://api.github.com/repos/aspnet/KestrelHttpServer/issues/2936","id":422909244,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMjkwOTI0NA==","user":{"login":"noelstieglitz","id":1135723,"node_id":"MDQ6VXNlcjExMzU3MjM=","avatar_url":"https://avatars1.githubusercontent.com/u/1135723?v=4","gravatar_id":"","url":"https://api.github.com/users/noelstieglitz","html_url":"https://github.com/noelstieglitz","followers_url":"https://api.github.com/users/noelstieglitz/followers","following_url":"https://api.github.com/users/noelstieglitz/following{/other_user}","gists_url":"https://api.github.com/users/noelstieglitz/gists{/gist_id}","starred_url":"https://api.github.com/users/noelstieglitz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noelstieglitz/subscriptions","organizations_url":"https://api.github.com/users/noelstieglitz/orgs","repos_url":"https://api.github.com/users/noelstieglitz/repos","events_url":"https://api.github.com/users/noelstieglitz/events{/privacy}","received_events_url":"https://api.github.com/users/noelstieglitz/received_events","type":"User","site_admin":false},"created_at":"2018-09-19T18:26:37Z","updated_at":"2018-09-19T18:26:37Z","author_association":"NONE","body":"@halter73 - thanks for the quick response. \r\n\r\nI did some more digging and found that the exception happened once on 8/21, 8/27, and 8/30.  Then a few times on the 14th and we've seen ~75 errors this week.  This could be due to increased traffic.  I'll need to write a fancy query to find out.\r\n\r\n> How long were you running your app without seeing issues before seeing issues yesterday? \r\nThis app has been running for months.\r\n\r\n> Did anything else notable change regarding your app around the time you starting seeing the request body rate timeouts?\r\n\r\nWe deployed a change that I initially suspected was related (based on the timing), but after rolling back that change I was still able to reproduce the error.  The only other piece that (I know) changed was the rollout of .net core `v2.1.4` to Azure App Services, which was applied to the specific app service instance on 9/14 (the same day I first see the error in app insights).  I didn't update the app's reference from `v2.1.2` to `v2.1.4` until yesterday (9/18), so I wouldn't expect that to be related?\r\n\r\n> Also, what makes this reminiscent #2151 to you?\r\nIt's the same exception type (`BadHttpRequestException`), HTTP response (`400`) and hosted in Azure App Services (proxied through IIS first).  It happens sporadically.  The stack trace is different though and it doesn't mention `MinRequestBodyDataRate`, so maybe not related at all.\r\n\r\n>  it can be increased to whatever you want without changing the actual rate that's expected for large reads.\r\n\r\nThe response body is not large, in fact for one of the operations, the response body is empty.  The other response (including headers) is about 1,300 bytes.  Neither operations is CPU intensive.  They both make several asynchronous calls to other endpoints (some of which are parallel).  Typically the entire call takes something like a few hundred ms to maybe 2k ms.  When it fails, the request takes ~6k-7k ms.  The related code hasn't changed recently. Given these details, do you still think it makes sense to adjust the `MinRequestBodyDataRate::GracePeriod`?\r\n\r\n> getting info from the reporter about how to reproduce the issue on our own hardware\r\nOf course.  Short of creating a demo project, what would be the best way to do that?  Fiddler session?\r\n"}}
{"actor":{"display_login":"noelstieglitz"}}