title: Azure table storage Client takes a lot of time retrieving records
body: <p>We have a windows service running on our server machines which listens to an Event Bus, handles the events by inserting or retrieving records from Azure Table Storage.
  The problem is, during peak hours almost every day it takes a lot of time to process a request made to Azure table storage, due to which there is a huge spike in the unprocessed events in the event bus.
  This is the timeout error:</p>
  
  <pre><code>{"EndOfTail":false,"UserFeeds":null,"Message":"The operation has timed out.","StackTrace":"   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Core\\Executor\\Executor.cs:line 649\r\n   at Microsoft.WindowsAzure.Storage.Table.TableQuery`1.&lt;&gt;c__DisplayClass7.&lt;ExecuteInternal&gt;b__6(IContinuationToken continuationToken) in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Table\\TableQuery.cs:line 430\r\n   at Microsoft.WindowsAzure.Storage.Core.Util.CommonUtility.&lt;LazyEnumerable&gt;d__0`1.MoveNext() in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\Common\\Core\\Util\\CommonUtility.cs:line 322\r\n   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()\r\n   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)\r\n   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)\r\n   at Rebtel.Service.UserFeed.DataLayer.UserFeedRepository.&lt;Execute&gt;d__31.MoveNext() in C:\\TeamCity\\buildAgent\\work\\18bc44b16693ee8a\\BackendServices\\Backend\\Rebtel.Service.UserFeed\\DataLayer\\UserFeedRepository.cs:line 531\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Rebtel.Service.UserFeed.DataLayer.UserFeedRepository.&lt;GetUserFeedsByTypes&gt;d__17.MoveNext() in C:\\TeamCity\\buildAgent\\work\\18bc44b16693ee8a\\BackendServices\\Backend\\Rebtel.Service.UserFeed\\DataLayer\\UserFeedRepository.cs:line 254\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Rebtel.Service.UserFeed.BusinessLayer.Actions.GetUserFeedAction.&lt;Execute&gt;d__7.MoveNext() in C:\\TeamCity\\buildAgent\\work\\18bc44b16693ee8a\\BackendServices\\Backend\\Rebtel.Service.UserFeed\\BusinessLayer\\Actions\\GetUserFeedAction.cs:line 68\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Rebtel.Service.UserFeed.UserFeedServiceImplementation.&lt;GetActions&gt;d__17.MoveNext() in C:\\TeamCity\\buildAgent\\work\\18bc44b16693ee8a\\BackendServices\\Backend\\Rebtel.Service.UserFeed\\UserFeedServiceImplementation.cs:line 182\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Rebtel.Platform.Service.ServiceRequestInterceptor.&lt;TryInvokeTaskWithServiceReplyResult&gt;d__10`1.MoveNext() in C:\\TeamCity\\buildAgent\\work\\18bc44b16693ee8a\\Platform\\Platform\\Rebtel.Platform.Backend\\Service\\ServiceRequestInterceptor.cs:line 0","Result":501}
  </code></pre>
  
  <p>And it took like 603,060 milliseconds to get this response.
  <strong>The problem is not with the scalability of the Azure storage account, because if we restart the windows service again with over a million events waiting it processes them instantly.</strong></p>
  
  <p>One of the queries looks like this:</p>
  
  <pre><code>query.WherePartitionKey(QueryComparisons.Equal, userId).AndRowKey(QueryComparisons.NotEqual, LastActionRowIdTableEntity.UniqueRowKey);
  </code></pre>
  
  <p>The strange thing is when I look at the monitoring graphs of the Azure table storage the average server latency is not more than 200ms.
  Do you guys have any idea why this is happening?  </p>
  
  <p>Edit: I have verified that we have taken into account all of the recommendations on Microsoft table storage documentation for performance.</p>
  
  <p>On the ServicePoint</p>
  
  <ol>
  <li>Nagle Algo is Off.</li>
  <li>Max connections are set to 1000.</li>
  </ol>