title: Azure Table Increased Latency
body: <p>I'm trying to create an app which can efficiently write data into Azure Table. In order to test storage performance, I created a simple console app, which sends hardcoded entities in a loop. Each entry is 0.1 kByte. Data is sent in batches (100 items in each batch, 10 kBytes each batch). For every batch, I prepare entries with the same partition key, which is generated by incrementing a global counter - so I never send more than one request to the same partition. Also, I control a degree of parallelism by increasing/decreasing the number of threads. Each thread sends batches synchronously (no request overlapping).
  If I use 1 thread, I see 5 requests per second (5 batches, 500 entities). At that time Azure portal metrics shows table latency below 100ms - which is quite good.
  If I increase the number of treads up to 12 I see x12 increase in outgoing requests. This rate stays stable for a few minutes. But then, for some reason I start being throttled - I see latency increase and requests amount drop.</p>
  
  <p>Below you can see account metrics - highlighted point shows 2K31 transactions (batches) per minute. It is 3850 entries per second. If threads are increased up to 50, then latency increases up to 4 seconds, and transaction rate drops to 700 requests per second.</p>
  
  <p><a href="https://i.stack.imgur.com/RvQnV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RvQnV.png" alt="account metrics"></a></p>
  
  <p>According to <a href="https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets" rel="nofollow noreferrer">documentation</a>, I should be able to send up to 20K transaction per second within one account (my test account is used only for my performance test). 20K batches mean 200K entries. So the question is why I'm being throttled after 3K entries?</p>
  
  <p>Test details:</p>
  
  <ul>
  <li>Azure Datacenter: West US 2. </li>
  <li>My location: Los Angeles.</li>
  <li>App is written in C#, uses <a href="https://www.nuget.org/packages/Microsoft.Azure.CosmosDB.Table/" rel="nofollow noreferrer">CosmosDB.Table nuget</a> with the following configuration: ServicePointManager.DefaultConnectionLimit = 250, Nagles Algorithm is disabled.</li>
  <li>Host machine is quite powerful with 1Gb internet link (i7, 8 cores, no high CPU, no high memory is observed during the test).</li>
  </ul>
  
  <p>PS: I've read docs</p>
  
  <blockquote>
    <p>The system's ability to handle a sudden burst of traffic to a partition is limited by the scalability of a single partition server until the load balancing operation kicks-in and rebalances the partition key range.</p>
  </blockquote>
  
  <p>and waited for 30 mins, but the situation didn't change.</p>
  
  <p><strong>EDIT</strong>
  I got a comment that E2E Latency doesn't reflect server problem.
  So below is a new graph which shows not only E2E latency but also the server's one. As you can see they are almost identical and that makes me think that the source of the problem is not on the client side.</p>
  
  <p><a href="https://i.stack.imgur.com/oVCl1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/oVCl1.png" alt="new graph"></a></p>