title: Loading million rows into partitioned Stateful service
body: <p>I'm trying to load 20 million rows into partitioned stateful service ReliableDictionary. I partitioned stateful service into 10 partitions. Based on MSDN documentation, I understood that I need to use some hashing algorithm to find the correct partition and send data to it to load into IReliabledictionary. So I used the <a href="https://github.com/turowicz/Hydra" rel="nofollow noreferrer">Hydra</a> to get the partition number based on the value. All I'm storing is a <code>List&lt;long&gt;</code>  in the <code>IReliableDictionary</code>. </p>
  
  <p>So I created a Stateless service as wrapper, </p>
  
  <ol>
  <li>which will fetch the rows from the SQL Server (20 million), </li>
  <li>get the partition number using <a href="https://github.com/turowicz/Hydra" rel="nofollow noreferrer">Hydra</a> for each row,</li>
  <li>group them by partition number </li>
  <li>call the Stateful service for each partition using ServiceRemoting. However, I get <code>fabric message too large</code> exception if I send 1 million rows of data per each request so I chunked it into 100000 per request. </li>
  </ol>
  
  <p>This is taking 74 minutes for it to complete. This is too long. Below is the code for uploading -</p>
  
  <p>Please advise.</p>
  
  <pre><code> foreach (var itemKvp in ItemsDictionary)
              {
                  var ulnv2Uri = new Uri("fabric:/TestApp/dataservice");
  
  
                  //Insert to the correct shard based on the hash algorithm 
                  var dataService = _serviceProxyFactory.CreateServiceProxy&lt;IDataService&gt;(
                  dataStoreUri,
                   new ServicePartitionKey(itemKvp.Key), TargetReplicaSelector.PrimaryReplica, "dataServiceRemotingListener");
  
                  var itemsShard = itemKvp.Value;
                  //if the total records count is greater then 100000 then send it in chunks
                  if (itemsShard.Count &gt; 1_000_000)
                  {
                      //var tasks = new List&lt;Task&gt;();
                      var totalCount = itemsShard.Count;
                      var pageSize = 100000;
                      var page = 1;
                      var skip = 0;
                      while (skip &lt; totalCount)
                      {
                          await dataService.InsertData(itemsShard.Skip(skip).Take(pageSize).ToList());
                          page++;
                          skip = pageSize * (page - 1);
                      }
  
                  }
                  else
                  {
                     //otherwise send all together
                      await dataService.InsertData(itemsShard);
                  }
  
              }
  </code></pre>