title: Best way to index database table data in Solr?
body: <p>I have a table with around 100,000 rows at the moment. I want to index the data in this table in a Solr Index.</p>
  
  <p>So the naive method would be to:</p>
  
  <ul>
  <li>Get all the rows</li>
  <li>For each row: convert to a SolrDocument and add each document to a request</li>
  <li>Once all rows are converted then post the request</li>
  </ul>
  
  <p>Some problems with this approach that I can think of are:</p>
  
  <ul>
  <li>Loading too much data (the content of the whole table) in to memory</li>
  <li>POSTing a big request</li>
  </ul>
  
  <p>However, some advantages:</p>
  
  <ul>
  <li>Only one request to the Database</li>
  <li>Only one POST request to Solr</li>
  </ul>
  
  <p>The approach is not scalable, I see that since as the table grows so will the memory requirements and the size of the POST request. I need to perhaps take <code>n</code> number of rows, process them, then take the next <code>n</code>?</p>
  
  <p>I'm wondering if any one has any advice about how to best implement this?</p>
  
  <p>(ps. I did search the site but I didn't find any questions that were similar to this.)</p>
  
  <p>Thanks.</p>