title: How can I iteratively query a large MySQL dataset from within Solr's data importer?
body: <p>I have 256 tables in my production database. Each table has around 10,000,000 rows -- I can't get all of the rows from any one table in one <code>SELECT</code> query.</p>
  
  <p>Actually I put the query-phases in solr/conf/data-config.xml just like this as a test; it has one table and I'm selecting all rows.</p>
  
  <p>I use Solr (which is based on Java). And I trigger the indexing of the data like so:</p>
  
  <p><code>curl --max-time 36000 -i localhost:portNum/solr/dataimport?command=full-import</code></p>
  
  <p>This is what's in the XML:</p>
  
  <p><code>&lt;entity name="file" query="SELECT *, LOWER(REPLACE(sw_name, ' ', '')) packed_sw_name, CAST(group_id as char) group_id_s FROM tblTmp1"/&gt;</code></p>
  
  <p>But now, I want to get the parameter for table number(tblNum) for data-import and select rows between id ranges iteratively -- with dollar-sign variable substitution, the query would look like this:</p>
  
  <p><code>&lt;entity name="file" query="SELECT *, LOWER(REPLACE(sw_name, ' ', '')) packed_sw_name, CAST(group_id as char) group_id_s FROM tblTmp1_</code><strong>$tblNum</strong><code>where id &gt;=</code><strong>$startSn</strong><code>and id&lt;</code><strong>$endSn</strong><code>" /&gt;</code></p>
  
  <p>How can I query the database iteratively, and in which file should I set this up?</p>