title: Sphinx composite (distributed) big indexes
body: <p>Experience problem with indexing lot's of content data. Searching for the suitable solution.</p>
  
  <p>The logic if following:</p>
  
  <ul>
  <li><p>Robot is uploading content every day to the database.</p></li>
  <li><p>Sphinx index must reindex only new (daily) data. I.e. the previous content is never being changed.</p></li>
  </ul>
  
  <p>Sphinx delta indexing is an exact solution for this, but with too much content the error is rising: too many string attributes (current index format allows up to 4 GB).</p>
  
  <p>Distributed indexing seems to be usable, but how to dynamically (without dirty hacks) add &amp; split indexing data? </p>
  
  <ul>
  <li><p>I.e.: day 1 there are total 10000 rows, day 2 - 20000 rows and etc. The index throws >4GB error on about 60000 rows. </p></li>
  <li><p>The expected index flow: 1-5 day there is 1 index (no matter distributed or not), 6-10 day - 1 distributed (composite) index (50000 + 50000 rows) and so on.</p></li>
  <li><p>The question is how to fill distributed index dynamically?</p>
  
  <pre><code> Daily iteration sample:
       main index
          chunk1  - 50000 rows
          chunk2  - 50000 rows
          chunk3  - 35000 rows           
  
       delta index
          10000 new rows
  
       rotate "delta"
  
       merge "delta" into "main"
  </code></pre></li>
  </ul>
  
  <p>Please, advice. </p>