title: Efficiency : String Slice Vs Custom Function
body: <p>I have a large string with length of the order <code>5*10^6</code>.</p>
  <p>I have to do some processing by dividing it into blocks of 16 characters. I used a custom made function to split the string assuming its performance would be better than the splice approach.</p>
  <p>The functions are as follows :</p>
  <pre><code>def spliceSplitter(s):
       sum = 0
       while len(s) &gt; 0:
               block = s[:16]
               # Assuming the process to be done with data block is calculating its length.
               sum += len(block)
               s = s[16:]
       return sum
  </code></pre>
  <p>And the custom Function :</p>
  <pre><code>def normalSplitter(s):
       sum = 0
       l = len(s)
       data =&quot;&quot;
       for i in xrange(l):
               if i%16 == 0:
                       # Assuming the process to be done with data block is calculating its length.
                       sum += len(data)
                       data = &quot;&quot;
               data += s[i]
       return sum+len(data)
  </code></pre>
  <p>I used a cProfiler on both of them and the results are as follows (time in seconds) :</p>
  <pre><code>String Length     |  Splice Splitter   |  Normal Splitter
  ---------------------------------------------------------
  5000000           |  289.0             |  1.274 
  500000            |  0.592             |  0.134
  50000             |  0.25              |  0.28
  5000              |  0.001             |  0.003 
  </code></pre>
  <p>I am generating the string as follows :</p>
  <pre><code>s = ''.join([str(random.randint(1,9)) for x in xrange(5000000)])
  </code></pre>
  <p>My Question :</p>
  <ul>
  <li>Is there a pythonic way to get the same or better Efficiency as the Custom Normal Splitter? Maybe splitting the entire string before hand, storing it into a list and then operating it iteratively.</li>
  <li>Why is the performance for the Splice Splitter better for smaller strings ? (Just Curious about this one)</li>
  </ul>
  <p><strong>NOTE</strong>: The <code>process(data)</code> that I have to perform has no return value.</p>
  <h1>EDIT</h1>
  <p>Using Yield and improved Splice Splitter the performace resulted in the following :</p>
  <pre><code>String Length     |  Splice Splitter   |  Normal Splitter  |  Yield/Generator
  -------------------------------------------------------------------------------
  5000000           |  0.148             |  1.274            |  0.223
  500000            |  0.016             |  0.134            |  0.29
  50000             |  0.003             |  0.28             |  0.005
  5000              |  ~0.000            |  0.003            |  ~0.000
  </code></pre>
  <p>Code :</p>
  <pre><code>def pythonicSplitter(s):
       gen = (s[i:i+16] for i in xrange(0,len(s),16))
       sum = 0
       for data in gen:
               sum += len(data)
       return sum
  def spliceSplitter(s):
      sum = 0
      for x in xrange(0, len(s), 16):
           block = s[x:x+16]
           # Assuming the process to be done with data block is calculating its length.
           sum += len(block)
      return sum
  </code></pre>
  <p>Reason for improved Performance :</p>
  <ul>
  <li>The Splice Splitter was repeatedly <strong>creating the new string in each iteration</strong> using splice. <code>s = s[16:]</code> as pointed out in Patrick Collin's answer. This was leading to a Time Complexity of <code>~O(N^2)</code>.</li>
  <li>Once the repeated creation of the string <code>s</code> was replaced with <code>s[x:x+16]</code>, the code complexity was reduced to <code>O(N*16)</code> thus improving its performance by a huge margin. The Yield/Generator Function does the same thing (<code>pythonicSplitter()</code>), but due to the <strong>Large number of calls</strong> to the generator(iterators), the time taken to complete the operation is slighly larger than the one used in Splice Splitter.</li>
  <li>The Normal Splitter is also doing the same thing, by creating blocks of length 16. But since in Python the <strong>strings are immutable</strong>, the time complexity of creating these blocks is significantly larger than the inbuilt optimized slice function.</li>
  </ul>