text: user3848: I discussed a question with @gaspanic Python/Biopython - Replace amino acid residue on MSA with "z" from a list of unaligned positions
  . The issue emerged was speeding up lists in Python. The solutions we discussed were
  
  Raw code
  Predefining a list size
  deque
  
  
  
  
  
  1. @gaspanic
  my code
  2. @gaspanic
  my code
  3. @gaspanic
  my code
  
  
  
  
  3.45-3.48
  7.17-7.2
  4.38-4.85
  7.09-7.23
  3.44-3.53
  7.21-7.39
  
  
  
  
  Run timings are seconds (3 replicated) above with @gaspanic's code being faster.
  What improvements would speed this up?
  Here I extended the data set
  from itertools import repeat
  def iterit(x):
      seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
      return ''.join(list(repeat(seq, x)))
  
  if __name__ == '__main__':
      seqE3 = iterit(2800) # length is 201600
      seqE4 = iterit(28000) # length is 2016000
      seqE5 = iterit(280000) # length is 20160000
  
  @gaspanic code
  for seqbit in [seqE3,seqE4,seqE5]:
      seq = seqbit
      pos = [2, 5, 10, 11, 14]    
      new_seq = []
      count = 0
      for s in seq:
          if s == "-":
              new_seq.append(s)
          else:
              count += 1
              if count in pos:
                  new_seq.append('Z')
              else:
                  new_seq.append(s)
     # print(''.join(new_seq))
  
  My code:
  # same variables and loop structure
  newseq = list()
  count = 0
  for n,s in enumerate(prot):    
      if str(s) == '-':
          count += 1
          s = '-'
      else:
          if (n + 1 - count) in pos:
              s = 'Z'
      newseq.append(s)
  # print(''.join(newseq))
  
  Predefining lists code (additional counter needed for @gaspanic's code:
  # same variables and loop structure
  new_seq = [] * len(seq)
  count = 0
  for n,s in enumerate(prot):    
      if str(s) == '-':
          count += 1
          s = '-'
      else:
          if (n + 1 - count) in pos:
              s = 'Z'
      newseq[n] = s
  
  Deque code simply any of the above substituting
  newseq = list() 
  
  with
  from collections import deque 
  newseq = deque()
  
  Suggestions for efficiency improvements welcome.
  A solution in Cython code would be cool. PyPI is not sought here.
  
  user3848: Huge thanks to @Chris_Rands, this solution increases runtime by ~ 0.5 seconds uniformly across all coded solutions, so this code has a runtime of just over 3 seconds runtime on the benchmark machine. I incorporated my solution with @Chris_Rands solution and the final runtime I got was:
  @gaspanic
  
  1.7 - 1.8 seconds
  
  This being twice the fastest speed in the question and four times the speed of my raw code.
  My code
  
  4.4 - 4.5 seconds
  
  However a unique combination is needed:
  
  tuple (Chris Rands)
  set - important (Chris Rands)
  an iterator
  a list comprehension
  
  In particular 3. and 4. are needed in combination. This is explained below.
  def iterit(x):
      seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
      return tuple(''.join(list(repeat(seq, x))))
  
  def mIterit(seqE3,seqE4,seqE5):
      pos = {2, 5, 10, 11, 14}
      for seqbit in (seqE3,seqE4,seqE5):
          seq = seqbit
          count = 0
          # loc = 0
          for s in seq:
              if s == "-":
                  yield '-'
              else:
                  count += 1
                  if count in pos:
                      yield 'Z'
                  else:
                      yield s
  
  f __name__ == '__main__':
      
      seqE3 = iterit(2800) # length is 201600
      seqE4 = iterit(28000) # length is 2016000
      seqE5 = iterit(280000) # length is 20160000
      
      listComp = [i for i in mIterit(seqE3,seqE4,seqE5)]
      
  
  The list comprehension (variable listComp) is the very last line and is crucial to the solution otherwise there is little increase in speed to @Chris_Rand's code. This alternative is to replace the last line with:
  newseq = list()
  for i in mIterit(seqE3,seqE4,seqE5):
      newseq.append(i)   
  
  A simple:
  for i in mIterit(seqE3,seqE4,seqE5):
      pass
  
  Is the same speed as the list comprehension, but its of no value because the output isn't saved to anything.
  I was impressed by the interaction between the iterator and list comprehension. The other notable influence on speed is the set described by @Chris_Rand.
  How does it work? tuples do increase speed by removing some functional of a list (immutability). Here the gain is small. Placing the pos variable outside the loop speaks for itself (0.1 seconds gain). set is interesting.
  The iterator spews out a stream of results via yield which is where it attains its speed. The problem is capturing that stream outside the iterator is expensive. However list comprehensions are faster than standard loops and in this scenerio carry little/no detectable overhead. I was impressed. Up to this point I thought list comprehensions were a bit faster and a lot 'cooler' than loops. I hadn't understood there was a synergy with an iterator.
  Is it all worth it? Python can easily achieve faster run times than Java using Cython but I will place the discussion in the chat forum next week.
  
  user104: I'd start by saying that Python is inherently slow and this feels like a case of perhaps unnecessary optimisation. With that in mind here are my suggested (untested) optimisations of @gaspanic code:
  pos = {2, 5, 10, 11, 14} # make pos a set for O(1) member testing, and don't define inside the loop
  for seq in (seqE3,seqE4,seqE5): # Define seq here rather than inside the loop, and use an immutable tuple instead of a mutable list  
      new_seq = []
      count = 0
      for s in seq:
          if s == "-":
              new_seq.append(s)
          else:
              count += 1
              new_seq.append('Z' if count in pos else s) # use ternary operator
  
  
original_text: 3848: <p>I discussed a question with @gaspanic <a href="https://bioinformatics.stackexchange.com/questions/14625/python-biopython-replace-amino-acid-residue-on-msa-with-z-from-a-list-of-una">Python/Biopython - Replace amino acid residue on MSA with &quot;z&quot; from a list of unaligned positions</a>
  . The issue emerged was speeding up lists in Python. The solutions we discussed were</p>
  <ol>
  <li>Raw code</li>
  <li>Predefining a list size</li>
  <li><code>deque</code></li>
  </ol>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">1. @gaspanic</th>
  <th style="text-align: left;">my code</th>
  <th style="text-align: left;">2. @gaspanic</th>
  <th style="text-align: left;">my code</th>
  <th style="text-align: left;">3. @gaspanic</th>
  <th style="text-align: left;">my code</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">3.45-3.48</td>
  <td style="text-align: left;">7.17-7.2</td>
  <td style="text-align: left;">4.38-4.85</td>
  <td style="text-align: left;">7.09-7.23</td>
  <td style="text-align: left;">3.44-3.53</td>
  <td style="text-align: left;">7.21-7.39</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Run timings are seconds (3 replicated) above with @gaspanic's code being faster.</p>
  <p>What improvements would speed this up?</p>
  <p>Here I extended the data set</p>
  <pre><code>from itertools import repeat
  def iterit(x):
      seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
      return ''.join(list(repeat(seq, x)))
  
  if __name__ == '__main__':
      seqE3 = iterit(2800) # length is 201600
      seqE4 = iterit(28000) # length is 2016000
      seqE5 = iterit(280000) # length is 20160000
  </code></pre>
  <p>@gaspanic code</p>
  <pre><code>for seqbit in [seqE3,seqE4,seqE5]:
      seq = seqbit
      pos = [2, 5, 10, 11, 14]    
      new_seq = []
      count = 0
      for s in seq:
          if s == &quot;-&quot;:
              new_seq.append(s)
          else:
              count += 1
              if count in pos:
                  new_seq.append('Z')
              else:
                  new_seq.append(s)
     # print(''.join(new_seq))
  </code></pre>
  <p>My code:</p>
  <pre><code># same variables and loop structure
  newseq = list()
  count = 0
  for n,s in enumerate(prot):    
      if str(s) == '-':
          count += 1
          s = '-'
      else:
          if (n + 1 - count) in pos:
              s = 'Z'
      newseq.append(s)
  # print(''.join(newseq))
  </code></pre>
  <p>Predefining lists code (additional counter needed for @gaspanic's code:</p>
  <pre><code># same variables and loop structure
  new_seq = [] * len(seq)
  count = 0
  for n,s in enumerate(prot):    
      if str(s) == '-':
          count += 1
          s = '-'
      else:
          if (n + 1 - count) in pos:
              s = 'Z'
      newseq[n] = s
  </code></pre>
  <p>Deque code simply any of the above substituting</p>
  <pre><code>newseq = list() 
  </code></pre>
  <p>with</p>
  <pre><code>from collections import deque 
  newseq = deque()
  </code></pre>
  <p>Suggestions for efficiency improvements welcome.</p>
  <p>A solution in <code>Cython</code> code would be cool. <code>PyPI</code> is not sought here.</p>
   3848: <p>Huge thanks to @Chris_Rands, this solution increases runtime by ~ 0.5 seconds uniformly across all coded solutions, so this code has a runtime of just over 3 seconds runtime on the benchmark machine. I incorporated my solution with @Chris_Rands solution and the final runtime I got was:</p>
  <p>@gaspanic</p>
  <blockquote>
  <p>1.7 - 1.8 seconds</p>
  </blockquote>
  <p>This being twice the fastest speed in the question and four times the speed of my raw code.</p>
  <p>My code</p>
  <blockquote>
  <p>4.4 - 4.5 seconds</p>
  </blockquote>
  <p>However a unique combination is needed:</p>
  <ol>
  <li><code>tuple</code> (Chris Rands)</li>
  <li><code>set</code> - important (Chris Rands)</li>
  <li>an <code>iterator</code></li>
  <li>a <code>list comprehension</code></li>
  </ol>
  <p>In particular 3. and 4. are needed in combination. This is explained below.</p>
  <pre><code>def iterit(x):
      seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
      return tuple(''.join(list(repeat(seq, x))))
  
  def mIterit(seqE3,seqE4,seqE5):
      pos = {2, 5, 10, 11, 14}
      for seqbit in (seqE3,seqE4,seqE5):
          seq = seqbit
          count = 0
          # loc = 0
          for s in seq:
              if s == &quot;-&quot;:
                  yield '-'
              else:
                  count += 1
                  if count in pos:
                      yield 'Z'
                  else:
                      yield s
  
  f __name__ == '__main__':
      
      seqE3 = iterit(2800) # length is 201600
      seqE4 = iterit(28000) # length is 2016000
      seqE5 = iterit(280000) # length is 20160000
      
      listComp = [i for i in mIterit(seqE3,seqE4,seqE5)]
      
  </code></pre>
  <p>The list comprehension (variable <code>listComp</code>) is the very last line and is crucial to the solution otherwise there is little increase in speed to @Chris_Rand's code. This alternative is to replace the last line with:</p>
  <pre><code>newseq = list()
  for i in mIterit(seqE3,seqE4,seqE5):
      newseq.append(i)   
  </code></pre>
  <p>A simple:</p>
  <pre><code>for i in mIterit(seqE3,seqE4,seqE5):
      pass
  </code></pre>
  <p>Is the same speed as the list comprehension, but its of no value because the output isn't saved to anything.</p>
  <p>I was impressed by the interaction between the <code>iterator</code> and <code>list comprehension</code>. The other notable influence on speed is the <code>set</code> described by @Chris_Rand.</p>
  <p>How does it work? <code>tuples</code> do increase speed by removing some functional of a <code>list</code> (immutability). Here the gain is small. Placing the <code>pos</code> variable outside the loop speaks for itself (0.1 seconds gain). <code>set</code> is interesting.</p>
  <p>The <code>iterator</code> spews out a stream of results via <code>yield</code> which is where it attains its speed. The problem is capturing that stream outside the <code>iterator</code> is expensive. However <code>list comprehensions</code> are faster than standard loops and in this scenerio carry little/no detectable overhead. I was impressed. Up to this point I thought <code>list comprehensions</code> were a bit faster and a lot 'cooler' than loops. I hadn't understood there was a synergy with an <code>iterator</code>.</p>
  <p>Is it all worth it? <code>Python</code> can easily achieve faster run times than <code>Java</code> using Cython but I will place the discussion in the <em>chat</em> forum next week.</p>
   104: <p>I'd start by saying that Python is inherently slow and this feels like a case of perhaps unnecessary optimisation. With that in mind here are my suggested (untested) optimisations of @gaspanic code:</p>
  <pre><code>pos = {2, 5, 10, 11, 14} # make pos a set for O(1) member testing, and don't define inside the loop
  for seq in (seqE3,seqE4,seqE5): # Define seq here rather than inside the loop, and use an immutable tuple instead of a mutable list  
      new_seq = []
      count = 0
      for s in seq:
          if s == &quot;-&quot;:
              new_seq.append(s)
          else:
              count += 1
              new_seq.append('Z' if count in pos else s) # use ternary operator
  </code></pre>
  