title: profiling the code with memory_profiler increasing the execution time
body: <p>I am writing a simple application which splits a large text file into smaller files, and i have written 2 versions of it, one using lists and one using generators. I profiled both the versions using memory_profiler module and it clearly showed the better memory efficiency  of the generators version, however strangely enough when then version using generators was profiled, it increases the execution time. Below demonstration explains what i mean</p>
  
  <p>Version using Lists</p>
  
  <pre><code>from memory_profiler import profile
  
  
  @profile()
  def main():
      file_name = input("Enter the full path of file you want to split into smaller inputFiles: ")
      input_file = open(file_name).readlines()
      num_lines_orig = len(input_file)
      parts = int(input("Enter the number of parts you want to split in: "))
      output_files = [(file_name + str(i)) for i in range(1, parts + 1)]
      st = 0
      p = int(num_lines_orig / parts)
      ed = p
      for i in range(parts-1):
          with open(output_files[i], "w") as OF:
              OF.writelines(input_file[st:ed])
          st = ed
          ed = st + p
  
      with open(output_files[-1], "w") as OF:
          OF.writelines(input_file[st:])
  
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>when run with profiler</p>
  
  <pre><code>$ time py36 Splitting\ text\ files_BAD_usingLists.py                                                                                                               
  
  Enter the full path of file you want to split into smaller inputFiles: /apps/nttech/rbhanot/Downloads/test.txt
  Enter the number of parts you want to split in: 3
  Filename: Splitting text files_BAD_usingLists.py
  
  Line #    Mem usage    Increment   Line Contents
  ================================================
       6     47.8 MiB      0.0 MiB   @profile()
       7                             def main():
       8     47.8 MiB      0.0 MiB       file_name = input("Enter the full path of file you want to split into smaller inputFiles: ")
       9    107.3 MiB     59.5 MiB       input_file = open(file_name).readlines()
      10    107.3 MiB      0.0 MiB       num_lines_orig = len(input_file)
      11    107.3 MiB      0.0 MiB       parts = int(input("Enter the number of parts you want to split in: "))
      12    107.3 MiB      0.0 MiB       output_files = [(file_name + str(i)) for i in range(1, parts + 1)]
      13    107.3 MiB      0.0 MiB       st = 0
      14    107.3 MiB      0.0 MiB       p = int(num_lines_orig / parts)
      15    107.3 MiB      0.0 MiB       ed = p
      16    108.1 MiB      0.7 MiB       for i in range(parts-1):
      17    107.6 MiB     -0.5 MiB           with open(output_files[i], "w") as OF:
      18    108.1 MiB      0.5 MiB               OF.writelines(input_file[st:ed])
      19    108.1 MiB      0.0 MiB           st = ed
      20    108.1 MiB      0.0 MiB           ed = st + p
      21                             
      22    108.1 MiB      0.0 MiB       with open(output_files[-1], "w") as OF:
      23    108.1 MiB      0.0 MiB           OF.writelines(input_file[st:])
  
  
  
  real    0m6.115s
  user    0m0.764s
  sys     0m0.052s
  </code></pre>
  
  <p>When run without profiler</p>
  
  <pre><code>$ time py36 Splitting\ text\ files_BAD_usingLists.py 
  Enter the full path of file you want to split into smaller inputFiles: /apps/nttech/rbhanot/Downloads/test.txt
  Enter the number of parts you want to split in: 3
  
  real    0m5.916s
  user    0m0.696s
  sys     0m0.080s
  </code></pre>
  
  <p>Now the one using generators</p>
  
  <pre><code>@profile()
  def main():
      file_name = input("Enter the full path of file you want to split into smaller inputFiles: ")
      input_file = open(file_name)
      num_lines_orig = sum(1 for _ in input_file)
      input_file.seek(0)
      parts = int(input("Enter the number of parts you want to split in: "))
      output_files = ((file_name + str(i)) for i in range(1, parts + 1))
      st = 0
      p = int(num_lines_orig / parts)
      ed = p
      for i in range(parts-1):
          file = next(output_files)
          with open(file, "w") as OF:
              for _ in range(st, ed):
                  OF.writelines(input_file.readline())
  
              st = ed
              ed = st + p
              if num_lines_orig - ed &lt; p:
                  ed = st + (num_lines_orig - ed) + p
              else:
                  ed = st + p
  
      file = next(output_files)
      with open(file, "w") as OF:
          for _ in range(st, ed):
              OF.writelines(input_file.readline())
  
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>When run with profiler option</p>
  
  <pre><code>$ time py36 -m memory_profiler Splitting\ text\ files_GOOD_usingGenerators.py                                                                                                                                      
  Enter the full path of file you want to split into smaller inputFiles: /apps/nttech/rbhanot/Downloads/test.txt
  Enter the number of parts you want to split in: 3
  Filename: Splitting text files_GOOD_usingGenerators.py
  
  Line #    Mem usage    Increment   Line Contents
  ================================================
       4   47.988 MiB    0.000 MiB   @profile()
       5                             def main():
       6   47.988 MiB    0.000 MiB       file_name = input("Enter the full path of file you want to split into smaller inputFiles: ")
       7   47.988 MiB    0.000 MiB       input_file = open(file_name)
       8   47.988 MiB    0.000 MiB       num_lines_orig = sum(1 for _ in input_file)
       9   47.988 MiB    0.000 MiB       input_file.seek(0)
      10   47.988 MiB    0.000 MiB       parts = int(input("Enter the number of parts you want to split in: "))
      11   48.703 MiB    0.715 MiB       output_files = ((file_name + str(i)) for i in range(1, parts + 1))
      12   47.988 MiB   -0.715 MiB       st = 0
      13   47.988 MiB    0.000 MiB       p = int(num_lines_orig / parts)
      14   47.988 MiB    0.000 MiB       ed = p
      15   48.703 MiB    0.715 MiB       for i in range(parts-1):
      16   48.703 MiB    0.000 MiB           file = next(output_files)
      17   48.703 MiB    0.000 MiB           with open(file, "w") as OF:
      18   48.703 MiB    0.000 MiB               for _ in range(st, ed):
      19   48.703 MiB    0.000 MiB                   OF.writelines(input_file.readline())
      20                             
      21   48.703 MiB    0.000 MiB               st = ed
      22   48.703 MiB    0.000 MiB               ed = st + p
      23   48.703 MiB    0.000 MiB               if num_lines_orig - ed &lt; p:
      24   48.703 MiB    0.000 MiB                   ed = st + (num_lines_orig - ed) + p
      25                                         else:
      26   48.703 MiB    0.000 MiB                   ed = st + p
      27                             
      28   48.703 MiB    0.000 MiB       file = next(output_files)
      29   48.703 MiB    0.000 MiB       with open(file, "w") as OF:
      30   48.703 MiB    0.000 MiB           for _ in range(st, ed):
      31   48.703 MiB    0.000 MiB               OF.writelines(input_file.readline())
  
  
  
  real    1m48.071s
  user    1m13.144s
  sys     0m19.652s
  </code></pre>
  
  <p>When run without profiler </p>
  
  <pre><code>$ time py36  Splitting\ text\ files_GOOD_usingGenerators.py 
  Enter the full path of file you want to split into smaller inputFiles: /apps/nttech/rbhanot/Downloads/test.txt
  Enter the number of parts you want to split in: 3
  
  real    0m10.429s
  user    0m3.160s
  sys     0m0.016s
  </code></pre>
  
  <p>So why profiling is making my code slow first of all ? Secondly if at profiling impacts execution speed, then why this effect is not showing on version of the code using lists.</p>