title: How to trace random MemoryError in python script?
body: <p>I have a python script, which is used to perform a lab measurement using several devices. The whole setup is rather involved, including communication over serial devices, API calls as well as the use of self-written and commercial drivers. In the end, however, everything boils down to two nested loops, which vary some parameters, collect data and write it to a file.</p>
  
  <p>My problem is that I observe random occurences of a <a href="https://docs.python.org/3.4/library/exceptions.html#MemoryError" rel="nofollow noreferrer"><code>MemoryError</code></a>, typically after 10 hours, equivalent to ~15k runs of the loops. At the moment, I don't have an idea, where it comes from or how I can trace it further. So I would be happy for suggestions, how to work on my problem. My observations up to this moment are as follows.</p>
  
  <ul>
  <li><p>The error occurs at random states of the program. Different runs will throw the <code>MemoryError</code> at different lines of my script.</p></li>
  <li><p>There is never any helpful error message. Python only says <code>MemoryError</code> without any error string. The traceback leads me to some point in the script, where memory is needed (e.g. when building a list), but it appears to be no specific instruction, which is the problem.</p></li>
  <li><p>My RAM is far from full. The python process in question typically consumes some ten MB of RAM when viewed in the task manager. In addition, the RAM usage appears to be stable for hours. Usually, it increases slowly for some time, just to drop to down to the previous level quickly, which I interpret as the garbage collector kicking in periodically.</p></li>
  <li><p>So far I did not find any indications for a memory leak. I used <a href="https://pypi.python.org/pypi/memory_profiler" rel="nofollow noreferrer"><code>memory_profiler</code></a> to trace the memory usage of my functions and found it to be stable. In addition, I followed <a href="http://mg.pov.lt/blog/hunting-python-memleaks" rel="nofollow noreferrer">this</a> blog entry to observe what the  garbage collector does in detail. Again, I could not find any hints for undeleted objects.</p></li>
  <li><p>I am stuck to Win7 x86 due to a driver, which will only work on a 32bit system. So I cannot follow suggestions like <a href="https://stackoverflow.com/a/4441958/5804088">this</a> to go to a 64 bit version of Windows. Anyway, I do not see, how this would help in my situation.</p></li>
  <li><p>The iPython console, from which the script is being launched, often behaves strange after the error occurred. Sometimes, a new <code>MemoryError</code> is thrown even for very simple operations. Often, the console is marked by Windows as "not responding" after some time. A menu pops up, where besides the usual options to wait for the process or to terminate it, there is a third option to "restore" the program (whatever that means). Doing so usually causes the console to work normal again.</p></li>
  </ul>
  
  <p>At this point, I am somewhat out of ideas on how to proceed. The general receipe to comment out parts of the script until it works is highly undesirable in my case. As stated above, each test run will take several hours, meaning a potential downtime of weeks for my lab equipment. Going that direction, appears unfeasable to me. Is there any more direct approach to learn, what is crashing behind the scenes? How can I understand that python apparently fails to malloc?</p>