title: Python code hangs with CPU utilization falling to 0% when dealing with large pandas dataframe
body: <p>I am working with a large pandas dataframe (around 180M rows, 6 columns, 6 GB) and I am facing a problem where an operation will suddenly stall at the python console, without any intimation. </p>
  
  <p>The only indication of the problem is the windows resource monitor showing 0% CPU utilization of the python process, although the memory utilization of the same process will be high (99%+). The disk usage will also be at 0 MB/s.</p>
  
  <p>After leaving the system as-it-is for around 10 - 15 minutes, the resource monitor will show a drop in memory utilization % as the python process is removed from the list. Meanwhile, the python console still hasn't finished and I don't see the ">>>" prompt come up.</p>
  
  <p>The Python code is probably not at fault, since I can run the command (a groupby on a column followed by transform('count')) on the first 5 rows of the dataframe via .head()</p>
  
  <p>I think this is due to a memory problem... I should point out that this doesn't happen immediately. On executing the code, the resource monitor shows memory usage steadily creep up and then consequently fall (probably due to swapping with disk, as there is a corresponding increase in disk utilization %). This happens multiple times (all the while showing CPU utilization in the 15% - 30% range) before halting (0% CPU and memory % maxed out).</p>
  
  <p>If it's relevant, I am using PyCharm IDE to run the code. I have a python console open within PyCharm, where I am running the command. My machine has 16 GB RAM.</p>
  
  <p>Any suggestions on how to troubleshoot this? My main gripe is the lack of useful error messages.</p>
  
  <p><strong>Similar Question:</strong></p>
  
  <p><a href="https://stackoverflow.com/questions/41942960/python-randomly-drops-to-0-cpu-usage-causing-the-code-to-hang-up-when-handl">Here</a> is a similar question asked earlier on SO. The root cause in that case was that the file was stored on a network drive, which was causing latency issues. In my case, the file is stored locally.</p>