title: How can I debug the time used by a function?
body: <p>I am loading a big Pandas Data frame into memory. I can measure the time needed with the <code>%%time</code> magic:</p>
  
  <pre><code>%%time
  df = pd.read_csv(data_file)
  
  CPU times: user 3min 53s, sys: 57.5 s, total: 4min 50s
  Wall time: 21min 2s
  </code></pre>
  
  <p>There are about 16 minutes that the system is giving the CPU to other processes. I know that my process will not take 100% of the time (<a href="https://service.futurequest.net/index.php?/Knowledgebase/Article/View/407" rel="nofollow noreferrer">Difference between CPU time and wall time</a>). But, because this is a Virtual Machine used exclusively by me, I guess that most of the 16 minutes are related to processes involved with the data loading task.</p>
  
  <p>If I make a visual check of the resources used with <code>htop</code>, most of the cores are idle, and the system increases the amount of Swap memory slowly. I guess that the I/O operations to Swap are taking most of the time, but I want to confirm it.</p>
  
  <p>I would like to see if there is something I can do to improve the reading process, but I do not know how to look into the time spent inside the <code>read_csv</code> function.</p>
  
  <p><strong>Edit</strong>: Thanks to Ori's answer, I could figure out that most of the time was spent on Pandas's <code>internals.py _stack_arrays()</code> method and the <code>read</code> call of the object <code>pandas.parser.TextReader</code>. Were Pandas figuring out the column type? I tried providing a dictionary with types, and the load time was reduced to 3 minutes.</p>