title: Python Pandas: What causes slowdown in different column selection methods?
body: <p>After seeing <a href="https://stackoverflow.com/questions/8916302/selecting-across-multiple-columns-with-python-pandas">this question</a> about replicating SQL select-statement-like behavior in Pandas, I added <a href="https://stackoverflow.com/a/13419324/567620">this answer</a> showing two ways that could shorten the verbose syntax given in <a href="https://stackoverflow.com/a/8916746/567620">the accepted answer</a> to that question.</p>
  
  <p>After playing around with them, my two shorter-syntax methods are significantly slower, and I am hoping someone can explain why</p>
  
  <p>You can assume any functions used below are either from Pandas, IPython, or from <a href="https://stackoverflow.com/questions/8916302/selecting-across-multiple-columns-with-python-pandas">the question and answers linked above</a>.</p>
  
  <pre><code>import pandas
  import numpy as np
  N = 100000
  df = pandas.DataFrame(np.round(np.random.rand(N,5)*10))
  
  def pandas_select(dataframe, select_dict):
      inds = dataframe.apply(lambda x: reduce(lambda v1,v2: v1 and v2,
                             [elem[0](x[key], elem[1])
                             for key,elem in select_dict.iteritems()]), axis=1)
      return dataframe[inds]
  
  
  
  %timeit _ = df[(df[1]==3) &amp; (df[2]==2) &amp; (df[4]==5)]
  %timeit _ = df[df.apply(lambda x: (x[1]==3) &amp; (x[2]==2) &amp; (x[4]==5), axis=1)]
  
  import operator
  select_dict = {1:(operator.eq,3), 2:(operator.eq,2), 4:(operator.eq,5)}
  %timeit _ = pandas_select(df, select_dict)
  </code></pre>
  
  <p>The output I get is:</p>
  
  <pre><code>In [6]: %timeit _ = df[(df[1]==3) &amp; (df[2]==2) &amp; (df[4]==5)]
  100 loops, best of 3: 4.91 ms per loop
  
  In [7]: %timeit _ = df[df.apply(lambda x: (x[1]==3) &amp; (x[2]==2) &amp; (x[4]==5), axis=1)]
  1 loops, best of 3: 1.23 s per loop
  
  In [10]: %timeit _ = pandas_select(df, select_dict)
  1 loops, best of 3: 1.6 s per loop
  </code></pre>
  
  <p>I can buy that the user of <code>reduce</code>, <code>operator</code> functions, and just the function overhead from my <code>pandas_select</code> function could slow it down. But it seems excessive. Inside of my function, I'm using the same syntax, <code>df[key] logical_op value</code>, but it's much slower.</p>
  
  <p>I'm also puzzled why the <code>apply</code> version along <code>axis=1</code> is so much slower. It should literally be just a shortening of the syntax, no?</p>