title: How to slice very large dataframe by days without walking the entire dataframe each time?
body: <p>I have two massive datetime indexed and sorted dataframes that I need compare groups from one to groups from another.</p>
  <pre><code>start, end = df.index.min(), df.index.max()
  
  for day in pd.date_range(start.date(), end.date()+a_day, freq='D'):
       current_df = df[df.index.date == day.date()]
       current_df2= df2[df2.index.date == day.date()]
  
       do_heavy_lift(df, df2)
  </code></pre>
  <p>I thought the heavy lift would take most of the time but the slicing by date takes &gt; 95% of the time.</p>
  <p>Upon reflection, each time I slice a dataframe it is probably walking the entire index.</p>
  <p>Is there any way to significantly improve this approach? The indexes are sorted. Can I:</p>
  <ol>
  <li>Tell it to stop searching after the end of the current day?</li>
  <li>Remember where it was at the end of the previous day and start there for the next day?</li>
  </ol>
  <p>You can create a sample dataset using:</p>
  <pre><code>import numpy as np
  from datetime import datetime, timedelta
  
  date_today = datetime.now()
  days = pd.date_range(date_today, date_today + timedelta(7000), freq='H')
  
  np.random.seed(seed=1111)
  data = np.random.randint(1, high=100, size=len(days))
  df = pd.DataFrame({'test': days, 'col2': data})
  df = df.set_index('test')
  
  np.random.seed(seed=1345)
  data = np.random.randint(1, high=100, size=len(days))
  df2 = pd.DataFrame({'test': days, 'col2': data})
  df2 = df2.set_index('test')
  df.sort_index(inplace=True)
  df2.sort_index(inplace=True)
  </code></pre>
  <blockquote>
  <p>CPU times: user 9min 50s, sys: 4.27 s, total: 9min 54s
  Wall time: 10min 45s</p>
  </blockquote>