title: Dropping discontinuous time series from dataframe - speed improvement?
body: <p>My dataframe contains multiple time series of various assets, for many users. I posted here an example df, with a time series of percent change in price of 2 assets. (Only 1 user shown)</p>
  <pre><code>             userId asset  pctChange
  timestamp
  2018-05-27   12345   BTC       -2.0
  2018-05-28   12345   BTC       -1.0
  2018-05-29   12345   BTC       -1.0
  2018-06-03   12345   BTC        8.0
  2018-06-04   12345   BTC       -2.0
  2018-06-05   12345   BTC       -2.0
  2018-05-27   12345   ETH       -1.0
  2018-05-28   12345   ETH       -9.0
  2018-05-29   12345   ETH       -2.0
  2018-05-30   12345   ETH       16.0
  2018-05-31   12345   ETH        0.0
  2018-06-05   12345   ETH       -3.0
  </code></pre>
  <p>The aim is to take only the most recent parts of the time series, until there is a gap of at least 1 day.</p>
  <p>In this example case, for user 12345, the ETH time series I  keep is from 2018-05-27 until 2018-05-29 while for BTC it's from 2018-05-27 until 2018-05-31.</p>
  <p>This is my current solution, and it works... however, it is slow!</p>
  <pre><code>def RemoveDiscontinuous(df):
      mask = df.index.to_series().diff() &gt; pd.Timedelta(days=1) 
      if sum(mask) ==0: 
          cut_off = df.index.min() 
      else:
          cut_off = mask.sort_index(ascending=False).idxmax()
      most_recent = df.index.max()
      series_result = df[(df.index &lt; most_recent) &amp;  (df.index &gt; cut_off)] 
      return series_result
  
  series_assets_today_continuous = series_assets_today.groupby([&quot;userId&quot;, &quot;asset&quot;]).apply(lambda x: RemoveDiscontinuous(x))
  </code></pre>
  <p>I first groupby userId, and asset, then find the first “true” in mask showing gaps &gt;= 1 day (i.e. where the series becomes discontinuous). Find first timepoint where the break happens. If there is no gap, and the series is continuous, just take the last value as cut off (i.e. entire time series). Then just find all the entries between the most recent, and the first gap.</p>
  <p>This runs painfully slow for my 2 million row dataframe. Any ideas how this could be done more efficiently?</p>
  <p>PS. Here is the dataframe to copy:</p>
  <pre><code>import pandas as pd
  pd.DataFrame({'timestamp': {0: '2018-05-27',  1: '2018-05-28',  2: '2018-05-29',  3: '2018-06-03',  4: '2018-06-04',  5: '2018-06-05',  6: '2018-05-27',  7: '2018-05-28',  8: '2018-05-29',  9: '2018-05-30',  10: '2018-05-31',  11: '2018-06-05'}, 'userId': {0: 12345,  1: 12345,  2: 12345,  3: 12345,  4: 12345,  5: 12345,  6: 12345,  7: 12345,  8: 12345,  9: 12345,  10: 12345,
    11: 12345}, 'asset': {0: 'BTC',  1: 'BTC',  2: 'BTC',  3: 'BTC',  4: 'BTC',  5: 'BTC',  6: 'ETH',  7: 'ETH',  8: 'ETH',  9: 'ETH',  10: 'ETH',  11: 'ETH'}, 'pctChange': {0: -2.0,
    1: -1.0,  2: -1.0,  3: 8.0,  4: -2.0,  5: -2.0,  6: -1.0,  7: -9.0,  8: -2.0,  9: 16.0,  10: 0.0,  11: -3.0}}).set_index(&quot;timestamp&quot;)
  </code></pre>