original_docstring: 
      Routine for clustering one year of ba data in chunks
  
      Parameters:
          dfr (DataFrame): A dataset to be clustered.
          chunk_size (int): target number of rows for a single chunk to cluster.
          Overlap (DataFrame): overlap points to be added to the first chunk.
          max_group_last (int): Group number increment. This will be added
              group labels to have unique group id across the record.
          eps (float): Distance threshold for DBSCAN clustering. Passed to
          cluster function.
  
      Returns:
          completed (dataframe): clustered dfr dataframe, without
          overlap if not last year.
          overlap (dataframe): densified points to be clustered
          with the following chunk.
  
      Todo:
          This does not have to be year based, since we have "day_since" column
          which gives a unique id in time for each pixel. Any consecutive
          time block would work. Chunking should be done on days_since!
      
docstring: Routine for clustering one year of ba data in chunks
  Parameters:
  dfr (DataFrame): A dataset to be clustered.
  chunk_size (int): target number of rows for a single chunk to cluster.
  Overlap (DataFrame): overlap points to be added to the first chunk.
  max_group_last (int): Group number increment. This will be added
  group labels to have unique group id across the record.
  eps (float): Distance threshold for DBSCAN clustering. Passed to
  cluster function.
  
  completed (dataframe): clustered dfr dataframe, without
  overlap if not last year.
  overlap (dataframe): densified points to be clustered
  with the following chunk.
  
  This does not have to be year based, since we have "day_since" column
  which gives a unique id in time for each pixel. Any consecutive
  time block would work. Chunking should be done on days_since!
docstring_tokens: Routine for clustering one year of ba data in chunks Parameters : dfr ( DataFrame ) : A dataset to be clustered . chunk_size ( int ) : target number of rows for a single chunk to cluster . Overlap ( DataFrame ) : overlap points to be added to the first chunk . max_group_last ( int ) : Group number increment . This will be added group labels to have unique group id across the record . eps ( float ) : Distance threshold for DBSCAN clustering . Passed to cluster function . completed ( dataframe ) : clustered dfr dataframe without overlap if not last year . overlap ( dataframe ) : densified points to be clustered with the following chunk . This does not have to be year based since we have " day_since " column which gives a unique id in time for each pixel . Any consecutive time block would work . Chunking should be done on days_since!
code: def cluster_in_chunks(dfr, chunk_size, max_group_last, eps, last_year=False):
      """
      Routine for clustering one year of ba data in chunks
  
      Parameters:
          dfr (DataFrame): A dataset to be clustered.
          chunk_size (int): target number of rows for a single chunk to cluster.
          Overlap (DataFrame): overlap points to be added to the first chunk.
          max_group_last (int): Group number increment. This will be added
              group labels to have unique group id across the record.
          eps (float): Distance threshold for DBSCAN clustering. Passed to
          cluster function.
  
      Returns:
          completed (dataframe): clustered dfr dataframe, without
          overlap if not last year.
          overlap (dataframe): densified points to be clustered
          with the following chunk.
  
      Todo:
          This does not have to be year based, since we have "day_since" column
          which gives a unique id in time for each pixel. Any consecutive
          time block would work. Chunking should be done on days_since!
      """
      labeler = LabelEncoder()
      overlap = None
      # determine number of chunks
      print(splits)
      done_ba = []
      done_af = []
      for nr, (split_start, split_end) in enumerate(splits, 1):
          print('Doing chunk ', nr)
          # if last chunk, do not extend beyond the split_end date
          if nr == len(splits):
              print('here')
              split_end += 1
              overlap_end = split_end - 21
          else:
              overlap_end = split_end
          print('splits', split_start, split_end)
          chunk = dfr.loc[(dfr.day_dense >= split_start) & (dfr.day_dense < split_end), :]
          if overlap is not None:
              chunk = pd.concat([overlap.drop('group', axis = 1), chunk])
          chunk['group'] = cluster(chunk[['x', 'y', 'day_dense']], eps)
          print('slave group: ', chunk.group.isnull().any())
          if (nr == len(splits)) and last_year:
              completed = chunk
              overlap = None
          else:
              completed, overlap = split_overlap(chunk, overlap_end)
          completed['group'] = labeler.fit_transform(completed.group)
          #increment group labels
          if max_group_last:
              completed['group'] += max_group_last
          max_group_last = completed['group'].max() + 1
          #un-densify completed part
          #completed = completed.groupby(['x', 'y', 'day_since'])['group'].first().reset_index()
          done_ba.append(completed_ba)
          done_af.append(completed_af)
      completed_ba = pd.concat(done_ba)
      pixel = completed_ba[(completed_ba.x == 27918)&(completed_ba.y == 24501)&(completed_ba.day_since == 699)]
      print(pixel)
      completed_af = pd.concat(done_af)
      elapsed = timeit.default_timer() - tt
      print('total {0} run time: {1:.2f}s'.format(year, elapsed))
      return completed_ba, completed_af, overlap
code_tokens: def cluster_in_chunks ( dfr , chunk_size , max_group_last , eps , last_year = False ) : labeler = LabelEncoder ( ) overlap = None print ( splits ) done_ba = [ ] done_af = [ ] for nr , ( split_start , split_end ) in enumerate ( splits , 1 ) : print ( 'Doing chunk ' , nr ) if nr == len ( splits ) : print ( 'here' ) split_end += 1 overlap_end = split_end - 21 else : overlap_end = split_end print ( 'splits' , split_start , split_end ) chunk = dfr . loc [ ( dfr . day_dense >= split_start ) & ( dfr . day_dense < split_end ) , : ] if overlap is not None : chunk = pd . concat ( [ overlap . drop ( 'group' , axis = 1 ) , chunk ] ) chunk [ 'group' ] = cluster ( chunk [ [ 'x' , 'y' , 'day_dense' ] ] , eps ) print ( 'slave group: ' , chunk . group . isnull ( ) . any ( ) ) if ( nr == len ( splits ) ) and last_year : completed = chunk overlap = None else : completed , overlap = split_overlap ( chunk , overlap_end ) completed [ 'group' ] = labeler . fit_transform ( completed . group ) if max_group_last : completed [ 'group' ] += max_group_last max_group_last = completed [ 'group' ] . max ( ) + 1 done_ba . append ( completed_ba ) done_af . append ( completed_af ) completed_ba = pd . concat ( done_ba ) pixel = completed_ba [ ( completed_ba . x == 27918 ) & ( completed_ba . y == 24501 ) & ( completed_ba . day_since == 699 ) ] print ( pixel ) completed_af = pd . concat ( done_af ) elapsed = timeit . default_timer ( ) - tt print ( 'total {0} run time: {1:.2f}s' . format ( year , elapsed ) ) return completed_ba , completed_af , overlap
short_docstring: Routine for clustering one year of ba data in chunks
  Parameters:
  dfr (DataFrame): A dataset to be clustered.
short_docstring_tokens: Routine for clustering one year of ba data in chunks Parameters : dfr ( DataFrame ) : A dataset to be clustered .
comment: """
      Routine for clustering one year of ba data in chunks
  
      Parameters:
          dfr (DataFrame): A dataset to be clustered.
          chunk_size (int): target number of rows for a single chunk to cluster.
          Overlap (DataFrame): overlap points to be added to the first chunk.
          max_group_last (int): Group number increment. This will be added
              group labels to have unique group id across the record.
          eps (float): Distance threshold for DBSCAN clustering. Passed to
          cluster function.
  
      Returns:
          completed (dataframe): clustered dfr dataframe, without
          overlap if not last year.
          overlap (dataframe): densified points to be clustered
          with the following chunk.
  
      Todo:
          This does not have to be year based, since we have "day_since" column
          which gives a unique id in time for each pixel. Any consecutive
          time block would work. Chunking should be done on days_since!
      """$SEP$# determine number of chunks$SEP$# if last chunk, do not extend beyond the split_end date$SEP$#increment group labels$SEP$#un-densify completed part$SEP$#completed = completed.groupby(['x', 'y', 'day_since'])['group'].first().reset_index()