title: Increased efficiency while looping through dataframe and appending to list using if conditions?
body: <p>I have a dataframe with ~40k lines and 52 columns. It describes locations of interest, with some overlap.</p>
  <p>I am trying to filter out locations that overlap, so that I only keep the last entry.</p>
  <p>An example dataset is:</p>
  <p><a href="https://i.stack.imgur.com/t6T4l.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/t6T4l.png" alt="enter image description here" /></a></p>
  <p>The expected output is:</p>
  <p><a href="https://i.stack.imgur.com/7DTk8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7DTk8.png" alt="enter image description here" /></a></p>
  <p>The first Track A entry is to be excluded.</p>
  <p>My code is below:</p>
  <pre><code>df_length = len(df.axes[0])
  
  duplicate_flag = False
  
  i = 0
  
  trim_value = 0.005
  
  data = []
  
  for j in range(df_length - 1):
      for k in range(df_length - 1):
          if k &gt;= i:
              if(df.at[i, 'Track'] == df.at[k+1, 'Track'] and
               df.at[i, 'Asset Name'] == df.at[k+1, 'Asset Name'] and
               df.at[i, 'Type'] == df.at[k+1, 'Type'] and
               df.at[i, 'Location From'] &gt;= (df.at[k+1, 'Location From'] - trim_value) and
               df.at[i, 'Location From'] &lt;= (df.at[k+1, 'Location From'] + trim_value) and
               df.at[i, 'Location To'] &gt;= (df.at[k+1, 'Location To'] - trim_value) and
               df.at[i, 'Location To'] &lt;= (df.at[k+1, 'Location To'] + trim_value)):
                  duplicate_flag = True
         
      if duplicate_flag == False:
          data.append(df.iloc[i])
          
      i += 1    
          
      duplicate_flag = False
  
  data.append(df.iloc[-1])
  df2 = pd.DataFrame(data)
  </code></pre>
  <p>For each row of the dataframe, I compare it to each row below it. If the Track, Asset Name and Type all equal, and the Location From and To values are + or - the trim value, I set a duplicate flag as true.</p>
  <p>After comparing to the rows below, if there was no duplicate found, I append that row of the original dataframe to a list, which is later converted into another dataframe.</p>
  <p>I then increment the counter i, and reset the duplicate_flag variable to False.</p>
  <p>My issue is this takes an extremely long time to run even with only 40k lines. I learnt that appending to a list and not to a dataframe is much more efficient, but I'm unsure what else is causing the bottleneck.</p>
  <p>My intuition is suggesting it is the conditional if statements, though unsure what would be an improvement.</p>
  <p>Any suggestions appreciated.</p>
  <p>Thanks.</p>