title: faster way to run a for loop for a very large dataframe list
body: <p>I am using two for loops inside each other to calculate a value using combinations of elements in a dataframe list. the list consists of large number of dataframes and using two for loops takes considerable amount of time.</p>
  <p>Is there a way i can do the operation faster?</p>
  <p>the functions I refer with dummy names are the ones where I calculate the results.</p>
  <p>My code looks like this:</p>
  <pre><code> conf_list = []
  
   for tr in range(len(trajectories)):
       df_1 = trajectories[tr]
  
       if len(df_1) == 0:
          continue
     
       for tt in range(len(trajectories)):
           df_2 = trajectories[tt]
  
           if len(df_2) == 0:
              continue
  
           if df_1.equals(df_2) or df_1['time'].iloc[0] &gt; df_2['time'].iloc[-1] or df_2['time'].iloc[0] &gt; df_1['time'].iloc[-1]:
              continue
  
           df_temp = cartesian_product_basic(df_1,df_2)
      
           flg, df_temp = another_function(df_temp)
      
           if flg == 0:
               continue
  
           flg_h = some_other_function(df_temp)
      
           if flg_h == 1:
              conf_list.append(1)
      
  </code></pre>
  <p>My input list consist of around 5000 dataframes looking like (having several hundreds of rows)</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>id</th>
  <th>x</th>
  <th>y</th>
  <th>z</th>
  <th>time</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>5</td>
  <td>7</td>
  <td>2</td>
  <td>5</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>and what i do is I get the cartesian product with combinations of two dataframes and for each couple I calculate another value 'c'. If this value c meets a condition then I add an element to my c_list so that I can get the final number of couples meeting the requirement.</p>
  <p>For further info;</p>
  <p>a_function(df_1, df_2) is a function getting the cartesian product of two dataframes.</p>
  <p>another_function looks like this:</p>
  <pre><code>  def another_function(df_temp):
        df_temp['z_dif'] =      nwh((df_temp['time_x'] == df_temp['time_y'])
                                            , abs(df_temp['z_x']-  df_temp['z_y']) , np.nan)
  
        df_temp = df_temp.dropna() 
  
        df_temp['vert_conf'] = nwh((df_temp['z_dif'] &gt;= 1000)
                                            , np.nan , 1)
        df_temp = df_temp.dropna() 
  
        if len(df_temp) == 0:
         flg = 0
        else:
         flg = 1
      
        return flg, df_temp
  </code></pre>
  <p>and some_other_function looks like this:</p>
  <pre><code>  def some_other_function(df_temp):
        df_temp['x_dif'] =   df_temp['x_x']*df_temp['x_y']
        df_temp['y_dif'] = df_temp['y_x']*df_temp['y_y']
        df_temp['hor_dif'] = hypot(df_temp['x_dif'], df_temp['y_dif'])
  
        df_temp['conf'] = np.where((df_temp['hor_dif']&lt;=5)
                                            , 1 , np.nan)
        if df_temp['conf'].sum()&gt;0:
           flg_h = 1
      
       return flg_h       
  </code></pre>