title: Optimize nested for-loops in terms of execution time
body: <p>I want to optimize my code in terms of execution time. The code runs on the dataframe <code>alldata</code> that contains around 300,000 entries, but the computation takes a very long time (around 10 hours or so). </p>
  
  <p>The logic of the computation is the following:</p>
  
  <p>For each missing (nan) value of dataframe's columns specified in the list <code>list_of_NA_features</code>, the function <code>fill_missing_values</code> searches the most similar row (the Cosine similarity is computed based on columns in the list <code>list_of_non_nan_features</code> that are never empty) and returns the value of the current column and row in <code>alldata</code>.</p>
  
  <pre><code>from scipy import spatial
  
  def fill_missing_values(param_nan,current_row,df):
      df_non_nan = df.dropna(subset=[param_nan])
      list_of_non_nan_features = ["f1","f2","f3","f4","f5"] 
      max_val = 0
      searched_val = 0
      vector1 = current_row[list_of_non_nan_features].values
      for index, row in df_non_nan.iterrows():
          vector2 = row[list_of_non_nan_features].values
          sim = 1 - spatial.distance.cosine(vector1, vector2)
          if (sim&gt;max_val):
              max_val = sim
              searched_val = row[param_nan]
      return searched_val
  
  
  list_of_NA_features = df_train.columns[df_train.isnull().any()]
  
  
  for feature in list_of_NA_features:
      for index,row in alldata.iterrows():
          if (pd.isnull(row[feature]) == True):
              missing_value = fill_missing_values(feature,row,alldata)
              alldata.ix[index,feature] = missing_value
  </code></pre>
  
  <p>Is it possible to optimize the code? For instance, I am thinking about the substitution of <code>for</code> loops with <code>lambda</code> functions. Is it possible?</p>