title: Efficient for loops for Pandas Dataframes
body: <p>I have 2 Pandas Dataframes, X_ol and y_ol,with a shape of 29000 x 29 and 29000 x 21, and I'm running a nested for loop through this data to generate more data(as you'll see below). What I'm trying to achieve with this for loop is something like this:</p>
  <pre><code>    DataFrame X_ol                              DataFrame y_ol
      id     Date      c1      c2      c3         c1      c2      c3
      1      2000      0       1       1          0       1       1
      2      2001      1       0       1          1       0       1
      3      2002      1       1       0          1       1       0
      4      2003      1       1       1          1       1       1
  
      # (New DataFrame X)                         # (Second New DataFrame, y)
      id     Date      c1      c2      c3         c1      c2      c3 
      1      2000      0       0       1          0       1       0
      1      2000      0       1       0          0       0       1
      2      2001      0       0       1          1       0       0
      2      2001      1       0       0          0       0       1
      3      2002      0       1       0          1       0       0
      3      2002      1       0       0          0       1       0
      4      2003      0       1       1          1       0       0
      4      2003      1       0       1          0       1       0
      4      2003      1       1       0          0       0       1
      
  </code></pre>
  <p>so it looks at the y_ol dataframe row by row and for each cell value of 1 it creates a new row in dataframe X, with that cell switched off, and creates a new row in the y dataframe with the corresponding cell on and all other values on the same row in the y Dataframe will now be switched off.
  I wrote this code that does it correctly, but takes so much time. 12+ minutes producing the 2 Data Frames of 60,000 rows, are there built in pandas functions/methods to use to make this more efficient or another method entirely that takes out the for loop?</p>
  <pre><code>for i in range(len(y_ol)):
      ab = y_ol.iloc[i].where(y_ol.iloc[i]==1)
      abInd = ab[ab==1.0].index
      for j in abInd:
          y_tmp = deepcopy(y_ol.iloc[i:i+1, :])
          y_ol[j][i] = 0
          conc = pd.concat([X_ol.iloc[i:i+1,:], y_ol.iloc[i:i+1, :]], axis=1)
          X = X.append(conc)
          y_tmp.iloc[:, :] = 0
          y_tmp[j] = 1
          y = y.append(y_tmp)
          y_ol[j][i] = 1
  </code></pre>
  <p>Thanks in advance</p>