title: Python getting conditional probabilities of all feature combinations in pandas dataframe
body: <p>I have a Pandas data frame with some categorical variables. Something like this - </p>
  
  <pre><code>&gt;&gt;df
   'a', 'x'
   'a', 'y'
  </code></pre>
  
  <p>Now, I want to return a matrix with the conditional probabilities of each level appearing with every other level. For the data frame above, it would look like - </p>
  
  <pre><code>[1, 0.5, 0.5],
  [1, 1, 0],
  [1, 0, 1]
  </code></pre>
  
  <p>The three entries correspond to the levels 'a', 'x' and 'y'.</p>
  
  <p>This is because conditional on the first column being 'a', the probabilities of 'x' and 'y' appearing are 0.5 each and so on.
  I have some code that does this (below). However, the problem is that it is excruciatingly slow. So slow that the application I want to use it in times out. Does anyone have any tips to make it faster?</p>
  
  <pre><code>df = pd.read_csv('pathToData.csv')
  df = df.fillna("null")
  cols = 0
  col_levels = []
  columns = {}
  num = 0
  for i in df.columns:
  cols += len(set(df[i]))
  col_levels.append(np.sort(list(set(df[i]))))
  for j in np.sort(list(set(df[i]))):
      columns[i + '_' + str(j)] = num
      num += 1
  
  res = np.eye(cols)
  
  for i in range(len(df.columns)):
    for j in range(len(df.columns)):
      if i != j:
          row_feature = df.columns[i]
          col_feature = df.columns[j]
          rowLevels = col_levels[i]
          colLevels = col_levels[j]
          for ii in rowLevels:
              for jj in colLevels:                
                  frst = (df[row_feature] == ii) * 1
                  scnd = (df[col_feature] == jj) * 1
                  prob = sum(frst*scnd)/(sum(frst) + 1e-9)
                  frst_ind = columns[row_feature + '_' + ii]
                  scnd_ind = columns[col_feature + '_' + jj]
                  res[frst_ind, scnd_ind] = prob
  </code></pre>
  
  <p>EDIT: Here is a bigger example:</p>
  
  <pre><code>&gt;&gt;df
  'a', 'x', 'l'
  'a', 'y', 'l'
  'b', 'x', 'l'
  </code></pre>
  
  <p>The number of distinct categories here are 'a', 'b', 'x', 'y' and 'l'. Since these are 5 categories, the output matrix should be 5x5. The first row and first column would be how often does 'a' appear conditional on 'a'. This is of course, 1 (as are all the diagonals). The first row and second column is conditional on 'a', what is the probability of 'b'. Since 'a' and 'b' are parts of the same column, this is zero. The first row and third column is the probability of 'x' conditional on 'a'. We see that 'a' appears twice but only once with 'x'. So, this probability is 0.5. And so on.</p>