title: Credit Risk Analysis_Feature selection_Duplicate values in Information Value and weight of evidence table python
body: <p><strong>WOE and IV</strong> are important concepts in credit risk analysis used to find out <strong>features which are relevant to predict</strong> whether or not a person is a possible <strong>loan</strong> <strong>defaulter</strong>. Lets say I have a person with some variables (could be age, income, past credit score etc., and I need to predict based on past data which ppl are likely to default. Now I have 1000 variables. All the variables will not be of equal importance while predicting loan default. So WOE and IV help me in finding out which variable out of all the variables should I use in any model. Below is a sample df and a generic function to find WOE and IV for any df. My doubt is stated below.</p>
  
  <p><strong>X_train</strong> (independent variables in credit risk analysis)</p>
  
  <pre><code>Var1  Var2     .............  Var1000
  30     Unknown .............  80000
  33     Success .............   90000
  45     Failure .............     899900
  </code></pre>
  
  <p><strong>y_train</strong> (dependent variable in credit risk analysis to denote customer default or not in a loan)</p>
  
  <pre><code>0
  1
  0
  0...and so on
  </code></pre>
  
  <p><strong>Code to find the most relevant and important variables from a dataframe (WOE and IV)</strong></p>
  
  <pre><code>import pandas as pd
  import numpy as np
  import pandas.core.algorithms as algos
  from pandas import Series
  import scipy.stats.stats as stats
  import re
  import traceback
  import string
  
  max_bin = 20
  force_bin = 3
  
  # define a binning function
  def mono_bin(Y, X, n = max_bin):
  
      df1 = pd.DataFrame({"X": X, "Y": Y})
      justmiss = df1[['X','Y']][df1.X.isnull()]
      notmiss = df1[['X','Y']][df1.X.notnull()]
      r = 0
      while np.abs(r) &lt; 1:
          try:
              d1 = pd.DataFrame({"X": notmiss.X, "Y": notmiss.Y, "Bucket": pd.qcut(notmiss.X, n)})
              d2 = d1.groupby('Bucket', as_index=True)
              r, p = stats.spearmanr(d2.mean().X, d2.mean().Y)
              n = n - 1 
          except Exception as e:
              n = n - 1
  
      if len(d2) == 1:
          n = force_bin         
          bins = algos.quantile(notmiss.X, np.linspace(0, 1, n))
          if len(np.unique(bins)) == 2:
              bins = np.insert(bins, 0, 1)
              bins[1] = bins[1]-(bins[1]/2)
          d1 = pd.DataFrame({"X": notmiss.X, "Y": notmiss.Y, "Bucket": pd.cut(notmiss.X, np.unique(bins),include_lowest=True)}) 
          d2 = d1.groupby('Bucket', as_index=True)
  
      d3 = pd.DataFrame({},index=[])
      d3["MIN_VALUE"] = d2.min().X
      d3["MAX_VALUE"] = d2.max().X
      d3["COUNT"] = d2.count().Y
      d3["EVENT"] = d2.sum().Y
      d3["NONEVENT"] = d2.count().Y - d2.sum().Y
      d3=d3.reset_index(drop=True)
  
      if len(justmiss.index) &gt; 0:
          d4 = pd.DataFrame({'MIN_VALUE':np.nan},index=[0])
          d4["MAX_VALUE"] = np.nan
          d4["COUNT"] = justmiss.count().Y
          d4["EVENT"] = justmiss.sum().Y
          d4["NONEVENT"] = justmiss.count().Y - justmiss.sum().Y
          d3 = d3.append(d4,ignore_index=True)
  
      d3["EVENT_RATE"] = d3.EVENT/d3.COUNT
      d3["NON_EVENT_RATE"] = d3.NONEVENT/d3.COUNT
      d3["DIST_EVENT"] = d3.EVENT/d3.sum().EVENT
      d3["DIST_NON_EVENT"] = d3.NONEVENT/d3.sum().NONEVENT
      d3["WOE"] = np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)
      d3["IV"] = (d3.DIST_EVENT-d3.DIST_NON_EVENT)*np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)
      d3["VAR_NAME"] = "VAR"
      d3 = d3[['VAR_NAME','MIN_VALUE', 'MAX_VALUE', 'COUNT', 'EVENT', 'EVENT_RATE', 'NONEVENT', 'NON_EVENT_RATE', 'DIST_EVENT','DIST_NON_EVENT','WOE', 'IV']]       
      d3 = d3.replace([np.inf, -np.inf], 0)
      d3.IV = d3.IV.sum()
  
      return(d3)
  
  def char_bin(Y, X):
  
      df1 = pd.DataFrame({"X": X, "Y": Y})
      justmiss = df1[['X','Y']][df1.X.isnull()]
      notmiss = df1[['X','Y']][df1.X.notnull()]    
      df2 = notmiss.groupby('X',as_index=True)
  
      d3 = pd.DataFrame({},index=[])
      d3["COUNT"] = df2.count().Y
      d3["MIN_VALUE"] = df2.sum().Y.index
      d3["MAX_VALUE"] = d3["MIN_VALUE"]
      d3["EVENT"] = df2.sum().Y
      d3["NONEVENT"] = df2.count().Y - df2.sum().Y
  
      if len(justmiss.index) &gt; 0:
          d4 = pd.DataFrame({'MIN_VALUE':np.nan},index=[0])
          d4["MAX_VALUE"] = np.nan
          d4["COUNT"] = justmiss.count().Y
          d4["EVENT"] = justmiss.sum().Y
          d4["NONEVENT"] = justmiss.count().Y - justmiss.sum().Y
          d3 = d3.append(d4,ignore_index=True)
  
      d3["EVENT_RATE"] = d3.EVENT/d3.COUNT
      d3["NON_EVENT_RATE"] = d3.NONEVENT/d3.COUNT
      d3["DIST_EVENT"] = d3.EVENT/d3.sum().EVENT
      d3["DIST_NON_EVENT"] = d3.NONEVENT/d3.sum().NONEVENT
      d3["WOE"] = np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)
      d3["IV"] = (d3.DIST_EVENT-d3.DIST_NON_EVENT)*np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)
      d3["VAR_NAME"] = "VAR"
      d3 = d3[['VAR_NAME','MIN_VALUE', 'MAX_VALUE', 'COUNT', 'EVENT', 'EVENT_RATE', 'NONEVENT', 'NON_EVENT_RATE', 'DIST_EVENT','DIST_NON_EVENT','WOE', 'IV']]      
      d3 = d3.replace([np.inf, -np.inf], 0)
      d3.IV = d3.IV.sum()
      d3 = d3.reset_index(drop=True)
  
      return(d3)
  
  def data_vars(df1, target):
  
      stack = traceback.extract_stack()
      filename, lineno, function_name, code = stack[-2]
      vars_name = re.compile(r'\((.*?)\).*$').search(code).groups()[0]
      final = (re.findall(r"[\w']+", vars_name))[-1]
  
      x = df1.dtypes.index
      count = -1
  
      for i in x:
          if i.upper() not in (final.upper()):
              if np.issubdtype(df1[i], np.number) and len(Series.unique(df1[i])) &gt; 2:
                  conv = mono_bin(target, df1[i])
                  conv["VAR_NAME"] = i
                  count = count + 1
              else:
                  conv = char_bin(target, df1[i])
                  conv["VAR_NAME"] = i            
                  count = count + 1
  
              if count == 0:
                  iv_df = conv
              else:
                  iv_df = iv_df.append(conv,ignore_index=True)
  
      iv = pd.DataFrame({'IV':iv_df.groupby('VAR_NAME').IV.max()})
      iv = iv.reset_index()
      return(iv_df,iv)
  </code></pre>
  
  <p><strong>Calling function</strong></p>
  
  <pre><code>    final_iv, IV = data_vars(X_train, y_train)
  </code></pre>
  
  <p><strong>Output</strong></p>
  
  <pre><code>VAR_NAME    MIN_VALUE   MAX_VALUE   COUNT   EVENT   EVENT_RATE  NONEVENT    NON_EVENT_RATE  DIST_EVENT  DIST_NON_EVENT   WOE           IV
  0   Var1    19          39        2290   259     0.113100     2031         0.886900      0.497121   0.50775         -0.021156   0.000452
  1   Var1    40          87          2231     262     0.117436     1969         0.882564      0.502879   0.49225         0.021363     0.000452
  64  Var2    failure     failure        490   63      0.128571     427          0.871429      0.120921   0.10675         0.124650    0.461890
  65  Var2    other       other          197  38       0.192893      159          0.807107     0.072937   0.03975         0.606982    0.461890
  66  Var2   success      success        129  83       0.643411       46           0.356589   0.159309    0.01150          2.628499   0.461890
  67  Var2    unknown     unknown        3705 337      0.090958      3368         0.909042    0.646833    0.84200          -0.263692  0.461890
  </code></pre>
  
  <p>and so on....</p>
  
  <p>If you see there are 2 values for the same variable <code>Var1</code> and 4 values for <code>Var2</code> variable. I am unable to understand what is the meaning of multiple rows for the same variable.
  What is this function data_vars trying to do? </p>