title: python, using logistic regression to see which variable is adding more weight towards a positive prediction
body: <p>so i have a banking data set, where i have to predict if customer would take a term deposit or not.
  i have a column called job; which is categorical and has the job types of each customer.
  i am currently in the EDA process and want to make out which job category contributes the most towards a positive prediction.</p>
  
  <p>i intend to do this with logistic regression (not sure if this is right, alternative method sugestions are welcome).</p>
  
  <p>so here is what i did;</p>
  
  <p>i did one k- hot encoding for each job category(and have 1-0 values for each job type), and the target i
  Did k-1 one hot, and have 1-0 values for Target_yes(1 = the customer accepted the term deposit and 0 the customer did not accept).</p>
  
  <pre><code>    job_management  job_technician  job_entrepreneur    job_blue-collar     job_unknown     job_retired     job_admin.  job_services    job_self-employed   job_unemployed  job_housemaid   job_student
  0   1   0   0   0   0   0   0   0   0   0   0   0
  1   0   1   0   0   0   0   0   0   0   0   0   0
  2   0   0   1   0   0   0   0   0   0   0   0   0
  3   0   0   0   1   0   0   0   0   0   0   0   0
  4   0   0   0   0   1   0   0   0   0   0   0   0
  ...     ...     ...     ...     ...     ...     ...     ...     ...     ...     ...     ...     ...
  45206   0   1   0   0   0   0   0   0   0   0   0   0
  45207   0   0   0   0   0   1   0   0   0   0   0   0
  45208   0   0   0   0   0   1   0   0   0   0   0   0
  45209   0   0   0   1   0   0   0   0   0   0   0   0
  45210   0   0   1   0   0   0   0   0   0   0   0   0
  
  45211 rows × 12 columns
  </code></pre>
  
  <p>The target column looks like this;</p>
  
  <pre><code>0        0
  1        0
  2        0
  3        0
  4        0
          ..
  45206    1
  45207    1
  45208    1
  45209    0
  45210    0
  Name: Target_yes, Length: 45211, dtype: int32
  </code></pre>
  
  <p>I fit this to a sklearn logistic regression model and got the coefficients. Unable to interpret them, i looked for an alternative and came across stat model version. did the same with the logit function. In the example i saw online, he had used sm.get_constants for the x variable.</p>
  
  <pre><code>from sklearn.linear_model import LogisticRegression
  from sklearn import metrics
  model = LogisticRegression(solver='liblinear')
  model.fit(vari,tgt)
  model.score(vari,tgt)
  df = pd.DataFrame(model.coef_)
  df['inter'] = model.intercept_
  print(df)
  </code></pre>
  
  <p>The model score and print()df results are as follows:</p>
  
  <pre><code>0.8830151954170445(model score)
  
  print(df)
            0         1         2         3         4         5         6  \
  0 -0.040404 -0.289274 -0.604957 -0.748797 -0.206201  0.573717 -0.177778   
  
            7         8         9        10        11     inter  
  0 -0.530802 -0.210549  0.099326 -0.539109  0.879504 -1.795323 
  </code></pre>
  
  <p>When i use sm.get_constats, i get coefficients similar to the sklearn logisticRegression, but the Zscores, (which i intended to use to find the job type which contributes the most towards a positive prediction) becomes nan.</p>
  
  <pre><code>import statsmodels.api as sm
  logit = sm.Logit(tgt, sm.add_constant(vari)).fit()
  logit.summary2()
  </code></pre>
  
  <p>Results are:</p>
  
  <pre><code>E:\Programs\Anaconda\lib\site-packages\numpy\core\fromnumeric.py:2495: FutureWarning:
  
  Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
  
  E:\Programs\Anaconda\lib\site-packages\statsmodels\base\model.py:1286: RuntimeWarning:
  
  invalid value encountered in sqrt
  
  E:\Programs\Anaconda\lib\site-packages\scipy\stats\_distn_infrastructure.py:901: RuntimeWarning:
  
  invalid value encountered in greater
  
  E:\Programs\Anaconda\lib\site-packages\scipy\stats\_distn_infrastructure.py:901: RuntimeWarning:
  
  invalid value encountered in less
  
  E:\Programs\Anaconda\lib\site-packages\scipy\stats\_distn_infrastructure.py:1892: RuntimeWarning:
  
  invalid value encountered in less_equal
  
  Optimization terminated successfully.
           Current function value: 0.352610
           Iterations 13
  
  Model:  Logit   Pseudo R-squared:   0.023
  Dependent Variable:     Target_yes  AIC:    31907.6785
  Date:   2019-11-18 10:17    BIC:    32012.3076
  No. Observations:   45211   Log-Likelihood:     -15942.
  Df Model:   11  LL-Null:    -16315.
  Df Residuals:   45199   LLR p-value:    3.9218e-153
  Converged:  1.0000  Scale:  1.0000
  No. Iterations:     13.0000         
                    Coef.     Std.Err.    z   P&gt;|z|   [0.025  0.975]
  const            -1.7968    nan     nan     nan     nan     nan
  job_management   -0.0390    nan     nan     nan     nan     nan
  job_technician   -0.2882    nan     nan     nan     nan     nan
  job_entrepreneur -0.6092    nan     nan     nan     nan     nan
  job_blue-collar  -0.7484    nan     nan     nan     nan     nan
  job_unknown      -0.2142    nan     nan     nan     nan     nan
  job_retired       0.5766    nan     nan     nan     nan     nan
  job_admin.       -0.1766    nan     nan     nan     nan     nan
  job_services     -0.5312    nan     nan     nan     nan     nan
  job_self-employed   -0.2106     nan     nan     nan     nan     nan
  job_unemployed  0.1011  nan     nan     nan     nan     nan
  job_housemaid   -0.5427     nan     nan     nan     nan     nan
  job_student     0.8857  nan     nan     nan     nan     nan
  </code></pre>
  
  <p>if i use the Stat models logit without the sm.get_constats, i get coefficients that are very different from the sklearn Logistic regression, but i get values for the zscore (which are all negative)</p>
  
  <pre><code>import statsmodels.api as sm
  logit = sm.Logit(tgt, vari).fit()
  logit.summary2()
  </code></pre>
  
  <p>results are:</p>
  
  <pre><code>Optimization terminated successfully.
           Current function value: 0.352610
           Iterations 6
  
  Model:  Logit   Pseudo R-squared:   0.023
  Dependent Variable:     Target_yes  AIC:    31907.6785
  Date:   2019-11-18 10:18    BIC:    32012.3076
  No. Observations:   45211   Log-Likelihood:     -15942.
  Df Model:   11  LL-Null:    -16315.
  Df Residuals:   45199   LLR p-value:    3.9218e-153
  Converged:  1.0000  Scale:  1.0000
  No. Iterations:     6.0000      
                    Coef.     Std.Err.    z        P&gt;|z|  [0.025  0.975]
  job_management  -1.8357     0.0299  -61.4917    0.0000  -1.8943     -1.7772
  job_technician  -2.0849     0.0366  -56.9885    0.0000  -2.1566     -2.0132
  job_entrepreneur -2.4060    0.0941  -25.5563    0.0000  -2.5905     -2.2215
  job_blue-collar  -2.5452    0.0390  -65.2134    0.0000  -2.6217     -2.4687
  job_unknown      -2.0110    0.1826  -11.0120    0.0000  -2.3689     -1.6531
  job_retired      -1.2201    0.0501  -24.3534    0.0000  -1.3183     -1.1219
  job_admin.       -1.9734    0.0425  -46.4478    0.0000  -2.0566     -1.8901
  job_services     -2.3280    0.0545  -42.6871    0.0000  -2.4349     -2.2211
  job_self-employed-2.0074    0.0779  -25.7739    0.0000  -2.1600     -1.8547
  job_unemployed   -1.6957    0.0765  -22.1538    0.0000  -1.8457     -1.5457
  job_housemaid    -2.3395    0.1003  -23.3270    0.0000  -2.5361     -2.1429
  job_student      -0.9111    0.0722  -12.6195    0.0000  -1.0526     -0.7696
  </code></pre>
  
  <p>Which of the two is better?
  or should i use a completly different approach ?</p>