# Logistic Regression tutorial
The science is is data science:
> Given raw data, constraints, and a problem statement, how to determine a solution.

Main steps:
1. You hold on to your existing best performer.
2. Once you have a new idea to prototype, set up an experiment wherein the two best models compete.
3. Rinse and repeat (while not overfitting).

## Classifiers
Classification involves mapping your data points to a finite set of labels or the probability of a given label or labels.

In building classifiers, the problem is: *given data, a real-world classification problem, and constraints, you need to determine:
1. Which classifier to use
2. Which optimization method to employ
3. Which loss function to minimize
4. Which features to take from the data
5. Which evaluation metric to use

**Focus of this case study**: User-level conversion prediction - forecasting when someone will click using logistic regression

**Difference between logistic regression and linear regression**
The aim of logistic regression is o output a probability while that of linear regression is to predict the actual value.

### The Underlying Math
You want a function that takes the data and transforms it into a single value bounded inside the closed interval $[0,1]$. An example is the inverse-logit function:
\begin{equation}
    P(t)=logit^{-1}(t)\equiv\frac{1}{\left(1+e^{-t}\right)}=\frac{e^t}{1+e^t}
\end{equation}

$logit(P(c=1\mid x_i))=\alpha+\beta^\tau x_i$, where $c_i$ is the labels or classes, and $x_i$ is teh vector of features for user i.

The parameters $\alpha$ and $\beta$ could be estimated using Maximum Likelihood Estimation (MLE):
\begin{equation}
\Theta_{MLE} = \underset{\Theta}{argmax}\prod ^n_1p^{c_i}_i\cdot\left(1-p_i\right)^{1-c_i}
\end{equation}
#### Evaluation
Some ways of evaluating the model:
- **Area under the receiver operating curve (AUC):** ranks points, according to varied thresholds, into two categories of a binary classification problem (e.g., positive or negative)
- **Area under the cumulative lift curve:** measures how many times it is better to use a model versus not using the model (i.e., just selecting at random)

Others are: accuracy, precision, recall, F-score - a harmonic mean of precision and recall.

For density estimation: mean squared error, root squared error, mean aboslute error.

### Tutorial details

**problem:** Given the data of a marketing campaigns/phone calls by a Portuguese banking institution, predict whether a client will subscribe to a term deposit or not. The dataset provides the bank customers' information and a label column for customer subscription status.
- This codes from this tutorial is followed from [here](https://towardsdatascience.com/building-a-logistic-regression-in-python-step-by-step-becd4d56c9c8).
- The dataset can be found [here.](https://raw.githubusercontent.com/madmashup/targeted-marketing-predictive-engine/master/banking.csv)


```python
import pandas as pd
import numpy as np
from sklearn import preprocessing
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import seaborn as sns
```


```python
plt.rc("font", size=14)
sns.set(style="white")
sns.set(style="whitegrid", color_codes=True)
%matplotlib inline
```


```python
data = pd.read_csv('banking.csv', header=0)
```


```python
data.dropna()
print(data.shape)
print(data.columns)
```

    (41188, 21)
    Index(['age', 'job', 'marital', 'education', 'default', 'housing', 'loan',
           'contact', 'month', 'day_of_week', 'duration', 'campaign', 'pdays',
           'previous', 'poutcome', 'emp_var_rate', 'cons_price_idx',
           'cons_conf_idx', 'euribor3m', 'nr_employed', 'y'],
          dtype='object')



```python
data.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>age</th>
      <th>job</th>
      <th>marital</th>
      <th>education</th>
      <th>default</th>
      <th>housing</th>
      <th>loan</th>
      <th>contact</th>
      <th>month</th>
      <th>day_of_week</th>
      <th>...</th>
      <th>campaign</th>
      <th>pdays</th>
      <th>previous</th>
      <th>poutcome</th>
      <th>emp_var_rate</th>
      <th>cons_price_idx</th>
      <th>cons_conf_idx</th>
      <th>euribor3m</th>
      <th>nr_employed</th>
      <th>y</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>44</td>
      <td>blue-collar</td>
      <td>married</td>
      <td>basic.4y</td>
      <td>unknown</td>
      <td>yes</td>
      <td>no</td>
      <td>cellular</td>
      <td>aug</td>
      <td>thu</td>
      <td>...</td>
      <td>1</td>
      <td>999</td>
      <td>0</td>
      <td>nonexistent</td>
      <td>1.4</td>
      <td>93.444</td>
      <td>-36.1</td>
      <td>4.963</td>
      <td>5228.1</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>53</td>
      <td>technician</td>
      <td>married</td>
      <td>unknown</td>
      <td>no</td>
      <td>no</td>
      <td>no</td>
      <td>cellular</td>
      <td>nov</td>
      <td>fri</td>
      <td>...</td>
      <td>1</td>
      <td>999</td>
      <td>0</td>
      <td>nonexistent</td>
      <td>-0.1</td>
      <td>93.200</td>
      <td>-42.0</td>
      <td>4.021</td>
      <td>5195.8</td>
      <td>0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>28</td>
      <td>management</td>
      <td>single</td>
      <td>university.degree</td>
      <td>no</td>
      <td>yes</td>
      <td>no</td>
      <td>cellular</td>
      <td>jun</td>
      <td>thu</td>
      <td>...</td>
      <td>3</td>
      <td>6</td>
      <td>2</td>
      <td>success</td>
      <td>-1.7</td>
      <td>94.055</td>
      <td>-39.8</td>
      <td>0.729</td>
      <td>4991.6</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>39</td>
      <td>services</td>
      <td>married</td>
      <td>high.school</td>
      <td>no</td>
      <td>no</td>
      <td>no</td>
      <td>cellular</td>
      <td>apr</td>
      <td>fri</td>
      <td>...</td>
      <td>2</td>
      <td>999</td>
      <td>0</td>
      <td>nonexistent</td>
      <td>-1.8</td>
      <td>93.075</td>
      <td>-47.1</td>
      <td>1.405</td>
      <td>5099.1</td>
      <td>0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>55</td>
      <td>retired</td>
      <td>married</td>
      <td>basic.4y</td>
      <td>no</td>
      <td>yes</td>
      <td>no</td>
      <td>cellular</td>
      <td>aug</td>
      <td>fri</td>
      <td>...</td>
      <td>1</td>
      <td>3</td>
      <td>1</td>
      <td>success</td>
      <td>-2.9</td>
      <td>92.201</td>
      <td>-31.4</td>
      <td>0.869</td>
      <td>5076.2</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 21 columns</p>
</div>




```python
data['education'].unique()
```




    array(['basic.4y', 'unknown', 'university.degree', 'high.school',
           'basic.9y', 'professional.course', 'basic.6y', 'illiterate'],
          dtype=object)




```python
data['education']=np.where(data['education']=='basic.9y','basic', data['education'])
data['education']=np.where(data['education']=='basic.4y','basic', data['education'])
data['education']=np.where(data['education']=='basic.6y','basic', data['education'])
```


```python
data['education'].unique()
```




    array(['basic', 'unknown', 'university.degree', 'high.school',
           'professional.course', 'illiterate'], dtype=object)




```python
data['y'].value_counts()
```




    0    36548
    1     4640
    Name: y, dtype: int64




```python
sns.countplot(x='y',data=data, palette='hls')
```


```python
count_no_sub = len(data[data['y']==0])
count_sub = len(data[data['y']==1])
pct_of_no_sub = count_no_sub/(count_no_sub+count_sub)
print("percentage of no subscription is", pct_of_no_sub*100)
pct_of_sub = count_sub/(count_no_sub+count_sub)
print("percentage of subscription", pct_of_sub*100)
```

    percentage of no subscription is 88.73458288821988
    percentage of subscription 11.265417111780131



```python
data.groupby('y').mean()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>age</th>
      <th>duration</th>
      <th>campaign</th>
      <th>pdays</th>
      <th>previous</th>
      <th>emp_var_rate</th>
      <th>cons_price_idx</th>
      <th>cons_conf_idx</th>
      <th>euribor3m</th>
      <th>nr_employed</th>
    </tr>
    <tr>
      <th>y</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>39.911185</td>
      <td>220.844807</td>
      <td>2.633085</td>
      <td>984.113878</td>
      <td>0.132374</td>
      <td>0.248875</td>
      <td>93.603757</td>
      <td>-40.593097</td>
      <td>3.811491</td>
      <td>5176.166600</td>
    </tr>
    <tr>
      <th>1</th>
      <td>40.913147</td>
      <td>553.191164</td>
      <td>2.051724</td>
      <td>792.035560</td>
      <td>0.492672</td>
      <td>-1.233448</td>
      <td>93.354386</td>
      <td>-39.789784</td>
      <td>2.123135</td>
      <td>5095.115991</td>
    </tr>
  </tbody>
</table>
</div>




```python
data.groupby('job').mean()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>age</th>
      <th>duration</th>
      <th>campaign</th>
      <th>pdays</th>
      <th>previous</th>
      <th>emp_var_rate</th>
      <th>cons_price_idx</th>
      <th>cons_conf_idx</th>
      <th>euribor3m</th>
      <th>nr_employed</th>
      <th>y</th>
    </tr>
    <tr>
      <th>job</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>admin.</th>
      <td>38.187296</td>
      <td>254.312128</td>
      <td>2.623489</td>
      <td>954.319229</td>
      <td>0.189023</td>
      <td>0.015563</td>
      <td>93.534054</td>
      <td>-40.245433</td>
      <td>3.550274</td>
      <td>5164.125350</td>
      <td>0.129726</td>
    </tr>
    <tr>
      <th>blue-collar</th>
      <td>39.555760</td>
      <td>264.542360</td>
      <td>2.558461</td>
      <td>985.160363</td>
      <td>0.122542</td>
      <td>0.248995</td>
      <td>93.656656</td>
      <td>-41.375816</td>
      <td>3.771996</td>
      <td>5175.615150</td>
      <td>0.068943</td>
    </tr>
    <tr>
      <th>entrepreneur</th>
      <td>41.723214</td>
      <td>263.267857</td>
      <td>2.535714</td>
      <td>981.267170</td>
      <td>0.138736</td>
      <td>0.158723</td>
      <td>93.605372</td>
      <td>-41.283654</td>
      <td>3.791120</td>
      <td>5176.313530</td>
      <td>0.085165</td>
    </tr>
    <tr>
      <th>housemaid</th>
      <td>45.500000</td>
      <td>250.454717</td>
      <td>2.639623</td>
      <td>960.579245</td>
      <td>0.137736</td>
      <td>0.433396</td>
      <td>93.676576</td>
      <td>-39.495283</td>
      <td>4.009645</td>
      <td>5179.529623</td>
      <td>0.100000</td>
    </tr>
    <tr>
      <th>management</th>
      <td>42.362859</td>
      <td>257.058140</td>
      <td>2.476060</td>
      <td>962.647059</td>
      <td>0.185021</td>
      <td>-0.012688</td>
      <td>93.522755</td>
      <td>-40.489466</td>
      <td>3.611316</td>
      <td>5166.650513</td>
      <td>0.112175</td>
    </tr>
    <tr>
      <th>retired</th>
      <td>62.027326</td>
      <td>273.712209</td>
      <td>2.476744</td>
      <td>897.936047</td>
      <td>0.327326</td>
      <td>-0.698314</td>
      <td>93.430786</td>
      <td>-38.573081</td>
      <td>2.770066</td>
      <td>5122.262151</td>
      <td>0.252326</td>
    </tr>
    <tr>
      <th>self-employed</th>
      <td>39.949331</td>
      <td>264.142153</td>
      <td>2.660802</td>
      <td>976.621393</td>
      <td>0.143561</td>
      <td>0.094159</td>
      <td>93.559982</td>
      <td>-40.488107</td>
      <td>3.689376</td>
      <td>5170.674384</td>
      <td>0.104856</td>
    </tr>
    <tr>
      <th>services</th>
      <td>37.926430</td>
      <td>258.398085</td>
      <td>2.587805</td>
      <td>979.974049</td>
      <td>0.154951</td>
      <td>0.175359</td>
      <td>93.634659</td>
      <td>-41.290048</td>
      <td>3.699187</td>
      <td>5171.600126</td>
      <td>0.081381</td>
    </tr>
    <tr>
      <th>student</th>
      <td>25.894857</td>
      <td>283.683429</td>
      <td>2.104000</td>
      <td>840.217143</td>
      <td>0.524571</td>
      <td>-1.408000</td>
      <td>93.331613</td>
      <td>-40.187543</td>
      <td>1.884224</td>
      <td>5085.939086</td>
      <td>0.314286</td>
    </tr>
    <tr>
      <th>technician</th>
      <td>38.507638</td>
      <td>250.232241</td>
      <td>2.577339</td>
      <td>964.408127</td>
      <td>0.153789</td>
      <td>0.274566</td>
      <td>93.561471</td>
      <td>-39.927569</td>
      <td>3.820401</td>
      <td>5175.648391</td>
      <td>0.108260</td>
    </tr>
    <tr>
      <th>unemployed</th>
      <td>39.733728</td>
      <td>249.451677</td>
      <td>2.564103</td>
      <td>935.316568</td>
      <td>0.199211</td>
      <td>-0.111736</td>
      <td>93.563781</td>
      <td>-40.007594</td>
      <td>3.466583</td>
      <td>5157.156509</td>
      <td>0.142012</td>
    </tr>
    <tr>
      <th>unknown</th>
      <td>45.563636</td>
      <td>239.675758</td>
      <td>2.648485</td>
      <td>938.727273</td>
      <td>0.154545</td>
      <td>0.357879</td>
      <td>93.718942</td>
      <td>-38.797879</td>
      <td>3.949033</td>
      <td>5172.931818</td>
      <td>0.112121</td>
    </tr>
  </tbody>
</table>
</div>




```python
data.groupby('marital').mean()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>age</th>
      <th>duration</th>
      <th>campaign</th>
      <th>pdays</th>
      <th>previous</th>
      <th>emp_var_rate</th>
      <th>cons_price_idx</th>
      <th>cons_conf_idx</th>
      <th>euribor3m</th>
      <th>nr_employed</th>
      <th>y</th>
    </tr>
    <tr>
      <th>marital</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>divorced</th>
      <td>44.899393</td>
      <td>253.790330</td>
      <td>2.61340</td>
      <td>968.639853</td>
      <td>0.168690</td>
      <td>0.163985</td>
      <td>93.606563</td>
      <td>-40.707069</td>
      <td>3.715603</td>
      <td>5170.878643</td>
      <td>0.103209</td>
    </tr>
    <tr>
      <th>married</th>
      <td>42.307165</td>
      <td>257.438623</td>
      <td>2.57281</td>
      <td>967.247673</td>
      <td>0.155608</td>
      <td>0.183625</td>
      <td>93.597367</td>
      <td>-40.270659</td>
      <td>3.745832</td>
      <td>5171.848772</td>
      <td>0.101573</td>
    </tr>
    <tr>
      <th>single</th>
      <td>33.158714</td>
      <td>261.524378</td>
      <td>2.53380</td>
      <td>949.909578</td>
      <td>0.211359</td>
      <td>-0.167989</td>
      <td>93.517300</td>
      <td>-40.918698</td>
      <td>3.317447</td>
      <td>5155.199265</td>
      <td>0.140041</td>
    </tr>
    <tr>
      <th>unknown</th>
      <td>40.275000</td>
      <td>312.725000</td>
      <td>3.18750</td>
      <td>937.100000</td>
      <td>0.275000</td>
      <td>-0.221250</td>
      <td>93.471250</td>
      <td>-40.820000</td>
      <td>3.313038</td>
      <td>5157.393750</td>
      <td>0.150000</td>
    </tr>
  </tbody>
</table>
</div>




```python
table = pd.crosstab(data.job, data.y).plot(kind='bar')
```


```python
table=pd.crosstab(data.marital,data.y)
table.div(table.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
plt.title('Stacked Bar Chart of Marital Status vs Purchase')
plt.xlabel('Marital Status')
plt.ylabel('Proportion of Customers')
plt.savefig('mariral_vs_pur_stack')
```


```python
table=pd.crosstab(data.education,data.y)
table.div(table.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
plt.title('Stacked Bar Chart of Education vs Purchase')
plt.xlabel('Education')
plt.ylabel('Proportion of Customers')
plt.savefig('edu_vs_pur_stack')
```


```python
pd.crosstab(data.day_of_week,data.y).plot(kind='bar')
plt.title('Purchase Frequency for Day of Week')
plt.xlabel('Day of Week')
plt.ylabel('Frequency of Purchase')
plt.savefig('pur_dayofweek_bar')
```


```python
pd.crosstab(data.month,data.y).plot(kind='bar')
plt.title('Purchase Frequency for Month')
plt.xlabel('Month')
plt.ylabel('Frequency of Purchase')
plt.savefig('pur_fre_month_bar')
```


```python
data.age.hist()
plt.title('Histogram of Age')
plt.xlabel('Age')
plt.ylabel('Frequency')
plt.savefig('hist_age')
```


```python
pd.crosstab(data.poutcome,data.y).plot(kind='bar')
plt.title('Purchase Frequency for Poutcome')
plt.xlabel('Poutcome')
plt.ylabel('Frequency of Purchase')
plt.savefig('pur_fre_pout_bar')
```

###  Create dummy variables


```python
cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']
for var in cat_vars:
    cat_list='var'+'_'+var
    cat_list = pd.get_dummies(data[var], prefix=var)
    data1=data.join(cat_list)
    data=data1
cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']
data_vars=data.columns.values.tolist()
to_keep=[i for i in data_vars if i not in cat_vars]
```


```python
data_final = data[to_keep]
```


```python
data_final.columns.values
```




    array(['age', 'duration', 'campaign', 'pdays', 'previous', 'emp_var_rate',
           'cons_price_idx', 'cons_conf_idx', 'euribor3m', 'nr_employed', 'y',
           'job_admin.', 'job_blue-collar', 'job_entrepreneur',
           'job_housemaid', 'job_management', 'job_retired',
           'job_self-employed', 'job_services', 'job_student',
           'job_technician', 'job_unemployed', 'job_unknown',
           'marital_divorced', 'marital_married', 'marital_single',
           'marital_unknown', 'education_basic', 'education_high.school',
           'education_illiterate', 'education_professional.course',
           'education_university.degree', 'education_unknown', 'default_no',
           'default_unknown', 'default_yes', 'housing_no', 'housing_unknown',
           'housing_yes', 'loan_no', 'loan_unknown', 'loan_yes',
           'contact_cellular', 'contact_telephone', 'month_apr', 'month_aug',
           'month_dec', 'month_jul', 'month_jun', 'month_mar', 'month_may',
           'month_nov', 'month_oct', 'month_sep', 'day_of_week_fri',
           'day_of_week_mon', 'day_of_week_thu', 'day_of_week_tue',
           'day_of_week_wed', 'poutcome_failure', 'poutcome_nonexistent',
           'poutcome_success'], dtype=object)



###  Solving class imbalance using SMOTE (Synthetic Minority Oversampling Technique)


```python
X=data_final.loc[:, data_final.columns != 'y']
y=data_final.loc[:, data_final.columns == 'y']
```


```python
from imblearn.over_sampling import SMOTE
os = SMOTE(random_state=0)
```


```python
X_train, X_test, y_train, y_test=train_test_split(X, y, test_size=0.3, random_state=0)
columns = X_train.columns
```


```python
os_data_X,os_data_y=os.fit_sample(X_train, y_train)
os_data_X = pd.DataFrame(data=os_data_X,columns=columns )
os_data_y= pd.DataFrame(data=os_data_y,columns=['y'])
```

    /home/bbrighttaer/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:761: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
      y = column_or_1d(y, warn=True)



```python
os_data_X.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>age</th>
      <th>duration</th>
      <th>campaign</th>
      <th>pdays</th>
      <th>previous</th>
      <th>emp_var_rate</th>
      <th>cons_price_idx</th>
      <th>cons_conf_idx</th>
      <th>euribor3m</th>
      <th>nr_employed</th>
      <th>...</th>
      <th>month_oct</th>
      <th>month_sep</th>
      <th>day_of_week_fri</th>
      <th>day_of_week_mon</th>
      <th>day_of_week_thu</th>
      <th>day_of_week_tue</th>
      <th>day_of_week_wed</th>
      <th>poutcome_failure</th>
      <th>poutcome_nonexistent</th>
      <th>poutcome_success</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>31.0</td>
      <td>259.0</td>
      <td>1.0</td>
      <td>999.0</td>
      <td>1.0</td>
      <td>-0.1</td>
      <td>93.200</td>
      <td>-42.0</td>
      <td>4.153</td>
      <td>5195.8</td>
      <td>...</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>44.0</td>
      <td>300.0</td>
      <td>1.0</td>
      <td>999.0</td>
      <td>0.0</td>
      <td>1.1</td>
      <td>93.994</td>
      <td>-36.4</td>
      <td>4.857</td>
      <td>5191.0</td>
      <td>...</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>43.0</td>
      <td>134.0</td>
      <td>1.0</td>
      <td>999.0</td>
      <td>0.0</td>
      <td>1.1</td>
      <td>93.994</td>
      <td>-36.4</td>
      <td>4.857</td>
      <td>5191.0</td>
      <td>...</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>35.0</td>
      <td>169.0</td>
      <td>2.0</td>
      <td>999.0</td>
      <td>0.0</td>
      <td>-0.1</td>
      <td>93.200</td>
      <td>-42.0</td>
      <td>4.153</td>
      <td>5195.8</td>
      <td>...</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>33.0</td>
      <td>255.0</td>
      <td>2.0</td>
      <td>999.0</td>
      <td>1.0</td>
      <td>-1.8</td>
      <td>92.893</td>
      <td>-46.2</td>
      <td>1.266</td>
      <td>5099.1</td>
      <td>...</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 61 columns</p>
</div>




```python
# we can Check the numbers of our data
print("length of oversampled data is ",len(os_data_X))
print("Number of no subscription in oversampled data",len(os_data_y[os_data_y['y']==0]))
print("Number of subscription",len(os_data_y[os_data_y['y']==1]))
print("Proportion of no subscription data in oversampled data is ",len(os_data_y[os_data_y['y']==0])/len(os_data_X))
print("Proportion of subscription data in oversampled data is ",len(os_data_y[os_data_y['y']==1])/len(os_data_X))
```

    length of oversampled data is  51134
    Number of no subscription in oversampled data 25567
    Number of subscription 25567
    Proportion of no subscription data in oversampled data is  0.5
    Proportion of subscription data in oversampled data is  0.5


### Recursive Feature Elimination


```python
data_final_vars = data_final.columns.values.tolist()
y=['y']
X=[i for i in data_final_vars if i not in y]
```


```python
len(X)
```




    61




```python
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
```


```python
logreg = LogisticRegression(solver='liblinear')
```


```python
rfe = RFE(logreg, 20)
```


```python
rfe = rfe.fit(os_data_X, os_data_y.values.ravel())
print(rfe.support_)
print(rfe.ranking_)
```

    [False False False False False False False False  True False False  True
     False  True False False  True False False False False False False False
     False  True False False  True False False False  True  True False False
     False False False False False  True  True  True  True  True  True  True
     False  True  True False  True False False False False False  True False
      True]
    [39 36 24 40  9 13 22 35  1 31  7  1  6  1  4 30  1  3 27  2  5 10 20 19
     15  1 14 18  1 17 16 21  1  1 41 25 38 33 29 37 32  1  1  1  1  1  1  1
     12  1  1 34  1 28 11 26 42 23  1  8  1]



```python
cols = [os_data_X.columns.values[i] for i,v in enumerate(rfe.support_.tolist()) if bool(v) is True]
```


```python
X=os_data_X[cols]
y=os_data_y['y']
```

###  Implementing the model


```python
import statsmodels.api as sm
```


```python
logit_model = sm.Logit(y, X)
```


```python
result = logit_model.fit()
```

    Warning: Maximum number of iterations has been exceeded.
             Current function value: 0.545860
             Iterations: 35


    /home/bbrighttaer/anaconda3/lib/python3.6/site-packages/statsmodels/base/model.py:508: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
      "Check mle_retvals", ConvergenceWarning)



```python
print(result.summary2())
```

                                    Results: Logit
    ==============================================================================
    Model:                   Logit                Pseudo R-squared:     0.212     
    Dependent Variable:      y                    AIC:                  55863.9960
    Date:                    2019-01-10 00:27     BIC:                  56040.8401
    No. Observations:        51134                Log-Likelihood:       -27912.   
    Df Model:                19                   LL-Null:              -35443.   
    Df Residuals:            51114                LLR p-value:          0.0000    
    Converged:               0.0000               Scale:                1.0000    
    No. Iterations:          35.0000                                              
    ------------------------------------------------------------------------------
                          Coef.    Std.Err.    z     P>|z|     [0.025     0.975]  
    ------------------------------------------------------------------------------
    euribor3m             -0.4637    0.0091 -50.9870 0.0000     -0.4815    -0.4459
    job_blue-collar       -0.1841    0.0285  -6.4615 0.0000     -0.2399    -0.1283
    job_housemaid         -0.3352    0.0779  -4.3054 0.0000     -0.4878    -0.1826
    job_self-employed     -0.1539    0.0613  -2.5127 0.0120     -0.2740    -0.0339
    marital_unknown        0.7487    0.2253   3.3232 0.0009      0.3071     1.1902
    education_illiterate   1.3600    0.4389   3.0990 0.0019      0.4999     2.2202
    default_no            16.1545 5406.2084   0.0030 0.9976 -10579.8193 10612.1282
    default_unknown       15.8968 5406.2084   0.0029 0.9977 -10580.0770 10611.8706
    contact_cellular     -13.3581 5406.2084  -0.0025 0.9980 -10609.3319 10582.6157
    contact_telephone    -13.4276 5406.2084  -0.0025 0.9980 -10609.4013 10582.5462
    month_apr             -1.4097    0.0696 -20.2461 0.0000     -1.5462    -1.2732
    month_aug             -1.2663    0.0694 -18.2586 0.0000     -1.4022    -1.1304
    month_dec             -0.9963    0.1531  -6.5072 0.0000     -1.2964    -0.6962
    month_jul             -0.9787    0.0695 -14.0733 0.0000     -1.1150    -0.8424
    month_jun             -1.0535    0.0673 -15.6494 0.0000     -1.1854    -0.9215
    month_may             -2.0478    0.0627 -32.6745 0.0000     -2.1706    -1.9249
    month_nov             -1.4031    0.0711 -19.7237 0.0000     -1.5425    -1.2637
    month_sep             -0.5806    0.1041  -5.5761 0.0000     -0.7847    -0.3765
    poutcome_failure      -0.5039    0.0363 -13.8884 0.0000     -0.5751    -0.4328
    poutcome_success       1.5751    0.0618  25.4749 0.0000      1.4539     1.6963
    ==============================================================================
    


The p-values for most of the variables are smaller than 0.05, except four variables, therefore, we will remove them.


```python
cols=['euribor3m', 'job_blue-collar', 'job_housemaid', 'marital_unknown', 'education_illiterate', 
      'month_apr', 'month_aug', 'month_dec', 'month_jul', 'month_jun', 'month_mar', 
      'month_may', 'month_nov', 'month_oct', "poutcome_failure", "poutcome_success"] 
X=os_data_X[cols]
y=os_data_y['y']
```


```python
logit_model=sm.Logit(y,X)
result=logit_model.fit()
print(result.summary2())
```

    Optimization terminated successfully.
             Current function value: 0.555890
             Iterations 7
                               Results: Logit
    =====================================================================
    Model:                Logit             Pseudo R-squared:  0.198     
    Dependent Variable:   y                 AIC:               56881.7793
    Date:                 2019-01-10 00:29  BIC:               57023.2546
    No. Observations:     51134             Log-Likelihood:    -28425.   
    Df Model:             15                LL-Null:           -35443.   
    Df Residuals:         51118             LLR p-value:       0.0000    
    Converged:            1.0000            Scale:             1.0000    
    No. Iterations:       7.0000                                         
    ---------------------------------------------------------------------
                          Coef.  Std.Err.    z     P>|z|   [0.025  0.975]
    ---------------------------------------------------------------------
    euribor3m            -0.4488   0.0074 -60.6838 0.0000 -0.4632 -0.4343
    job_blue-collar      -0.2097   0.0278  -7.5345 0.0000 -0.2643 -0.1552
    job_housemaid        -0.2793   0.0762  -3.6636 0.0002 -0.4287 -0.1299
    marital_unknown       0.7618   0.2244   3.3953 0.0007  0.3220  1.2016
    education_illiterate  1.3085   0.4346   3.0108 0.0026  0.4567  2.1604
    month_apr             1.2863   0.0380  33.8252 0.0000  1.2117  1.3608
    month_aug             1.3948   0.0411  33.9435 0.0000  1.3143  1.4754
    month_dec             1.8090   0.1441  12.5523 0.0000  1.5265  2.0915
    month_jul             1.6763   0.0424  39.5459 0.0000  1.5932  1.7594
    month_jun             1.5586   0.0408  38.1651 0.0000  1.4786  1.6387
    month_mar             2.8220   0.0908  31.0943 0.0000  2.6442  2.9999
    month_may             0.5867   0.0304  19.2814 0.0000  0.5270  0.6463
    month_nov             1.2735   0.0445  28.5964 0.0000  1.1863  1.3608
    month_oct             2.7225   0.0815  33.3924 0.0000  2.5627  2.8823
    poutcome_failure     -0.2803   0.0351  -7.9904 0.0000 -0.3490 -0.2115
    poutcome_success      1.9619   0.0602  32.5957 0.0000  1.8439  2.0798
    =====================================================================
    



```python
from sklearn import metrics
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
```

    /home/bbrighttaer/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)





    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='warn',
              n_jobs=None, penalty='l2', random_state=None, solver='warn',
              tol=0.0001, verbose=0, warm_start=False)




```python
y_pred = logreg.predict(X_test)
```


```python
y_pred
```




    array([0, 1, 1, ..., 0, 1, 0])




```python
print('Accuracy of logistic regression classifier on test set: {:.2f}'.format(logreg.score(X_test, y_test)))
```

    Accuracy of logistic regression classifier on test set: 0.74



```python
from sklearn.metrics import confusion_matrix
confusion_matrix = confusion_matrix(y_test, y_pred)
print(confusion_matrix)
```

    [[6124 1542]
     [2505 5170]]



```python
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))
```

                  precision    recall  f1-score   support
    
               0       0.71      0.80      0.75      7666
               1       0.77      0.67      0.72      7675
    
       micro avg       0.74      0.74      0.74     15341
       macro avg       0.74      0.74      0.74     15341
    weighted avg       0.74      0.74      0.74     15341
    



```python
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
logit_roc_auc = roc_auc_score(y_test, logreg.predict(X_test))
fpr, tpr, thresholds = roc_curve(y_test, logreg.predict_proba(X_test)[:,1])
plt.figure()
plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)' % logit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
# plt.savefig('Log_ROC')
plt.show()
```


```python

```

7d07c0c8c03e783a52f791911563a8fb5c17ce29
ipynb
Jupyter Notebook
logisticRegression.ipynb
bbrighttaer/data_science_nbs
21c1b088e758b0cf801bc9c8da87dfd916561163
MIT
logisticRegression.ipynb
bbrighttaer/data_science_nbs
21c1b088e758b0cf801bc9c8da87dfd916561163
MIT
logisticRegression.ipynb
bbrighttaer/data_science_nbs
21c1b088e758b0cf801bc9c8da87dfd916561163
MIT
Qwen/Qwen-72B
1. YES
2. YES