# Count Based Vectorizer


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import (
    TfidfVectorizer, CountVectorizer, TfidfTransformer
)
```

## References 

https://en.wikipedia.org/wiki/Tf%E2%80%93idf

## Count Vectorizer

frequency based on the set of words


```python
corpus = [
    'this is the first document',
    'this document is the second document',
    'and this is the third one'
]
```


```python
count_vec = CountVectorizer()
count_vec.fit(corpus)
```




    CountVectorizer()




```python
count_vec.get_feature_names()
```




    ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']




```python
count_data = count_vec.transform(corpus).toarray()
count_data
```




    array([[0, 1, 1, 1, 0, 0, 1, 0, 1],
           [0, 2, 0, 1, 0, 1, 1, 0, 1],
           [1, 0, 0, 1, 1, 0, 1, 1, 1]])




```python
pd.DataFrame(
    data=count_data,
    columns=count_vec.get_feature_names()
)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>and</th>
      <th>document</th>
      <th>first</th>
      <th>is</th>
      <th>one</th>
      <th>second</th>
      <th>the</th>
      <th>third</th>
      <th>this</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0</td>
      <td>2</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
</div>



So the first line is `this is the first document`. and then the first vector is reflecting as document is 1, first is 1, is is 1 etc.

## Term Frequency Transformer

Number of times term t, appear in the document d.

\begin{align}
    tf_{t,d} &= \frac{n_{t,d}}{\sum_k{n_{k,d}}} 
\end{align}



```python
tf_transformer = TfidfTransformer(use_idf=False)
tf_transformer.fit(count_data)
```




    TfidfTransformer(use_idf=False)




```python
tf_data = tf_transformer.transform(count_data).toarray()
tf_data
```




    array([[0.        , 0.4472136 , 0.4472136 , 0.4472136 , 0.        ,
            0.        , 0.4472136 , 0.        , 0.4472136 ],
           [0.        , 0.70710678, 0.        , 0.35355339, 0.        ,
            0.35355339, 0.35355339, 0.        , 0.35355339],
           [0.40824829, 0.        , 0.        , 0.40824829, 0.40824829,
            0.        , 0.40824829, 0.40824829, 0.40824829]])




```python
pd.DataFrame(
    data=tf_data,
    columns=count_vec.get_feature_names()
)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>and</th>
      <th>document</th>
      <th>first</th>
      <th>is</th>
      <th>one</th>
      <th>second</th>
      <th>the</th>
      <th>third</th>
      <th>this</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.000000</td>
      <td>0.447214</td>
      <td>0.447214</td>
      <td>0.447214</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.447214</td>
      <td>0.000000</td>
      <td>0.447214</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.000000</td>
      <td>0.707107</td>
      <td>0.000000</td>
      <td>0.353553</td>
      <td>0.000000</td>
      <td>0.353553</td>
      <td>0.353553</td>
      <td>0.000000</td>
      <td>0.353553</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.408248</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.408248</td>
      <td>0.408248</td>
      <td>0.000000</td>
      <td>0.408248</td>
      <td>0.408248</td>
      <td>0.408248</td>
    </tr>
  </tbody>
</table>
</div>



## Inverse Document Frequency Transformer


> `Document Frequency` : number of ducuments that the term appears / number of ducuments

\begin{align}
    probability = df_{t,d,D} &= \frac{d \in D : t \in d}{D} 
\end{align}

The log of the number of documents D divided by the number of documents that contain the word t. Inverse data frequency determines the weight of rare words across all documents in the corpus.



\begin{align}
    idf &= -log(p)\\
    idf &= log(\frac{1}{p})\\
    idf_{t,d,D} &= log(\frac{D}{d \in D : t \in d})
\end{align}

> document d is in all the documents D, and term t is in the document d.


```python
tfidf_transformer = TfidfTransformer()
tfidf_transformer.fit(count_data)
```




    TfidfTransformer()




```python
tfidf_data = tfidf_transformer.transform(count_data).toarray()
tfidf_data
```




    array([[0.        , 0.46941728, 0.61722732, 0.3645444 , 0.        ,
            0.        , 0.3645444 , 0.        , 0.3645444 ],
           [0.        , 0.7284449 , 0.        , 0.28285122, 0.        ,
            0.47890875, 0.28285122, 0.        , 0.28285122],
           [0.49711994, 0.        , 0.        , 0.29360705, 0.49711994,
            0.        , 0.29360705, 0.49711994, 0.29360705]])




```python
pd.DataFrame(
    data=tfidf_data,
    columns=count_vec.get_feature_names()
)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>and</th>
      <th>document</th>
      <th>first</th>
      <th>is</th>
      <th>one</th>
      <th>second</th>
      <th>the</th>
      <th>third</th>
      <th>this</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.00000</td>
      <td>0.469417</td>
      <td>0.617227</td>
      <td>0.364544</td>
      <td>0.00000</td>
      <td>0.000000</td>
      <td>0.364544</td>
      <td>0.00000</td>
      <td>0.364544</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.00000</td>
      <td>0.728445</td>
      <td>0.000000</td>
      <td>0.282851</td>
      <td>0.00000</td>
      <td>0.478909</td>
      <td>0.282851</td>
      <td>0.00000</td>
      <td>0.282851</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.49712</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.293607</td>
      <td>0.49712</td>
      <td>0.000000</td>
      <td>0.293607</td>
      <td>0.49712</td>
      <td>0.293607</td>
    </tr>
  </tbody>
</table>
</div>



## TFIDF Vectorizer

Count Vectorizer --> Tfidf Transformer = Tfidf Vectorizer

$tfidf_{t,d,D} = tf_{t,d}.idf_{t,d,D}$


```python
tfidf_vec = TfidfVectorizer()
tfidf_vec.fit(corpus)
```




    TfidfVectorizer()




```python
tfidf_data = tfidf_vec.transform(corpus).toarray()
```


```python
tfidf_vec.get_feature_names()
```




    ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']




```python
pd.DataFrame(
    data=tfidf_data,
    columns=tfidf_vec.get_feature_names()
)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>and</th>
      <th>document</th>
      <th>first</th>
      <th>is</th>
      <th>one</th>
      <th>second</th>
      <th>the</th>
      <th>third</th>
      <th>this</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.00000</td>
      <td>0.469417</td>
      <td>0.617227</td>
      <td>0.364544</td>
      <td>0.00000</td>
      <td>0.000000</td>
      <td>0.364544</td>
      <td>0.00000</td>
      <td>0.364544</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.00000</td>
      <td>0.728445</td>
      <td>0.000000</td>
      <td>0.282851</td>
      <td>0.00000</td>
      <td>0.478909</td>
      <td>0.282851</td>
      <td>0.00000</td>
      <td>0.282851</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.49712</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.293607</td>
      <td>0.49712</td>
      <td>0.000000</td>
      <td>0.293607</td>
      <td>0.49712</td>
      <td>0.293607</td>
    </tr>
  </tbody>
</table>
</div>



081927506152f0d7f7806493a0739fc94860204c
ipynb
Jupyter Notebook
Concepts/tfidf.ipynb
NishantBaheti/NatualLanguageProcessing
5437c5bbbba1ef13211c62cda14b95607aa5456e
MIT
2022-02-10T06:49:19.000Z
2022-02-10T06:49:19.000Z
Concepts/tfidf.ipynb
NishantBaheti/NaturalLanguageProcessing
5437c5bbbba1ef13211c62cda14b95607aa5456e
MIT
Concepts/tfidf.ipynb
NishantBaheti/NaturalLanguageProcessing
5437c5bbbba1ef13211c62cda14b95607aa5456e
MIT
Qwen/Qwen-72B
1. YES
2. YES