# 0. Imports


```python
import numpy as np
import pandas as pd

from sklearn.model_selection import train_test_split
from metrics import euclidean_similarity, pearson_similarity
import numpy.ma as ma

import scipy.sparse as scp

from metrics import mapk

from elementary import ElementaryRecommendation
```

# 1. Models description

Let $U$ be the set of users, $I$ - set of Items


## Baseline

### **Elementary Recommendation**

The most simlest way to predict ratings could be just a predicting average rating of all items, but this is not good at all. So for a baseline we can consider users and items bias. For example, some users rate the movies only in range 1-4 out of 10: rating 5 for them is something phenomenal; or users can also rate movies only highly.

Let $\mu$ be the average rating of all items, $r_{ui}$ - rating of item $i$ by user $u$, $I_u$ - items that were rated by user $u$. Then we can calculate user bias $b_u$ as:

$$
\begin{align*}
b_u=&\frac{1}{|I_u|+\alpha}\sum_{i\in{}I_u}(r_{ui} - \mu)
\end{align*}, 
$$

where $\alpha$ is smoothing coefficient.

We can similarly define item bias, but now let's take into account calculated users biases. Let $U_i$ be the users that rated item $i$. So then item bias $b_i$ is:

$$
\begin{align*}
b_i=&\frac{1}{|U_i|+\alpha}\sum_{u\in{}U_i}(r_{ui} - b_u - \mu)
\end{align*}
$$

**Final prediction** will be:

$$
\begin{align*}
r_{ui} = \mu+b_u+b_i
\end{align*}
$$

> ##### See the implementation of this approach in **elementary.py** file.

## Similarity based

### **Metrics**

How can the similarity of users/items be compared? Let's make an assumption that two users are similar if they rate items in a similar way and two items are similar if they are rated by users in a similar way. In this case we will use similarity metrics based on Euclidean distance and Pearson correlation coefficient which are suitable for this approach.

#### **Euclidean similarity**

Euclidean distance between two points $x$ and $y$ in Euclidean n-space can be calculated by the following equation:

$$
\begin{equation}
d(x, y) = \sqrt{\sum_{k=1}^n (x_k-y_k)^2}
\end{equation}
$$

Euclidean distance is inversely propotional to similarity so we should use inverted value of $d(x, y)$. Adding 1 in the denominator to avoid zero division we get the Euclidean similarity:

$$
\begin{equation}
sim(p, q) = \frac{1}{1 + d(p, q)}
\end{equation}
$$

#### **Pearson similarity**

Let $I_{uv}$ be the items that were rated both by users $u$ and $v$, $U_{ij}$ - users that rated both items $i$ and $j$:

$$
I_{uv} = \{i \in I \space | \space \exists r_{ui} \space \& \space \exists r_{vi}\}
$$
$$
U_{ij} = \{u \in U \space | \space \exists r_{ui} \space \& \space \exists r_{uj}\}
$$

Then the Pearson similarity between two users $u$ and $v$ can be calculated as:

$$
\begin{equation}
sim(u, v) = \frac {\sum_{i \in I_{uv}} \left( r_{ui} - \mu_u \right) \left( r_{vi} - \mu_v \right)}
                  {\sqrt{\sum_{i \in I_{uv}} \left( r_{ui} - \mu_u \right)^2}
                   \sqrt{\sum_{i \in I_{uv}} \left( r_{vi} - \mu_v \right)^2}}
\end{equation}
$$

And between two items $i$ and $j$:

$$
\begin{equation}
sim(i, j) = \frac {\sum_{u \in U_{ij}} \left( r_{ui} - \mu_i \right) \left( r_{uj} - \mu_j \right)}
                  {\sqrt{\sum_{u \in U_{ij}} \left( r_{ui} - \mu_i \right)^2}
                   \sqrt{\sum_{u \in U_{ij}} \left( r_{uj} - \mu_j \right)^2}}
\end{equation}
$$

> ##### See the implementation of these metrics in **metrics.py** file.

### **Trivial Recommendation**

Now let's consider an approach using similarity.

Let $i_0$ be the item that was highly rated by user $u_0$. We can find all users that rated this item (some kind of collaboration):

$$
U(i_0) = \{ u \in U \space | \space \exists r_{ui_0} \}
$$

Next we can comprise the set of items that were rated by users from $U(i_0)$:

$$
I(i_0) = \{ i \in I \space | \space \exists r_{ui}, \space u \in U(i_0) \}
$$

**Final prediction** will be top $N$ items from the set of items $I(i_0)$ sorted by similarity $sim(i, i_0)$ to the starting item $i_0$ .

> ##### See the implementation of this approach in **trivial.py** file.

### **User-based Recommendation**

Let $u_0$ be the user to whom we want to recommend. Let's comprize $U(u_0)$ as a set of top $K$ users that are most similar to $u_0$. We can find $U(u_0)$ as Nearest Neighbors to $u_0$ by one of the similarity metrics defined above.

The value of ratings user $u_0$ supposedly gives to item $i$ is calculated as an aggregation of some similar users rating of the item. In this case we will use:

$$
r_{u_{0}i}=\bar{r}_{u_0} + \frac{\sum_{u \in U(u_0)} sim(u, u_0)(r_{ui}-\bar{r}_u)}{\sum_{u \in U(u_0)}|sim(u, u_0)|},
$$

where $\bar{r}_{u^\prime}$ is the average rating of user $u^\prime$ for all the items rated by $u^\prime$.

**Final prediction** will be top $N$ items that have the highest predicted rating.

> ##### See the implementation of this approach in **user_based.py** file.

# 2. Data preparing

We will use subset of **The Movies Dataset** from **kaggle** to evaluate our models.


```python
data = pd.read_csv('ratings_small.csv', index_col=False)
data.shape
```




    (100004, 4)




```python
data.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>userId</th>
      <th>movieId</th>
      <th>rating</th>
      <th>timestamp</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>31</td>
      <td>2.5</td>
      <td>1260759144</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>1029</td>
      <td>3.0</td>
      <td>1260759179</td>
    </tr>
    <tr>
      <th>2</th>
      <td>1</td>
      <td>1061</td>
      <td>3.0</td>
      <td>1260759182</td>
    </tr>
    <tr>
      <th>3</th>
      <td>1</td>
      <td>1129</td>
      <td>2.0</td>
      <td>1260759185</td>
    </tr>
    <tr>
      <th>4</th>
      <td>1</td>
      <td>1172</td>
      <td>4.0</td>
      <td>1260759205</td>
    </tr>
  </tbody>
</table>
</div>




```python
print((data.userId.min(), data.userId.max(), len(data.userId.unique())))
print((data.movieId.min(), data.movieId.max(), len(data.movieId.unique())))
```

    (1, 671, 671)
    (1, 163949, 9066)


There are many missings id's so it would be more convenient if we changed the numbering.


```python
user_to_idx = {userId : idx for idx, userId in enumerate(data.userId.unique())}
movie_to_idx = {movieId : idx for idx, movieId in enumerate(data.movieId.unique())}

data.replace({'userId': user_to_idx, 'movieId': movie_to_idx}, inplace=True)
```


```python
print((data.userId.min(), data.userId.max(), len(data.userId.unique())))
print((data.movieId.min(), data.movieId.max(), len(data.movieId.unique())))
```

    (0, 670, 671)
    (0, 9065, 9066)


Let's split our data into train and test this way: we will remove the top 5 ratings from the most active users and put them in test data.


```python
train_data = data.copy()
test_data = pd.DataFrame([], columns=train_data.columns)

active_users = train_data.userId.value_counts()[:10].index

for user_id in active_users:
    test = train_data[train_data.userId == user_id].sort_values(by='rating', ascending=False)[:5]
    test_data = test_data.append(test, ignore_index=True)
    train_data = train_data[~((train_data.userId == user_id) & (train_data.movieId.isin(test.movieId.values)))]

train_data.shape, test_data.shape
```




    ((99954, 4), (50, 4))



It would be rather well to use [sparse matrix](https://en.wikipedia.org/wiki/Sparse_matrix) for storing all ratings, but in this case let's use dense one for simplifying some actions.


```python
X = scp.coo_matrix(
    (
        train_data['rating'],  # users ratings
        (train_data['userId'], train_data['movieId'])  # users and movies id's whose ratings are known
        ),
        shape=(len(user_to_idx), len(movie_to_idx))  # ratings matrix shape
    ).tocsr().A
```


```python
X
```




    array([[2.5, 3. , 3. , ..., 0. , 0. , 0. ],
           [0. , 0. , 0. , ..., 0. , 0. , 0. ],
           [0. , 0. , 0. , ..., 0. , 0. , 0. ],
           ...,
           [0. , 0. , 0. , ..., 0. , 0. , 0. ],
           [0. , 0. , 0. , ..., 0. , 0. , 0. ],
           [0. , 0. , 0. , ..., 0. , 0. , 0. ]])



# 3. Quantifying the quality of recommendations.


```python
predicted = []

er = ElementaryRecommendation()
er.fit(X)

for id in test_data.userId.unique():
    predicted.append(er.make_recommendation(user_id=id, n_recommendations=100))
```


```python
actual = []

for id in test_data.userId.unique():
    actual.append(test_data[test_data.userId == id]['movieId'].astype(int).values)
```


```python
print(mapk(actual=actual, predicted=predicted, k=5))
print(mapk(actual=actual, predicted=predicted, k=10))
print(mapk(actual=actual, predicted=predicted, k=100))
```

    0.0
    0.0
    2.2222222222222223e-05



```python
from metrics import recsys_euclidean_similarity
```


```python

```

a366be32d786ba93512a6869d6dade92fdd606dd
ipynb
Jupyter Notebook
research.ipynb
UsefulTornado/Recommendation-systems
80f303c53918aabccef4f06406e86eb26d8a6237
MIT
research.ipynb
UsefulTornado/Recommendation-systems
80f303c53918aabccef4f06406e86eb26d8a6237
MIT
research.ipynb
UsefulTornado/Recommendation-systems
80f303c53918aabccef4f06406e86eb26d8a6237
MIT
Qwen/Qwen-72B
1. YES
2. YES