```python
import math
import torch
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader

import io
import torchtext
from torchtext.utils import download_from_url, extract_archive
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator

import pytorch_lightning as pl
from pytorch_lightning.trainer.trainer import Trainer
from pytorch_lightning import seed_everything
seed_everything(42)

```

    42

# Dataset Preparation

This tutorial uses ``torchtext`` to generate Wikitext-2 dataset. The
vocab object is built based on the train dataset and is used to numericalize
tokens into tensors.

With the alphabet as the sequence (total length of 26)
and a batch size of 4, we would divide the alphabet into 4 sequences of
length 6:

\begin{align}\begin{bmatrix}
  \text{A} & \text{B} & \text{C} & \ldots & \text{X} & \text{Y} & \text{Z}
  \end{bmatrix}
  \Rightarrow
  \begin{bmatrix}
  \begin{bmatrix}\text{A} \\ \text{B} \\ \text{C} \\ \text{D} \\ \text{E} \\ \text{F}\end{bmatrix} &
  \begin{bmatrix}\text{G} \\ \text{H} \\ \text{I} \\ \text{J} \\ \text{K} \\ \text{L}\end{bmatrix} &
  \begin{bmatrix}\text{M} \\ \text{N} \\ \text{O} \\ \text{P} \\ \text{Q} \\ \text{R}\end{bmatrix} &
  \begin{bmatrix}\text{S} \\ \text{T} \\ \text{U} \\ \text{V} \\ \text{W} \\ \text{X}\end{bmatrix}
  \end{bmatrix}\end{align}

These columns are treated as independent by the model, which means that
the dependence of ``G`` and ``F`` can not be learned, but allows more
efficient batch processing.

# Inputa and Target

For the language modeling task, the model needs the following words as Target. For example, with a seq_len value of 2, we’d get the following two Variables.

**Note:** It should be noted that the chunks are along dimension 0, consistent with the S dimension in the Transformer model. The batch dimension N is along dimension 1.

```python
class AlarmDataset(Dataset):
    def __init__(self,data,seq_len,batch_size):
        self.length = len(data)//seq_len # how much data i have
        self.data = data
        self.seq_len = seq_len
        self.batch_size = batch_size

    def __getitem__(self, index: int):
        x = self.data[index*self.seq_len:(index*self.seq_len)+seq_len]
        y = self.data[1+index*self.seq_len:1+(index*self.seq_len)+seq_len]
        return x,y

    def __len__(self) -> int:
        return self.length

class MyDataModule(pl.LightningDataModule):

    def __init__(self, data_path:str, batch_size:int, seq_len:int):
        super().__init__()
        self.batch_size = batch_size

        url = data_path
        test_filepath, valid_filepath, train_filepath = extract_archive(download_from_url(url))
        self.tokenizer = get_tokenizer('basic_english')
        self.vocab = build_vocab_from_iterator(map(self.tokenizer,iter(io.open(train_filepath,encoding="utf8"))))

        train_data = self.data_process(iter(io.open(train_filepath, encoding="utf8")))
        val_data = self.data_process(iter(io.open(valid_filepath, encoding="utf8")))
        test_data = self.data_process(iter(io.open(test_filepath, encoding="utf8")))

        # print(train_data[0:100])
        # print([])

        self.train_dataset = AlarmDataset(train_data, seq_len,self.batch_size)
        self.valid_dataset = AlarmDataset(val_data,seq_len,self.batch_size)
        self.test_dataset = AlarmDataset(test_data, seq_len,self.batch_size)

    def data_process(self, raw_text_iter):
        data = [torch.tensor([self.vocab[token] for token in self.tokenizer(item)],dtype=torch.long) for item in raw_text_iter]
        return torch.cat(tuple(filter(lambda t: t.numel() > 0, data)))

    def setup(self, stage: None):
        return None

    def train_dataloader(self) -> DataLoader:
        return DataLoader(self.train_dataset, batch_size=self.batch_size, shuffle=False,num_workers=1,drop_last=True, pin_memory=True)

    def val_dataloader(self) -> DataLoader:
        return DataLoader(self.valid_dataset, batch_size=self.batch_size, shuffle=False,num_workers=1,drop_last=True, pin_memory=True)

    def test_dataloader(self):
        return DataLoader(self.test_dataset, batch_size=self.batch_size, shuffle=False,num_workers=1,drop_last=True, pin_memory=True)

```

# Transformer Model

In this tutorial, we train ``nn.TransformerEncoder`` model on a
language modeling task. The language modeling task is to assign a
probability for the likelihood of a given word (or a sequence of words)
to follow a sequence of words. A sequence of tokens are passed to the embedding
layer first, followed by a positional encoding layer to account for the order
of the word (see the next paragraph for more details). The
``nn.TransformerEncoder`` consists of multiple layers of
`nn.TransformerEncoderLayer <https://pytorch.org/docs/master/nn.html?highlight=transformerencoderlayer#torch.nn.TransformerEncoderLayer>`__. Along with the input sequence, a square
attention mask is required because the self-attention layers in
``nn.TransformerEncoder`` are only allowed to attend the earlier positions in
the sequence. For the language modeling task, any tokens on the future
positions should be masked. To have the actual words, the output
of ``nn.TransformerEncoder`` model is sent to the final Linear
layer, which is followed by a log-Softmax function.

# Positional Encoding

``PositionalEncoding`` module injects some information about the
relative or absolute position of the tokens in the sequence. The
positional encodings have the same dimension as the embeddings so that
the two can be summed. Here, we use ``sine`` and ``cosine`` functions of
different frequencies.

```python
class TransformerModel(pl.LightningModule):

    def __init__(self, ntoken, ninp, nhead, nhid, nlayers, dropout=0.5, seq_len=None):
        super(TransformerModel, self).__init__()
        self.model_type = 'Transformer'
        self.ntoken = ntoken
        self.pos_encoder = PositionalEncoding(ninp, dropout)
        encoder_layers = torch.nn.TransformerEncoderLayer(ninp, nhead, nhid, dropout)
        self.transformer_encoder = torch.nn.TransformerEncoder(encoder_layers, nlayers)
        self.encoder = torch.nn.Embedding(ntoken, ninp)
        self.ninp = ninp
        self.decoder = torch.nn.Linear(ninp, ntoken)
        self.src_mask = self.generate_square_subsequent_mask(seq_len)
        self.seq_len = seq_len
        self.init_weights()

    def generate_square_subsequent_mask(self, sz):
        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask

    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, src, src_mask):
        src_mask = src_mask.to(self.device)
        src = self.encoder(src) * math.sqrt(self.ninp)
        src = self.pos_encoder(src)
        src_mask = src_mask.to(self.device)

        output = self.transformer_encoder(src, src_mask)
        output = self.decoder(output)

        return output

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=0.001, weight_decay=0.0000001)
        return optimizer

    def training_step(self,batch,batch_idx):
        x,y = batch
        x = x.T
        y = y.T.reshape(-1)

        # print("Training Shape: ", x.size(),y.size())

        if x.size(0) != self.seq_len:
           self.src_mask =  self.generate_square_subsequent_mask(x.size(0))

        y_hat = self(x,self.src_mask)

        loss = F.cross_entropy(y_hat.view(-1, self.ntoken),y)
        self.log('train_loss', loss,on_step=True, prog_bar=True, logger=True)
        self.log("train_ppl",math.exp(loss.item()),on_step=True, prog_bar=True, logger=True)
        return loss

    def validation_step(self,batch, batch_idx):
        x,y = batch
        x = x.T
        y = y.T.reshape(-1)

        # print("Validation Shape: ", x.size(),y.size())

        if x.size(0) != self.seq_len:
           self.src_mask =  self.generate_square_subsequent_mask(x.size(0))

        y_hat = self(x,self.src_mask)
        # print("> y-hat",y_hat.size())
        loss = F.cross_entropy(y_hat.view(-1, self.ntoken),y)
        self.log('val_loss', loss, on_step=True, prog_bar=True, logger=True)
        self.log("val_ppl",math.exp(loss.item()),on_step=True, prog_bar=True, logger=True)
        return {'val_loss':loss}

    def test_step(self,batch, batch_idx):
        x,y = batch
        x = x.T
        y = y.T.reshape(-1)

        # print("Validation Shape: ", x.size(),y.size())

        if x.size(0) != self.seq_len:
           self.src_mask =  self.generate_square_subsequent_mask(x.size(0))

        y_hat = self(x,self.src_mask)
        # print("> y-hat",y_hat.size())
        loss = F.cross_entropy(y_hat.view(-1, self.ntoken),y)
        self.log('test_loss', loss, on_step=True, prog_bar=True, logger=True)
        self.log("test_ppl",math.exp(loss.item()),on_step=True, prog_bar=True, logger=True)
        return {'test_loss':loss}

    def training_epoch_end(self, outputs):
        avg_loss = torch.stack([d['loss']  for d in outputs]).mean()
        print(f"> Avg Training loss = {avg_loss}")

    def validation_epoch_end(self, outputs):
        # print(outputs)
        avg_loss = torch.stack([d['val_loss'] for d in outputs]).mean()
        print(f"> Average Valid Loss = {avg_loss}")

    def test_epoch_end(self, outputs):
        avg_loss = torch.stack([d['test_loss'] for d in outputs]).mean()
        print(f"> Average Test Loss = {avg_loss}")

class PositionalEncoding(torch.nn.Module):

    def __init__(self, d_model, dropout=0.1, max_len=5000):
        super(PositionalEncoding, self).__init__()
        self.dropout = torch.nn.Dropout(p=dropout)

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(0), :]
        return self.dropout(x)
```

# Hyperparameters

```python
url = 'https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip'

bsize = 20
seq_len = 35
dm = MyDataModule(url,bsize,seq_len)

ntokens = len(dm.vocab.stoi) # the size of vocabulary
emsize = 200 # embedding dimension
nhid = 200 # the dimension of the feedforward network model in nn.TransformerEncoder
nlayers = 2 # the number of nn.TransformerEncoderLayer in nn.TransformerEncoder
nhead = 2 # the number of heads in the multiheadattention models
dropout = 0.2 # the dropout value

model = TransformerModel(ntokens, emsize, nhead, nhid, nlayers, dropout,seq_len=seq_len)
trainer = Trainer(gpus=1,max_epochs=1,check_val_every_n_epoch=1,deterministic=True, gradient_clip_val=0.5)
trainer.fit(model,dm) # traning and validation
```

    36718lines [00:01, 29608.89lines/s]
    GPU available: True, used: True
    INFO:lightning:GPU available: True, used: True
    TPU available: None, using: 0 TPU cores
    INFO:lightning:TPU available: None, using: 0 TPU cores
    LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
    INFO:lightning:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

      | Name                | Type               | Params
    -----------------------------------------------------------
    0 | pos_encoder         | PositionalEncoding | 0
    1 | transformer_encoder | TransformerEncoder | 484 K
    2 | encoder             | Embedding          | 5.8 M
    3 | decoder             | Linear             | 5.8 M
    -----------------------------------------------------------
    12.0 M    Trainable params
    0         Non-trainable params
    12.0 M    Total params
    INFO:lightning:
      | Name                | Type               | Params
    -----------------------------------------------------------
    0 | pos_encoder         | PositionalEncoding | 0
    1 | transformer_encoder | TransformerEncoder | 484 K
    2 | encoder             | Embedding          | 5.8 M
    3 | decoder             | Linear             | 5.8 M
    -----------------------------------------------------------
    12.0 M    Trainable params
    0         Non-trainable params
    12.0 M    Total params

    Validation sanity check: |          | 0/? [00:00<?, ?it/s]

    > Average Valid Loss = 10.613945960998535

    Training: |          | 0/? [00:00<?, ?it/s]

    Validating: |          | 0/? [00:00<?, ?it/s]

    > Average Valid Loss = 5.521843433380127
    > Avg Training loss = 6.123049259185791

    1

```python
trainer.test(datamodule=dm) # testing
# %%
```

    Testing: |          | 0/? [00:00<?, ?it/s]

    > Average Test Loss = 5.444727897644043
    --------------------------------------------------------------------------------
    DATALOADER:0 TEST RESULTS
    {'test_loss': tensor(5.7169, device='cuda:0'),
     'test_loss_epoch': tensor(5.4447, device='cuda:0'),
     'test_ppl': 303.96751971676264,
     'test_ppl_epoch': tensor(241.8977),
     'val_loss': tensor(5.3272, device='cuda:0'),
     'val_loss_epoch': tensor(5.5218, device='cuda:0'),
     'val_ppl': 205.85485347893808,
     'val_ppl_epoch': tensor(258.1052)}
    --------------------------------------------------------------------------------

    [{'val_loss_epoch': 5.521843910217285,
      'val_ppl_epoch': 258.105224609375,
      'val_loss': 5.327171325683594,
      'val_ppl': 205.85485347893808,
      'test_loss_epoch': 5.444727897644043,
      'test_ppl_epoch': 241.89773559570312,
      'test_loss': 5.716920852661133,
      'test_ppl': 303.96751971676264}]


# [Module 2.1] 피쳐 엔지니어링

이 노트북은 아래와 같은 피쳐 엔지니어링을 통하여 새로운 피쳐를 생성 합니다.
- 날짜관련 피쳐 생성(월, 일, 요일)
- 기존의 피쳐들을 결합하여 새로운 피쳐 생성 (피쳐1 + 피쳐2 = 뉴피쳐)
- Product_ID를 기준으로 Target Encoding 하여 새로운 피쳐 생성
- Product_ID를 기준으로 Target Encoding Smoothing 하여 새로운 피쳐 생성
- Category 피쳐를 레이블 인코딩 함
- 로컬에 데이터 저장
    - 최종 레이블 인코딩 된 데이터 세트 저장 (XGBoost, CatBoost 용)
    - 레이블 인코딩 안한 데이터 세트 저장 (AutoGluon 용)

```python
import pandas as pd
pd.options.display.max_rows=5
import numpy as np
```

```python
%store -r full_data_file_name
```

### 데이터 로딩 및 셔플링

```python
df = pd.read_csv(full_data_file_name)
df = df.sample(frac=1.0, random_state=1000)
df
```

      classes
      order_approved_at
      customer_id
      customer_zip_code_prefix
      customer_city
      customer_state
      price
      freight_value
      product_id
      product_weight_g
      product_length_cm
      product_height_cm
      product_width_cm
      product_category_name_english
      seller_zip_code_prefix
      seller_city
      seller_state

      37413
      3
      2018-07-14 13:04:07
      4ee61c3905a5c398d44b089108961bb3
      28950
      armacao dos buzios
      RJ
      105.00
      27.04
      2f13d1dc8b4e1d9d8027be50339546a9
      2650.0
      30.0
      30.0
      30.0
      furniture_decor
      3204
      sao paulo
      SP

      54762
      3
      2017-03-25 10:25:16
      959292edcade77d6b60dc8f49f01cd71
      37880
      cabo verde
      MG
      23.99
      14.52
      b000447e24e31a4d7e628ca4d0622131
      250.0
      19.0
      4.0
      11.0
      telephony
      3504
      sao paulo
      SP

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      18782
      3
      2018-07-16 18:20:34
      609b34a18fd0d61719d0b3190ec05231
      21240
      rio de janeiro
      RJ
      200.00
      19.50
      92d5ae8e42c4599266f210bf0469ae9b
      439.0
      18.0
      11.0
      13.0
      watches_gifts
      14050
      ribeirao preto
      SP

      3776
      1
      2018-05-25 07:06:30
      dc275c5e585b7c3a3cddef527e34fc19
      26540
      nilopolis
      RJ
      18.90
      7.55
      15de022edf1005363381e66bed514528
      100.0
      16.0
      3.0
      23.0
      furniture_decor
      20270
      rio de janeiro
      RJ

67176 rows × 17 columns

```python
df.columns
```

    Index(['classes', 'order_approved_at', 'customer_id',
           'customer_zip_code_prefix', 'customer_city', 'customer_state', 'price',
           'freight_value', 'product_id', 'product_weight_g', 'product_length_cm',
           'product_height_cm', 'product_width_cm',
           'product_category_name_english', 'seller_zip_code_prefix',
           'seller_city', 'seller_state'],
          dtype='object')

## 날짜 피쳐 생성: Month, Day, WeeoOfDay(요일)

```python
def create_date_feature(raw_df):
    df = raw_df.copy()
    df['order_date'] = pd.to_datetime(df['order_approved_at'])
    df['order_weekday'] = df['order_date'].dt.weekday
    df['order_day'] = df['order_date'].dt.day
    df['order_month'] = df['order_date'].dt.month
    return df

f_df = create_date_feature(df)
f_df
```

      classes
      order_approved_at
      customer_id
      customer_zip_code_prefix
      customer_city
      customer_state
      price
      freight_value
      product_id
      product_weight_g
      ...
      product_height_cm
      product_width_cm
      product_category_name_english
      seller_zip_code_prefix
      seller_city
      seller_state
      order_date
      order_weekday
      order_day
      order_month

      37413
      3
      2018-07-14 13:04:07
      4ee61c3905a5c398d44b089108961bb3
      28950
      armacao dos buzios
      RJ
      105.00
      27.04
      2f13d1dc8b4e1d9d8027be50339546a9
      2650.0
      ...
      30.0
      30.0
      furniture_decor
      3204
      sao paulo
      SP
      2018-07-14 13:04:07
      5
      14
      7

      54762
      3
      2017-03-25 10:25:16
      959292edcade77d6b60dc8f49f01cd71
      37880
      cabo verde
      MG
      23.99
      14.52
      b000447e24e31a4d7e628ca4d0622131
      250.0
      ...
      4.0
      11.0
      telephony
      3504
      sao paulo
      SP
      2017-03-25 10:25:16
      5
      25
      3

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      18782
      3
      2018-07-16 18:20:34
      609b34a18fd0d61719d0b3190ec05231
      21240
      rio de janeiro
      RJ
      200.00
      19.50
      92d5ae8e42c4599266f210bf0469ae9b
      439.0
      ...
      11.0
      13.0
      watches_gifts
      14050
      ribeirao preto
      SP
      2018-07-16 18:20:34
      0
      16
      7

      3776
      1
      2018-05-25 07:06:30
      dc275c5e585b7c3a3cddef527e34fc19
      26540
      nilopolis
      RJ
      18.90
      7.55
      15de022edf1005363381e66bed514528
      100.0
      ...
      3.0
      23.0
      furniture_decor
      20270
      rio de janeiro
      RJ
      2018-05-25 07:06:30
      4
      25
      5

67176 rows × 21 columns

## 기존 피쳐 결합하여 새로운 피쳐 생성 (컬럼1 + 컬럼2 = 뉴피쳐)

```python
def change_var_type(f_df):
    df = f_df.copy()
    df['customer_zip_code_prefix'] = df['customer_zip_code_prefix'].astype(str)
    df['seller_zip_code_prefix'] = df['seller_zip_code_prefix'].astype(str)
    return df

def comnbine_columns(f_df,src_col1, src_col2,new_col):
    df = f_df.copy()
    df[new_col] = df[str(src_col1)] + '_' + df[str(src_col2)]
    print("df shape: ", df.shape)
    return df

f_df = change_var_type(f_df)
```

### custoemr_state + seller_state

```python
f_df = comnbine_columns(f_df,src_col1='customer_state', src_col2='seller_state',new_col='customer_seller_state')
```

    df shape:  (67176, 22)

### custoemr_city + seller_city

```python
f_df = comnbine_columns(f_df,src_col1='customer_city', src_col2='seller_city',new_col='customer_seller_city')
```

    df shape:  (67176, 23)

### custoemr_zip + seller_zip

```python
f_df = comnbine_columns(f_df,src_col1='customer_zip_code_prefix',
                        src_col2='seller_zip_code_prefix',new_col='customer_seller_zip_code_prefix')
```

    df shape:  (67176, 24)

```python
f_df
```

      classes
      order_approved_at
      customer_id
      customer_zip_code_prefix
      customer_city
      customer_state
      price
      freight_value
      product_id
      product_weight_g
      ...
      seller_zip_code_prefix
      seller_city
      seller_state
      order_date
      order_weekday
      order_day
      order_month
      customer_seller_state
      customer_seller_city
      customer_seller_zip_code_prefix

      37413
      3
      2018-07-14 13:04:07
      4ee61c3905a5c398d44b089108961bb3
      28950
      armacao dos buzios
      RJ
      105.00
      27.04
      2f13d1dc8b4e1d9d8027be50339546a9
      2650.0
      ...
      3204
      sao paulo
      SP
      2018-07-14 13:04:07
      5
      14
      7
      RJ_SP
      armacao dos buzios_sao paulo
      28950_3204

      54762
      3
      2017-03-25 10:25:16
      959292edcade77d6b60dc8f49f01cd71
      37880
      cabo verde
      MG
      23.99
      14.52
      b000447e24e31a4d7e628ca4d0622131
      250.0
      ...
      3504
      sao paulo
      SP
      2017-03-25 10:25:16
      5
      25
      3
      MG_SP
      cabo verde_sao paulo
      37880_3504

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      18782
      3
      2018-07-16 18:20:34
      609b34a18fd0d61719d0b3190ec05231
      21240
      rio de janeiro
      RJ
      200.00
      19.50
      92d5ae8e42c4599266f210bf0469ae9b
      439.0
      ...
      14050
      ribeirao preto
      SP
      2018-07-16 18:20:34
      0
      16
      7
      RJ_SP
      rio de janeiro_ribeirao preto
      21240_14050

      3776
      1
      2018-05-25 07:06:30
      dc275c5e585b7c3a3cddef527e34fc19
      26540
      nilopolis
      RJ
      18.90
      7.55
      15de022edf1005363381e66bed514528
      100.0
      ...
      20270
      rio de janeiro
      RJ
      2018-05-25 07:06:30
      4
      25
      5
      RJ_RJ
      nilopolis_rio de janeiro
      26540_20270

67176 rows × 24 columns

## product volume 컬럼 생성 (가로 * 세로 * 높이 의 계산값)

```python
def add_product_volume(raw_df):
    df = raw_df.copy()
    df['product_volume'] = df.product_length_cm * df.product_width_cm * df.product_height_cm
    return df

f_df = add_product_volume(f_df)
```

```python
f_df.columns
```

    Index(['classes', 'order_approved_at', 'customer_id',
           'customer_zip_code_prefix', 'customer_city', 'customer_state', 'price',
           'freight_value', 'product_id', 'product_weight_g', 'product_length_cm',
           'product_height_cm', 'product_width_cm',
           'product_category_name_english', 'seller_zip_code_prefix',
           'seller_city', 'seller_state', 'order_date', 'order_weekday',
           'order_day', 'order_month', 'customer_seller_state',
           'customer_seller_city', 'customer_seller_zip_code_prefix',
           'product_volume'],
          dtype='object')

## Train, Test 데이터 셋 분리

```python

def split_data_2(raw_df, sort_col='order_approved_at',val_ratio=0.3):
    '''
    train, test 데이터 분리
    '''
    df = raw_df.copy()
    val_ratio = 1 - val_ratio # 1 - 0.3  = 0.7

    df = df.sort_values(by= sort_col) # 시간 순으로 정렬
    # One-Hot-Encoding
    data1,data2, = np.split(df,
                     [int(val_ratio * len(df))])   # Randomly sort the data then split out first 70%, second 20%, and last 10%

    print(f"data1, data2 shape: {data1.shape},{data2.shape}")

    return data1, data2

train_df, test_df = split_data_2(f_df, val_ratio=0.2)

```

    data1, data2 shape: (53740, 25),(13436, 25)

## Target Encoding 관련 피쳐 생성
- Product_ID 별 Classes의 평균, 갯수 (te_pdid_mean, te_pdid_count)
- Target Error (classes - te_pdid_mean)

## Target Encoding with Smoothing
아래 비디오 및 코드 참조 함
- Feature Engineering
    - RecSys 2020 Tutorial: Feature Engineering for Recommender Systems
        - https://www.youtube.com/watch?v=uROvhp7cj6Q
    - Git Repo
        - https://github.com/rapidsai/deeplearning/tree/main/RecSys2020Tutorial

\begin{equation} \label{eq:te}
TE_{target}([Categories]) = \frac{count([Categories]) * mean_{target}([Categories]) + w_{smoothing} * mean_{target}(global)}{count([Categories]) + w_{smoothing}}
\end{equation}

```python

def create_target_encoding(cat, raw_df):
    '''
    te_mean, te_count 피쳐 생성
    '''
    df = raw_df.copy()
    te = df.groupby(cat).classes.agg(['mean','count']).reset_index()
    te_mean_col = 'te_' + cat + '_mean'
    te_count_col = 'te_' + cat + '_count'

    cat = [cat]
    te.columns = cat + [te_mean_col,te_count_col]
    te_df = df.merge(te, on=cat, how='left')

    return te_df

w = 20 # global 평균에 더 높은 가중치를 주는 값
def create_target_encoding_smoothe(cat, raw_df, w):
    '''
    te_mean, te_count를 기반으로 te_mean_smoothed 생성
    '''
    df = raw_df.copy()
    te_mean_col = 'te_' + cat + '_mean'
    te_count_col = 'te_' + cat + '_count'
    te_target_col = 'te_' + cat + '_mean_smoothed'

    classes_mean_global = df.classes.mean()
    # print(" classes_mean_global: ", classes_mean_global)
    df[te_target_col] = (df[te_mean_col] * df[te_count_col]) + (classes_mean_global * w) / (df[te_count_col] + w)
    df.drop([te_mean_col,te_count_col],axis=1, inplace=True)

    return df

def add_te_on_test(raw_train_df, raw_test_df, join_col, te_col_name):
    '''
    train의 te_mean_smoothed를 기반으로 test데이터에도 te_mean_smoothed를 생성함.
    만일 train에 해당 값이 없으면, 전체 평균값으로 대체 함.
    '''
    train_df = raw_train_df[[join_col,te_col_name]]
    test_df = raw_test_df.copy()

    # global_mean = train_df[te_col_name].mean() # 전체 레코드의 평균
    avg_train = train_df.groupby(join_col)[te_col_name].mean() # join_col 의 평균값을 구함
    avg_train_df = pd.DataFrame(avg_train).reset_index()
    # display(avg_train_df)

    target_df = pd.merge(
        test_df,
        avg_train_df,
        on = join_col,
        how = 'left'
    )

    global_mean = avg_train_df[te_col_name].mean() # 전체 레코드의 평균

    # print("global mean: ", global_mean)
    # train에 있고 test에 없으면 train의 평균값을 채움
    target_df[te_col_name].fillna(global_mean, inplace=True)

    print("targe_df and  test shape: ", target_df.shape, test_df.shape)
    assert(test_df.shape[0] == target_df.shape[0])

    return target_df

```

```python
def add_noise(series, noise_level):
    return series * (1 + noise_level * np.random.randn(len(series)))

def target_encode(trn_series=None,
                  tst_series=None,
                  target=None,
                  min_samples_leaf=1,
                  smoothing=1,
                  noise_level=0):
    """
    Smoothing is computed like in the following paper by Daniele Micci-Barreca
    https://kaggle2.blob.core.windows.net/forum-message-attachments/225952/7441/high%20cardinality%20categoricals.pdf
    trn_series : training categorical feature as a pd.Series
    tst_series : test categorical feature as a pd.Series
    target : target data as a pd.Series
    min_samples_leaf (int) : minimum samples to take category average into account
    smoothing (int) : smoothing effect to balance categorical average vs prior
    """
    assert len(trn_series) == len(target)
    assert trn_series.name == tst_series.name
    temp = pd.concat([trn_series, target], axis=1)
    # Compute target mean
    averages = temp.groupby(by=trn_series.name)[target.name].agg(["mean", "count"])
#    display(averages)
    # Compute smoothing
    smoothing = 1 / (1 + np.exp(-(averages["count"] - min_samples_leaf) / smoothing))
    # display(smoothing)
    # Apply average function to all target data
    prior = target.mean()
    # The bigger the count the less full_avg is taken into account
    averages[target.name] = prior * (1 - smoothing) + averages["mean"] * smoothing
    averages.drop(["mean", "count"], axis=1, inplace=True)
    display(averages)
    # Apply averages to trn and tst series
    ft_trn_series = pd.merge(
        trn_series.to_frame(trn_series.name),
        averages.reset_index().rename(columns={'index': target.name, target.name: 'average'}),
        on=trn_series.name,
        how='left')['average'].rename(trn_series.name + '_mean').fillna(prior)
    # pd.merge does not keep the index so restore it
    ft_trn_series.index = trn_series.index
    ft_tst_series = pd.merge(
        tst_series.to_frame(tst_series.name),
        averages.reset_index().rename(columns={'index': target.name, target.name: 'average'}),
        on=tst_series.name,
        how='left')['average'].rename(trn_series.name + '_mean').fillna(prior)
    # pd.merge does not keep the index so restore it
    ft_tst_series.index = tst_series.index
    return add_noise(ft_trn_series, noise_level), add_noise(ft_tst_series, noise_level)
```

## Target Encoding 실행

```python
def add_new_te(raw_train, raw_test):
    train_df = raw_train.copy()
    test_df = raw_test.copy()

    cat = 'product_id'
    trn, sub = target_encode(train_df[cat],
                             test_df[cat],
                             target=train_df.classes,
                             min_samples_leaf=100,
                             smoothing=10,
                             noise_level=0.01)
    te_col_name = 'te_' + cat + '_mean_smoothed'
    train_df[te_col_name] = trn
    test_df[te_col_name] = sub

    return train_df, test_df

train_df, test_df = add_new_te(train_df, test_df)

```

      classes

      product_id

      00088930e925c41fd95ebfe695fd2655
      2.178716

      000d9be29b5207b54e86aa1b1ac54872
      2.178666

      ...
      ...

      fff9553ac224cec9d15d49f5a263411f
      2.178716

      fffdb2d0ec8d6a61f0a0a0db3f25b441
      2.178610

20342 rows × 1 columns

```python
display(train_df.head(2))
display(test_df.head(2))
```

      classes
      order_approved_at
      customer_id
      customer_zip_code_prefix
      customer_city
      customer_state
      price
      freight_value
      product_id
      product_weight_g
      ...
      seller_state
      order_date
      order_weekday
      order_day
      order_month
      customer_seller_state
      customer_seller_city
      customer_seller_zip_code_prefix
      product_volume
      te_product_id_mean_smoothed

      2605
      3
      2016-10-04 10:19:23
      7812fcebfc5e8065d31e1bb5f0017dae
      12030
      taubate
      SP
      29.99
      10.96
      e2a1d45a73dc7f5a7f9236b043431b89
      9000.0
      ...
      SP
      2016-10-04 10:19:23
      1
      4
      10
      SP_SP
      taubate_limeira
      12030_13481
      2640.0
      2.166231

      41063
      2
      2016-10-04 13:46:31
      aadd27185177fc7ac9b364898ac09343
      78075
      cuiaba
      MT
      23.90
      26.82
      43bb8825dd6838251606e5e4130cfff4
      1500.0
      ...
      SP
      2016-10-04 13:46:31
      1
      4
      10
      MT_SP
      cuiaba_bauru
      78075_17051
      14625.0
      2.180797

2 rows × 26 columns

      classes
      order_approved_at
      customer_id
      customer_zip_code_prefix
      customer_city
      customer_state
      price
      freight_value
      product_id
      product_weight_g
      ...
      seller_state
      order_date
      order_weekday
      order_day
      order_month
      customer_seller_state
      customer_seller_city
      customer_seller_zip_code_prefix
      product_volume
      te_product_id_mean_smoothed

      4927
      3
      2018-06-19 03:36:38
      956346db615f7adb9ea991c5e4648c55
      89219
      joinville
      SC
      105.00
      23.89
      a62e25e09e05e6faf31d90c6ec1aa3d1
      1000.0
      ...
      RJ
      2018-06-19 03:36:38
      1
      19
      6
      SC_RJ
      joinville_rio de janeiro
      89219_21840
      7632.0
      2.399600

      6617
      0
      2018-06-19 03:36:39
      a3777228aa1a73a3bf9a3127609cf68a
      9950
      diadema
      SP
      99.97
      15.80
      ed1e39b938c6cc6867d8f5fd408ce319
      650.0
      ...
      MG
      2018-06-19 03:36:39
      1
      19
      6
      SP_MG
      diadema_betim
      9950_32677
      9600.0
      2.181332

2 rows × 26 columns

```python
def create_fe_target_encoding(raw_train, raw_test, cat_cols):
    train_df = raw_train.copy()
    test_df = raw_test.copy()
    for col in cat_cols:
        print("target col: ", col)
        train_df = create_target_encoding(col, train_df)
        train_df = create_target_encoding_smoothe(col, train_df, w)
        te_col_name = 'te_' + col + '_mean_smoothed'
        test_df = add_te_on_test(train_df, test_df, join_col = col, te_col_name= te_col_name)

    return train_df, test_df

cat_cols = ['product_category_name_english',
            'seller_state','seller_city','seller_zip_code_prefix',
            'customer_seller_city','customer_seller_state','customer_seller_zip_code_prefix']

# cat_cols = ['product_id','product_category_name_english',
#             'seller_state','seller_city','seller_zip_code_prefix',
#             'customer_seller_city','customer_seller_state','customer_seller_zip_code_prefix']

train2_df, test2_df = create_fe_target_encoding(train_df, test_df, cat_cols)

```

    target col:  product_category_name_english
    targe_df and  test shape:  (13436, 27) (13436, 26)
    target col:  seller_state
    targe_df and  test shape:  (13436, 28) (13436, 27)
    target col:  seller_city
    targe_df and  test shape:  (13436, 29) (13436, 28)
    target col:  seller_zip_code_prefix
    targe_df and  test shape:  (13436, 30) (13436, 29)
    target col:  customer_seller_city
    targe_df and  test shape:  (13436, 31) (13436, 30)
    target col:  customer_seller_state
    targe_df and  test shape:  (13436, 32) (13436, 31)
    target col:  customer_seller_zip_code_prefix
    targe_df and  test shape:  (13436, 33) (13436, 32)

```python
train2_df.head(2)
```

      classes
      order_approved_at
      customer_id
      customer_zip_code_prefix
      customer_city
      customer_state
      price
      freight_value
      product_id
      product_weight_g
      ...
      customer_seller_zip_code_prefix
      product_volume
      te_product_id_mean_smoothed
      te_product_category_name_english_mean_smoothed
      te_seller_state_mean_smoothed
      te_seller_city_mean_smoothed
      te_seller_zip_code_prefix_mean_smoothed
      te_customer_seller_city_mean_smoothed
      te_customer_seller_state_mean_smoothed
      te_customer_seller_zip_code_prefix_mean_smoothed

      0
      3
      2016-10-04 10:19:23
      7812fcebfc5e8065d31e1bb5f0017dae
      12030
      taubate
      SP
      29.99
      10.96
      e2a1d45a73dc7f5a7f9236b043431b89
      9000.0
      ...
      12030_13481
      2640.0
      2.166231
      313.325175
      80621.001131
      704.106277
      289.234266
      5.980614
      35917.001966
      5.074929

      1
      2
      2016-10-04 13:46:31
      aadd27185177fc7ac9b364898ac09343
      78075
      cuiaba
      MT
      23.90
      26.82
      43bb8825dd6838251606e5e4130cfff4
      1500.0
      ...
      78075_17051
      14625.0
      2.180797
      7805.012365
      80621.001131
      204.396123
      40.117269
      4.074929
      614.204570
      4.074929

2 rows × 33 columns

```python
test2_df
```

      classes
      order_approved_at
      customer_id
      customer_zip_code_prefix
      customer_city
      customer_state
      price
      freight_value
      product_id
      product_weight_g
      ...
      customer_seller_zip_code_prefix
      product_volume
      te_product_id_mean_smoothed
      te_product_category_name_english_mean_smoothed
      te_seller_state_mean_smoothed
      te_seller_city_mean_smoothed
      te_seller_zip_code_prefix_mean_smoothed
      te_customer_seller_city_mean_smoothed
      te_customer_seller_state_mean_smoothed
      te_customer_seller_zip_code_prefix_mean_smoothed

      0
      3
      2018-06-19 03:36:38
      956346db615f7adb9ea991c5e4648c55
      89219
      joinville
      SC
      105.00
      23.89
      a62e25e09e05e6faf31d90c6ec1aa3d1
      1000.0
      ...
      89219_21840
      7632.0
      2.399600
      6506.014877
      4961.018757
      2484.036771
      365.281119
      17.815563
      175.544669
      4.34846

      1
      0
      2018-06-19 03:36:39
      a3777228aa1a73a3bf9a3127609cf68a
      9950
      diadema
      SP
      99.97
      15.80
      ed1e39b938c6cc6867d8f5fd408ce319
      650.0
      ...
      9950_32677
      9600.0
      2.181332
      6718.013196
      10105.010047
      391.201729
      116.495153
      5.074929
      3818.025616
      4.34846

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      13434
      0
      2018-08-29 15:05:22
      496630b6740bcca28fce9ba50d8a26ef
      9541
      sao caetano do sul
      SP
      24.90
      8.33
      c7f27c5bef2338541c772b5776403e6a
      450.0
      ...
      9541_1320
      4788.0
      2.182778
      32.177662
      80621.001131
      27766.003039
      68.310888
      75.537944
      35917.001966
      4.34846

      13435
      0
      2018-08-29 15:10:26
      898b7fee99c4e42170ab69ba59be0a8b
      13483
      limeira
      SP
      84.99
      8.76
      d04857e7b4b708ee8b8b9921163edba3
      450.0
      ...
      13483_4102
      1280.0
      2.189203
      8250.012020
      80621.001131
      27766.003039
      293.203614
      41.854382
      35917.001966
      4.34846

13436 rows × 33 columns

```python
print(train2_df.shape)
print(test2_df.shape)
```

    (53740, 33)
    (13436, 33)

## Category 레이블 Encoding

```python
# from sklearn import preprocessing
from sklearn.preprocessing import LabelEncoder
class LabelEncoderExt(object):
    '''
    Source:
    # https://stackoverflow.com/questions/21057621/sklearn-labelencoder-with-never-seen-before-values
    '''
    def __init__(self):
        """
        It differs from LabelEncoder by handling new classes and providing a value for it [Unknown]
        Unknown will be added in fit and transform will take care of new item. It gives unknown class id
        """
        self.label_encoder = LabelEncoder()
        # self.classes_ = self.label_encoder.classes_

    def fit(self, data_list):
        """
        This will fit the encoder for all the unique values and introduce unknown value
        :param data_list: A list of string
        :return: self
        """
        self.label_encoder = self.label_encoder.fit(list(data_list) + ['Unknown'])
        self.classes_ = self.label_encoder.classes_

        return self

    def transform(self, data_list):
        """
        This will transform the data_list to id list where the new values get assigned to Unknown class
        :param data_list:
        :return:
        """
        new_data_list = list(data_list)
        for unique_item in np.unique(data_list):
            if unique_item not in self.label_encoder.classes_:
                new_data_list = ['Unknown' if x==unique_item else x for x in new_data_list]

        return self.label_encoder.transform(new_data_list)

def make_test_label_encoding(raw_train_df, raw_test_df,cols):
    train_df = raw_train_df.copy()
    test_df = raw_test_df.copy()

    for lb_col in cols:
        print(lb_col)
        le = LabelEncoderExt()
        le = le.fit(train_df[lb_col])

        train_en = le.transform(train_df[lb_col])
        test_en = le.transform(test_df[lb_col])
        lb_col_name = 'lb_' + lb_col
        print("new col name: ", lb_col_name)
        train_df[lb_col_name] = train_en
        test_df[lb_col_name] = test_en

    return train_df, test_df

```

### Category 변수의 레이블 인코딩 실행

```python
label_cols = ['customer_city','customer_state','customer_zip_code_prefix']
train2_lb, test2_lb = make_test_label_encoding(train2_df, test2_df, label_cols)
```

    customer_city
    new col name:  lb_customer_city
    customer_state
    new col name:  lb_customer_state
    customer_zip_code_prefix
    new col name:  lb_customer_zip_code_prefix

```python
pd.options.display.max_rows = 10
show_rows = 5
print(train2_lb.customer_state.value_counts()[0:show_rows])
# print(train2_lb[train2_lb.lb_customer_city == 185])
print(test2_lb.customer_state.value_counts()[0:show_rows])
```

    SP    28232
    MG     6763
    RJ     6034
    PR     2912
    RS     2385
    Name: customer_state, dtype: int64
    SP    6642
    MG    1541
    RJ    1491
    PR     715
    RS     663
    Name: customer_state, dtype: int64

## 레이블 Encoding 안하고 바로 사용(AutoGluon 용)

```python
# no_encoding_cate = tes_df
```

## 최종 사용할 컬럼 지정
### XGBoost, CatBoost 알고리즘 용

```python
def filter_df(raw_df, cols):
    df = raw_df.copy()
    df = df[cols]
    return df

cols = ['classes',
        'lb_customer_city',
        'lb_customer_state',
        'lb_customer_zip_code_prefix',
        'price', 'freight_value',
        'product_weight_g',
        'product_volume',
        'order_weekday',
        'order_day', 'order_month',
        'te_product_id_mean_smoothed',
        'te_product_category_name_english_mean_smoothed',
        'te_seller_state_mean_smoothed', 'te_seller_city_mean_smoothed',
        'te_seller_zip_code_prefix_mean_smoothed',
        'te_customer_seller_city_mean_smoothed',
        'te_customer_seller_state_mean_smoothed',
        'te_customer_seller_zip_code_prefix_mean_smoothed',
       ]

encode_train = filter_df(train2_lb, cols)
encode_test = filter_df(test2_lb, cols)

# no_encode_train = filter_df(encode_train, cols)
# no_encode_val = filter_df(encode_val, cols)
# no_encode_test = filter_df(encode_test, cols)

```

## 피쳐 변환한  AutoGluon 용

```python
cols = ['classes',
        'customer_city',
        'customer_state',
        'customer_zip_code_prefix',
        'product_category_name_english',
        'price', 'freight_value',
        'product_weight_g',
        'product_volume',
        'order_weekday',
        'order_day', 'order_month',
        'te_product_id_mean_smoothed',
        'te_product_category_name_english_mean_smoothed',
        'te_seller_state_mean_smoothed', 'te_seller_city_mean_smoothed',
        'te_seller_zip_code_prefix_mean_smoothed',
        'te_customer_seller_city_mean_smoothed',
        'te_customer_seller_state_mean_smoothed',
        'te_customer_seller_zip_code_prefix_mean_smoothed',
       ]

auto_train = filter_df(train2_lb, cols)
auto_test = filter_df(test2_lb, cols)

```

## 펴쳐 변환 없이 AutoGluon 용

```python
train_df.columns
```

    Index(['classes', 'order_approved_at', 'customer_id',
           'customer_zip_code_prefix', 'customer_city', 'customer_state', 'price',
           'freight_value', 'product_id', 'product_weight_g', 'product_length_cm',
           'product_height_cm', 'product_width_cm',
           'product_category_name_english', 'seller_zip_code_prefix',
           'seller_city', 'seller_state', 'order_date', 'order_weekday',
           'order_day', 'order_month', 'customer_seller_state',
           'customer_seller_city', 'customer_seller_zip_code_prefix',
           'product_volume', 'te_product_id_mean_smoothed'],
          dtype='object')

```python
cols = ['classes',
       'customer_zip_code_prefix', 'customer_city', 'customer_state', 'price',
       'freight_value', 'product_weight_g',
       'product_category_name_english', 'seller_zip_code_prefix',
       'seller_city', 'seller_state', 'order_date', 'order_weekday',
       'order_day', 'order_month', 'customer_seller_state',
       'customer_seller_city', 'customer_seller_zip_code_prefix',
       'product_volume']

no_auto_train = filter_df(train_df, cols)
no_auto_test = filter_df(test_df, cols)

```

## 로컬에 데이터 저장

```python
import os

def save_local(train_data, test_data, preproc_folder):
    train_df = pd.concat([train_data['classes'], train_data.drop(['classes'], axis=1)], axis=1)
    train_file_name = os.path.join(preproc_folder, 'train.csv')
    train_df.to_csv(train_file_name, index=False)
    print(f'{train_file_name} is saved')

    test_df = pd.concat([test_data['classes'], test_data.drop(['classes'], axis=1)], axis=1)
    test_file_name = os.path.join(preproc_folder, 'test.csv')
    test_df.to_csv(test_file_name, index=False)
    print(f'{test_file_name} is saved')

    return train_file_name, test_file_name

preproc_folder = 'preproc_data/feature_engineering'
os.makedirs(preproc_folder, exist_ok=True)
pre_train_file, pre_test_file = save_local(encode_train, encode_test, preproc_folder)

preproc_folder = 'preproc_data/auto_feature_engineering'
os.makedirs(preproc_folder, exist_ok=True)
auto_train_file,auto_test_file  = save_local(auto_train, auto_test, preproc_folder)

preproc_folder = 'preproc_data/auto_no_fe'
os.makedirs(preproc_folder, exist_ok=True)
no_auto_train_file,no_auto_test_file  = save_local(no_auto_train, no_auto_test, preproc_folder)
```

    preproc_data/feature_engineering/train.csv is saved
    preproc_data/feature_engineering/test.csv is saved
    preproc_data/auto_feature_engineering/train.csv is saved
    preproc_data/auto_feature_engineering/test.csv is saved
    preproc_data/auto_no_fe/train.csv is saved
    preproc_data/auto_no_fe/test.csv is saved

```python
%store pre_train_file
%store pre_test_file

%store auto_train_file
%store auto_test_file

%store no_auto_train_file
%store no_auto_test_file
```

    Stored 'pre_train_file' (str)
    Stored 'pre_test_file' (str)
    Stored 'auto_train_file' (str)
    Stored 'auto_test_file' (str)
    Stored 'no_auto_train_file' (str)
    Stored 'no_auto_test_file' (str)

```python

```

```python

```

```python

```


Unit 10

Case Study 4: Object Recognition with Deep Learning

IST 718 – Big Data Analytics

Daniel E. Acuna

http://acuna.io

# Object recognition problem
- A bit of neuroscience.
- From logistic regression to neural networks.
- The perceptron.
- Multi-layer perceptron.
- Training neural networks.
- Demo with multi-layer perceptron.

# The object recognition problem
- Detect an object present in an image.

Caltech101 dataset (year 2003)
101 categories of objects, each category has 40 to 800 images

# A high dimensional problem
- Each image is 300 x 200 x 3 pixels (width x height x channels.)
  - 180,000 "raw" features.

- Relationship between raw features and classes is highly non-linear.
  - Spatial features that could appear anywhere on the image.
  - Local features (e.g., shapes) vs global features (e.g., contrast.)

# A high dimensional problem (2)

Credit: https://www.st-andrews.ac.uk/~www_pa/Scots_Guide/info/signals/pixels/

# A problem with logistic regression
- We are trying to learn:

$$p(y \mid X) = \frac{1}{1 + \exp(-(\theta_0 + \sum_{j>0}{x_j\theta_j}))}$$

- So only a linear relationship between pixels $x$ and class $y$.

# Classic data science
- Features fed into the algorithms are meaningful and provided by experts.

- Machine learning algorithms only provide 1 or 2 step transformations.

- Complexity of problems is relatively small.

# A bit of neuroscience
- Deep learning takes loose inspiration from how the brain works.

- At an architectural level, the brain works by combining specialized neurons in hierarchies.

- E.g., the visual system is organized into several areas (around 8.)

# A bit of neuroscience (2)
- The brain has relatively "simple" processing units: neurons.

Credit: http://www.intechopen.com/source/html/39067/media/image1.png

$$p(y \mid X) = \frac{1}{1 + \exp(-(\theta_0 + \sum_{j>0}{x_j\theta_j}))}$$
**A model for the probability of an action potential!**

# Of course reality is a lot more complicated

# Of course reality is a lot more complicated

# The general idea of artificial neural networks
- Simple processing units with linear or non-linear functions $f$.

# The general idea of artificial neural networks (2)
- Multiple units belong to layers and those layers are interconnected to other layers.

# A highly non-linear function
- The output now depends on nested functions:

$$y = \underbrace{f_3\left(\;\underbrace{f_2\left(\;\underbrace{f_1(x)}_{\text{first layer}} + b_0\right)}_{\text{second layer}} + b_1\right) + b_2}_{\text{third layer (nested non-linearities)}}$$

# Advantages and disadvantages of ANN
- Pros:
  - Can fit complex non-linear relationships.
  - Easy to try different types of layers.
  - Inspired by a very effective machine (the brain!)

- Cons:
  - Many parameters.
  - Easy to overfit.
  - Needs lots of training data.
  - Takes a long time to fit.
  - The use of brain research is really minimal.

# How to train a neural network?
- We define a loss function and try to find best value:

$$\arg\min_{\Theta} {l(\;f_\Theta(X),y)}$$

- For example:

$$l(\;f_\Theta(X),y) = \sum(\;f_\Theta(X_i) - y_i)^2$$

where $f_\Theta(X)$ is the function represented by the entire network.

# Optimizing the loss function
- There is no close solution to the loss function:

$$\arg\min_{\Theta} {l(\;f_\Theta(X),y)}$$

- In all but the trivial cases, the loss function is *non-convex* (hard to optimize.)

- A very effective approach to perform stochastic gradient descent.

# Gradient descent
- Gradient descent is a simple approach to iteratively minimize a loss function.

- Intuitively, we want to follow the *negative* of the gradient to find the minimum of a loss function:

$$\Theta_{t+1} = \Theta_{t} - \eta \nabla l(\;f_\Theta(X),y)$$

# Gradient descent: example

# Gradient descent: example (2)

# Gradient descent: example (3)

# Gradient descent: example (4)

# Gradient descent: example (5)

# Gradient descent: problems
- It finds the global optimum when function is convex (e.g., a parabola) and the learning rate is appropriate.
- But:

# Stochastic gradient descent
- If we look at a fraction of the training data, we only observe a (noisy) sample of the loss function.

# Stochastic gradient descent (2)
- If we look at a fraction of the training data, we only observe a (noisy) sample of the loss function.

# Stochastic gradient descent (3)
- If we look at a fraction of the training data, we only observe a (noisy) sample of the loss function.

- Stochastic gradient descent is good for big data (we only need pieces at a time.)

- Stochastic gradient descent can escape local minima.

- As long as we can compute gradients, we can use function we want!

# Worked-out example
- A simple neural network with one neuron and one layer.

- More assumptions sigmoid activation $f$ and $a = y$.

# Worked-out example (2)
- A simple neural network with one neuron and one layer.

- More assumptions sigmoid activation $f$ and $a = y$.

**Similar to Logistic Regression!**

# Worked-out example (3)
- Using quadratic loss function, let’s work out the stochastic gradient descent update rule:

$$\Theta_{t+1} = \Theta_{t} - \eta \nabla l(\;f_\Theta(X),y)$$

$$l(\;f_\Theta(X),y) = (\;f(z) - y)^2 = \left(\sigma\left(\sum{x_j\theta_j} - y\right)\right)^2$$

$$\nabla{l(\;f_\Theta(X),y)} = \left[\frac{dl}{d\theta_0}, \frac{dl}{d\theta_1}, \ldots, \frac{dl}{d\theta_m}\right]$$

# Worked-out example (4)
- Let’s pick one the gradients:

$$\begin{align}
\nabla{l(\;f_\Theta(X),y)} &= \left[\frac{dl}{d\theta_0}, \frac{dl}{d\theta_1}, \ldots, \frac{dl}{d\theta_m}\right] \\
\frac{dl}{d\theta_j} &= \frac{d\left(\sigma\left(\sum{x_j\theta_j} - y\right)\right)^2}{d\theta_j}
\end{align}$$

- We are trying to compute the gradient of a nested function, what to do?

# Worked-out example (5)

<left></left>

- We can apply basic calculus.

- By representing each nested function on its own:

$$\frac{d\left(\sigma\left(\sum{x_j\theta_j} - y\right)\right)^2}{d\theta_j} = \frac{d}{d\theta_j} l(a(z(\theta_0, \cdots, \theta_m, x), y)$$

- Where $a$ is the activation, and $z$ is the summation.

- Now we can simply apply the chain rule!

# Worked-out example (6)
- We can apply basic calculus.
- By representing each nested function on its own:
$$\frac{d}{d\theta_j} l(a(z(\theta_0, \cdots, \theta_m, x), y) = \frac{dl}{da}\frac{da}{dz}\frac{dz}{d\theta_j}$$

$$
\frac{dl}{da} = 2(a - y) \qquad
\frac{da}{dz} = a(1-a) \qquad
\frac{dz}{d\theta_j} = x_j
$$

$$\begin{align}
\Theta_j^{\text{new}} &\leftarrow \Theta_j^{\text{old}} - \eta \;{2(a-y)} \;{a(1-a)} \;{x_j} \\
&\leftarrow \Theta_j^{\text{old}} - \eta \;{\left(\sigma\left(\sum{x_j\theta_j}\right) - y\right)} \;{\sigma\left(\sum{x_j\theta_j}\right)} \;{\left(1 - \sigma\left(\sum{x_j\theta_j}\right)\right)} {x_j}
\end{align}$$

# Backpropagation
- The previous example is the basis of stochastic gradient descent for several layers.
- **Backpropagation** is a general algorithm that exploits the structure of the chain rule when applied to layers.
- To understand backpropagation, we need to understand middle layers:

# Backpropagation (2)
- $\theta_{ij}^{L-2}$: connection between neuron $j$ in layer $L-2$ to neuron $i$ in layer $L-1$.

- Representing all connections from $L-2$ to $L-2$ as one matrix $\theta^{L-2}$, we can predict all activation of layer $L-1$ as:

$$a^{L-1} = f(\theta^{L-2} a^{L-2})$$

# Backpropagation (3)
- Then the gradient of a connection $\theta_{ij}^{L-2}$ with respect to the loss function will be:

$$\frac{dl}{d\theta_{ij}^{L-2}} = \frac{dz_i^{L-1}}{d\theta_{ij}{L-2}}\frac{da_i^{L-1}}{dz_i^{L-1}} \sum{\frac{dz^L}{da_i^{L-1}} \frac{da^L}{dz^L} \frac{dl}{da^L}}$$

# Backpropagation (4)
- Then the gradient of a connection $\theta_{jk}^{L-3}$ with respect to the loss function will be:

$$\frac{dl}{d\theta_{jk}^{L-3}} = \frac{dz_j^{L-2}}{d\theta_{jk}^{L-3}}\frac{da_j^{L-2}}{dz_j^{L-2}} \sum{\frac{dz_i^{L-1}}{da_j^{L-2}} \frac{da_i^{L-1}}{dz_i^{L-1}}} \sum{\frac{dz^L}{da_i^{L-1}} \frac{da^L}{dz^L} \frac{dl}{da^L}} $$

# Backpropagation (5)
- The same structure gets repeated during the chain rule:

$$\frac{dl}{d\theta_{ij}^{L-2}} = \frac{dz_i^{L-1}}{d\theta_{ij}{L-2}}\frac{da_i^{L-1}}{dz_i^{L-1}} \sum{\frac{dz^L}{da_i^{L-1}} \frac{da^L}{dz^L} \frac{dl}{da^L}}$$

$$\frac{dl}{d\theta_{jk}^{L-3}} = \frac{dz_j^{L-2}}{d\theta_{jk}^{L-3}}\frac{da_j^{L-2}}{dz_j^{L-2}} \sum{\frac{dz_i^{L-1}}{da_j^{L-2}} \frac{da_i^{L-1}}{dz_i^{L-1}}} \sum{\frac{dz^L}{da_i^{L-1}} \frac{da^L}{dz^L} \frac{dl}{da^L}}$$

- Therefore, we can *backpropagate* the gradient of the loss function.
- Notice also, that we reuse $a$ and $z$ multiple times, therefore we could precompute them.
- Consequently, we could forward propagate $a$ and $z$ and *backpropagate* the errors.

# Backpropagation algorithm

Forward propagation
1. From layer $1$ thought $L$
  1.1 Compute $a$ and $z$

Backpropagation
1. Define $\delta_i^L = a^L-y$ (gradient of loss function)
2. From layer $L–1$ to $1$
  2.1 Define $\delta_j^{L-1} = a_j^{L-1} (1 - a_j^{L-1}) \sum{\theta_{ij}^{L-1} \delta_i^L}$

The gradients will be computed as follows:
$$\frac{dl}{d\theta_{ij}^k} = a_j^k a_i^{k+1} (1 - a_i^{k+1}) \sum{a_i^{k+1} \delta_i^{k+1}} $$

# Demo

# Problems with classic methods
- The more layers, the more complex representations can be learned.

- However, more layers also implies longer chains.

- This produces numerical errors and small changes in learning.

- Groundbreaking paper in 2006 proposes to train deep architectures in a novel way.

# Deep learning
- Also, new developments in:
  - Training techniques (2006)
  - GPU computing (2008)
  - Big data.
  - New types of architectures (new neurons and layers.)

- This opened the door to train increasingly more complex models.

- Next class we will take a look at these advances

# Other architectures
- Multi-layer perceptron looses spatial location (specially in images.)

- Some features are independent of location.

# Convolutional neural networks
- Create translational-invariant spatial filters (share weights.)

http://cs231n.github.io/convolutional-networks/

# Convolutional neural networks (2)
- It can learn filters similar to what is observed in nature!

# Convolutional neural networks (3)
- Digits classification: classic MNIST problem.

http://cs.stanford.edu/people/karpathy/convnetjs/demo/mnist.html

# Dealing with temporal data
- Sometimes we want to capture temporal-invariance:
  - E.g., in chess, the state of the next move only depends on the current move.
  - Makes learning more efficient.
  - Allows to interpret results.

# Recurrent neural networks

# Recurrent neural networks (2)
- Naïve recurrent neural network training leads to numerical problems:
$$y = h(A(A(A(\ldots A(y_{t-T}))$$

- Why? How would the chain rule apply here?

- Other more advanced neural network allows to learn long-term dependencies easier.

# Recurrent neural networks (3)
- Handwriting generation:
  http://www.cs.toronto.edu/~graves/handwriting.html

- Generating Wikipedia articles:

# More complex temporal problems

Can you think of examples for each of this cases?

# More complex temporal problems (2)

# More complex temporal problems (3)

# More complex temporal problems (4)

# Reinforcement learning
- Given semi-supervised feedback, learn how to perform actions so as to maximize total reward.

- Classical framework to study behavior in humans and animals.

# Reinforcement learning (2)

# Reinforcement learning (3)

http://cs.stanford.edu/people/karpathy/convnetjs/demo/rldemo.html

# Inferring intentions

# WaveNet

https://deepmind.com/blog/wavenet-generative-model-raw-audio/

# WaveNet

https://deepmind.com/blog/wavenet-generative-model-raw-audio/

# WaveNet

https://deepmind.com/blog/wavenet-generative-model-raw-audio/

# Adversarial Neural Networks

# Adversarial Neural Networks (2)

# Adversarial Neural Networks (3)

# Adversarial Neural Networks (4)

# Take home messages
- Neural networks try to mimic how the brain works: hierarchical layers of non-linear functions.

- Neural networks take time to train and overfit badly.

- Neural networks need big data.

- Neural networks are hard or impossible to interpret.

- Backpropagation allows efficient computation of gradients.