# Data Analyst's Toolbox: R and Python

> Institute of Political Science, NSYSU, 2020-12-25

Kuo, Yao-Jen <yaojenkuo@datainpoint.com> from [DATAINPOINT](https://www.datainpoint.com)

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
```

## TL; DR

Source: <https://memes.tw/>

## About me

## Teaching practical data science online/offline, for individuals

- [如何成為資料分析師：從問題解決到行動方案，Hahow 好學校](https://hahow.in/cr/dajourney)
- Visualization and modern data science, Adjunct Instructor, National Taiwan University
- Programming and business analytics, Adjunct Instructor, National Taiwan Normal University
- Python for data analysis, Instructor, Chunghwa Telecom Academy
- Python for data science, Machine learning from scratch, Senior Instructor, CSIE Training Program, National Taiwan University

## Also for commercial banking clients

- 2020 DBS Training Program
- 2019 HNCB Training Program
- 2017 ESUN Training Program

## Writing books

- [新手村逃脫！初心者的 Python 機器學習攻略](https://www.books.com.tw/products/0010867390)
- [進擊的資料科學](https://www.books.com.tw/products/0010827812)
- [輕鬆學習 R 語言](https://www.books.com.tw/products/0010835361)

## Writing blogs

- [Medium](https://medium.com/@tonykuoyj)
- [Substack](https://datainpoint.substack.com/about)
- [方格子](https://vocus.cc/user/@yaojenkuo)

## Before being a instructor

- Working experience
    - Senior Data Analyst, Coupang Shanghai
    - Analytical Consultant, SAS Taiwan
    - Management Associate, Chinatrust Banking Corporation Taiwan
    - Research Assistant, McKinsey & Company Taiwan
- Education
    - MBA, National Taiwan University
    - BA, National Taiwan University

## Loves running with a marathon PR of 2:43:12 at 2019 Seoul Marathon

Source: <https://giphy.com>

## What is data analysis

## The definition

> We generate questions about a specific topic, we search for answers by exploring, transforming, and modelling data referring to our topic. And then use what we've learned to refine questions or generate new questions.

Source: [R for Data Science](https://r4ds.had.co.nz/)

## Why data analysis

> It is now an era of data-driven strategic thinking, and is probably never coming back.

## The three means of persuasion that an orator must rely on

- Ethos
- Pathos
- Logos

Source: [Aristotle, Rhetoric](https://en.wikipedia.org/wiki/Rhetoric)

## It is a lot easier to persuade via ethos or pathos, but it takes time

However, logos can be easily acquired once it is a fact and can be proven. Hence, data analysis is often the express way to logos.

## Modern data analysis can be illustrated as the flow of data

Source: [R for Data Science](https://r4ds.had.co.nz/)

## The funny definitions

Source: <https://twitter.com/cdixon/status/428914681911070720/photo/1>

Source: <https://www.warnerbros.com/tv/friends/>

Source: <https://www.warnerbros.com/tv/friends/>

## The serious definition

> Modern data analysis invovles applications and tools like importing, tidying, transformation, visualization, modeling, and communication. Surrounding all these is programming.

```python
def get_value_ratios(df, col_name):
    return df[col_name].value_counts() / df[col_name].value_counts().sum()
def get_checkbox_ratios(df, col_pattern):
    channel_names = []
    channel_checks = []
    for col_name in df.columns:
        if (col_pattern in col_name) and ('OTHER_TEXT' not in col_name):
            channel_name = df[col_name].value_counts().index
            channel_check = df[col_name].value_counts().values
            if channel_name.size != 0:
                channel_names.append(channel_name[0])
                channel_checks.append(channel_check[0])
    channel_counts = pd.Series(channel_checks, index=channel_names)
    channel_ratios = channel_counts / channel_counts.sum()
    channel_ratios_sorted = channel_ratios.sort_values(ascending=False)
    return channel_ratios_sorted
def plot_ans_38(ans_38_ser):
    fig = plt.figure()
    ax = plt.axes()
    ans_38_ser = ans_38_ser[::-1]
    ax.barh(ans_38_ser.index, ans_38_ser.values, color=['c', 'c', 'c', 'c', 'c', 'r'])
    ax.set_title("What is the primary tool that you use at work or school to analyze data?")
    ax.set_xticks([0, 0.1, 0.2, 0.3, 0.4, 0.5])
    ax.set_xticklabels(['0%', '10%', '20%', '30%', '40%', '50%'])
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.spines['bottom'].set_visible(False)
    plt.show()
def get_ax(ans_ser, plot_title, ax):
    ax = ax
    ans_ser = ans_ser[:10][::-1]
    ax.barh(ans_ser.index, ans_ser.values, color=['c', 'c', 'c', 'c', 'c', 'c', 'c', 'r', 'r', 'r'])
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.spines['bottom'].set_visible(False)
    ax.set_title(plot_title)
def plot_ans(ans_ser, question_str):
    fig, axes = plt.subplots()
    get_ax(ans_ser, question_str, axes)
    plt.show()
survey = pd.read_csv("https://kaggle-ml-ds-survey.s3-ap-northeast-1.amazonaws.com/kaggle-survey-2020/kaggle_survey_2020_responses.csv", skiprows=[1])
ans_7 = get_checkbox_ratios(survey, 'Q7')
ans_8 = get_value_ratios(survey, 'Q8')
ans_38 = get_value_ratios(survey, 'Q38')
```

## Use programming language to analyze data

Let's review a question from [2020 Kaggle ML & DS Survey](https://www.kaggle.com/c/kaggle-survey-2020):

> What is the primary tool that you use at work or school to analyze data?

```python
plot_ans_38(ans_38)
```

## It seems inevitable to write codes in modern data analysis

Source: <https://giphy.com/>

## Simply put, we can choose any programming language as long as it is capable of

- Importing data
- Tidying data
- Transforming data
- Visualizing data
- Modeling data
- Communicating data

## Well, actually a lot of programming languages are capable of doing these

- Python
- R
- Julia
- Scala
- Matlab
- SAS
- ...etc.

## How to choose among so many alternatives?

- The philosophy of "Eating a water mellon".
- The full support of scientific computing.
- Our objectivity.

## The philosophy of "Eating a water mellon"

Let's review another 2 questions from [2020 Kaggle ML & DS Survey](https://www.kaggle.com/c/kaggle-survey-2020):

- What programming languages do you use on a regular basis?
- What programming language would you recommend an aspiring data scientist to learn first?

```python
plot_ans(ans_7, "What programming languages do you use on a regular basis?")
```

```python
plot_ans(ans_8, "What programming language would you recommend \n an aspiring data scientist to learn first?")
```

## R and Python in Stack Overflow Trends

<https://insights.stackoverflow.com/trends?tags=python%2Cr%2Cjulia%2Cscala%2Cmatlab%2Csas>

## The full support of scientific computing

- Does the language support vectorization?
- Does the language support various data format?
- Does the language support visualization?

## Both R and Python support vectorization

- R uses built-in `vector` and `matrix`.
- Python uses a third-party `ndarray`.

## Both R and Python support various data format

- R uses
    - built-in named `list` to support key-value storage
    - built-in `data.frame` to support tabular data
- Python uses
    - built-in `dict` to support key-value storage
    - third-party `DataFrame` to support tabular data

## Both R and Python support visualization

- R uses
    - built-in basic plotting system to support static plotting
    - third-party `ggplot2` to support high-end static plotting
    - third-party `shiny` to support dynamic plotting
- Python uses
    - third-party `matplotlib` to support static plotting
    - third-party `seaborn` to support high-end static plotting
    - third-party `plotly` to support dynamic plotting

## Last but not least, it depends on our objectivity

- Specific or general-purposed?
- Functional or object-oriented?
- ...etc.

## We will generate our own objectivity once we start coding

Source: <https://giphy.com>

## Let's write some codes to analyze data

## Bringing up  a topic

> 大選開票看哪個里最準？ 「章魚里」神預測告訴你。每次到了選舉，總是會有幾個里開票與大選結果相似，因此被各界視為重點關注的開票區域。

Source: <https://www.cw.com.tw/article/5093012>

## We can generate some questions regarding this topic

- How to define 「章魚里」?
- Can we find out 「章魚里」 based on 2020 presidential data?
- Can we find the similarity of our own village?

## How to define 「章魚里」？

Basically, after a few literature search, you may find the definition of 「章魚里」 is quite ambigious. So we are using a much fancier metric: **cosine similarity**.

## What is cosine similarity

> Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space. It is defined to equal the cosine of the angle between them, which is also the same as the inner product of the same vectors normalized to both have length 1.

\begin{equation}
a = (a_1, a_2, a_3) \\
b = (b_1, b_2, b_3)
\end{equation}

\begin{align}
cos\theta &= \frac{\sum_i(a_i \times b_i)}{\sqrt{\sum_i a_i^2} \times \sqrt{\sum_i b_i^2}} \\
&= \frac{a \cdot b}{\parallel a \parallel \times \parallel b \parallel}
\end{align}

Source: <https://en.wikipedia.org/wiki/Cosine_similarity>

## Can we find out 「章魚里」 based on 2020 presidential data？

Definitely, we all have access to the [Central Election Commission](https://db.cec.gov.tw/).

## We've downloaded these spreadsheets and prepared an in-browser environment for you.

[](https://mybinder.org/v2/gh/yaojenkuo/talks/HEAD)

## We can access to a Python notebook, R notebook, or RStudio in browser, no strings attached.

## Besides a few kernels to execute, we also attached some data.

- Reading a CSV file.
- Reading a Excel spreadsheet.

```python
# reading data via Python's pandas library
csv_df = pd.read_csv('presidential_2020.csv')
excel_df = pd.read_excel('presidential-2020/總統-A05-4-候選人得票數一覽表-各投開票所(南投縣).xls', skiprows=[0, 1, 3, 4])
```

## Our CSV file is an integrated file after manipulations

```python
csv_df.head()
```

      county
      town
      village
      office
      number
      candidate
      votes

      0
      宜蘭縣
      宜蘭市
      民族里
      1
      1
      宋楚瑜/余湘
      37

      1
      宜蘭縣
      宜蘭市
      民族里
      2
      1
      宋楚瑜/余湘
      31

      2
      宜蘭縣
      宜蘭市
      建軍里
      3
      1
      宋楚瑜/余湘
      19

      3
      宜蘭縣
      宜蘭市
      建軍里
      4
      1
      宋楚瑜/余湘
      29

      4
      宜蘭縣
      宜蘭市
      泰山里
      5
      1
      宋楚瑜/余湘
      25

## Our Excel spreadsheets are the original files downloaded from [Central Election Commission](https://db.cec.gov.tw/)

```python
excel_df.head()
```

      Unnamed: 0
      Unnamed: 1
      Unnamed: 2
      (1)\n宋楚瑜\n余湘
      (2)\n韓國瑜\n張善政
      (3)\n蔡英文\n賴清德
      Unnamed: 6
      Unnamed: 7
      Unnamed: 8
      Unnamed: 9
      Unnamed: 10
      Unnamed: 11
      Unnamed: 12
      Unnamed: 13

      0
      總　計
      NaN
      NaN
      13,315
      133,791
      152,046
      299,152
      3,555
      302,707
      13
      302,720
      110,765
      413,485
      73.2087

      1
      南投市
      NaN
      NaN
      3,077
      26,690
      30,910
      60,677
      693
      61,370
      3
      61,373
      20,480
      81,853
      74.9759

      2
      NaN
      龍泉里
      1.0
      26
      241
      391
      658
      8
      666
      0
      666
      228
      894
      74.4966

      3
      NaN
      康壽里
      2.0
      30
      216
      266
      512
      4
      516
      0
      516
      128
      644
      80.1242

      4
      NaN
      康壽里
      3.0
      25
      239
      306
      570
      8
      578
      0
      578
      154
      732
      78.9617

## We can also try importing via the RStudio interface.

```r
library(readxl)

csv_df = read.csv('presidential_2020.csv')
excel_df = read_excel('presidential-2020/總統-A05-4-候選人得票數一覽表-各投開票所(南投縣).xls')
head(csv_df)
head(excel_df)
```

## We write codes to integrate these spreadsheets into a CSV file

```python
from presidential import Presidential

presidential = Presidential('presidential-2020')
presidential_df = presidential.adjust_presidential_df()
presidential_df.to_csv('presidential_2020.csv', index=False)
```

    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(宜蘭縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(彰化縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(金門縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(桃園市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(苗栗縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(臺南市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(雲林縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(南投縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(高雄市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(臺北市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(新北市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(花蓮縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(新竹市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(新竹縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(基隆市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(連江縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(嘉義縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(嘉義市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(屏東縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(澎湖縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(臺東縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(臺中市).xls

```python
presidential_df.head()
```

      county
      town
      village
      office
      number
      candidate
      votes

      0
      宜蘭縣
      宜蘭市
      民族里
      1
      1
      宋楚瑜/余湘
      37

      1
      宜蘭縣
      宜蘭市
      民族里
      2
      1
      宋楚瑜/余湘
      31

      2
      宜蘭縣
      宜蘭市
      建軍里
      3
      1
      宋楚瑜/余湘
      19

      3
      宜蘭縣
      宜蘭市
      建軍里
      4
      1
      宋楚瑜/余湘
      29

      4
      宜蘭縣
      宜蘭市
      泰山里
      5
      1
      宋楚瑜/余湘
      25

```python
presidential_df.tail()
```

      county
      town
      village
      office
      number
      candidate
      votes

      51673
      臺中市
      和平區
      梨山里
      1845
      3
      蔡英文/賴清德
      132

      51674
      臺中市
      和平區
      梨山里
      1846
      3
      蔡英文/賴清德
      107

      51675
      臺中市
      和平區
      梨山里
      1847
      3
      蔡英文/賴清德
      40

      51676
      臺中市
      和平區
      平等里
      1848
      3
      蔡英文/賴清德
      24

      51677
      臺中市
      和平區
      平等里
      1849
      3
      蔡英文/賴清德
      102

## Check if the summations are right with Python

```python
ttl_votes = presidential_df['votes'].sum()
ttl_votes_by_candidates = presidential_df.groupby('number')['votes'].sum()
ttl_votes_by_candidates
```

    number
    1     608590
    2    5522119
    3    8170231
    Name: votes, dtype: int64

## Check if the summations are right with R

```r
library(dplyr)

csv_df %>%
    group_by(number) %>%
    summarise(ttl_votes = sum(votes))
```

## National percentage is our target vector to be compared

```python
national_percentage = ttl_votes_by_candidates / ttl_votes
national_percentage
```

    number
    1    0.042556
    2    0.386137
    3    0.571307
    Name: votes, dtype: float64

## Total votes for each village

```python
combined_key = presidential_df['county'].str.cat(presidential_df['town']).str.cat(presidential_df['village'])
presidential_df = presidential_df.assign(combined_key=combined_key)
ttl_votes_by_combined_key = presidential_df.groupby(['combined_key'])['votes'].sum()
ttl_votes_by_combined_key
```

    combined_key
    南投縣中寮鄉中寮村      443
    南投縣中寮鄉內城村      297
    南投縣中寮鄉八仙村      535
    南投縣中寮鄉和興村      422
    南投縣中寮鄉崁頂村      304
                 ...
    高雄市鼓山區鼓岩里      847
    高雄市鼓山區鼓峰里     1425
    高雄市鼓山區龍井里      906
    高雄市鼓山區龍子里    11410
    高雄市鼓山區龍水里    16333
    Name: votes, Length: 7737, dtype: int64

## Votes percentage by each candidate and village

```python
ttl_votes_by_combined_key_candidates = presidential_df.groupby(['combined_key', 'number'])['votes'].sum()
soong = ttl_votes_by_combined_key_candidates[:, '1'] / ttl_votes_by_combined_key
han = ttl_votes_by_combined_key_candidates[:, '2'] / ttl_votes_by_combined_key
tsai = ttl_votes_by_combined_key_candidates[:, '3'] / ttl_votes_by_combined_key
votes_obtained = pd.concat([soong, han, tsai], axis=1)
votes_obtained.columns = ['soong', 'han', 'tsai']
```

```python
votes_obtained
```

      soong
      han
      tsai

      combined_key

      南投縣中寮鄉中寮村
      0.040632
      0.489842
      0.469526

      南投縣中寮鄉內城村
      0.057239
      0.474747
      0.468013

      南投縣中寮鄉八仙村
      0.039252
      0.435514
      0.525234

      南投縣中寮鄉和興村
      0.021327
      0.500000
      0.478673

      南投縣中寮鄉崁頂村
      0.052632
      0.381579
      0.565789

      ...
      ...
      ...
      ...

      高雄市鼓山區鼓岩里
      0.014168
      0.309327
      0.676505

      高雄市鼓山區鼓峰里
      0.032982
      0.473684
      0.493333

      高雄市鼓山區龍井里
      0.023179
      0.367550
      0.609272

      高雄市鼓山區龍子里
      0.032340
      0.381420
      0.586240

      高雄市鼓山區龍水里
      0.037654
      0.398947
      0.563399

7737 rows × 3 columns

## Calculate cosine similarity

```python
a = national_percentage.values
a_norm = np.linalg.norm(a)
cos_similarities = []
for i in range(votes_obtained.shape[0]):
    b = votes_obtained.iloc[i, :].values
    b_norm = np.linalg.norm(b)
    ab = np.dot(a, b)
    cos_similarity = np.dot(a, b) / (a_norm*b_norm)
    cos_similarities.append(cos_similarity)
votes_obtained = votes_obtained.assign(cosine_similarity=cos_similarities)
votes_obtained = votes_obtained.reset_index()
```

```python
votes_obtained.head()
```

      combined_key
      soong
      han
      tsai
      cosine_similarity

      0
      南投縣中寮鄉中寮村
      0.040632
      0.489842
      0.469526
      0.977648

      1
      南投縣中寮鄉內城村
      0.057239
      0.474747
      0.468013
      0.980246

      2
      南投縣中寮鄉八仙村
      0.039252
      0.435514
      0.525234
      0.995217

      3
      南投縣中寮鄉和興村
      0.021327
      0.500000
      0.478673
      0.977015

      4
      南投縣中寮鄉崁頂村
      0.052632
      0.381579
      0.565789
      0.999882

## Sort by cosine similarity with descending order to find 「章魚里」

```python
votes_obtained.sort_values(['cosine_similarity', 'combined_key'], ascending=[False, True]).reset_index(drop=True).head(10)
```

      combined_key
      soong
      han
      tsai
      cosine_similarity

      0
      嘉義縣番路鄉內甕村
      0.042553
      0.386018
      0.571429
      1.000000

      1
      臺南市東區關聖里
      0.042450
      0.386295
      0.571255
      1.000000

      2
      臺南市中西區南門里
      0.043410
      0.385460
      0.571130
      0.999999

      3
      新北市汐止區保長里
      0.042833
      0.386847
      0.570320
      0.999999

      4
      新北市金山區五湖里
      0.043765
      0.385632
      0.570603
      0.999998

      5
      臺北市南港區東新里
      0.042036
      0.385298
      0.572666
      0.999997

      6
      臺北市內湖區西湖里
      0.041285
      0.386008
      0.572707
      0.999997

      7
      新北市中和區清穗里
      0.041865
      0.385347
      0.572788
      0.999997

      8
      臺南市北區重興里
      0.042837
      0.384831
      0.572331
      0.999997

      9
      新北市板橋區景星里
      0.042515
      0.387607
      0.569878
      0.999996

## Can we find the similarity of our own village?

Definitely.

```python
def find_my_village(my_village, df):
    df = df.sort_values(['cosine_similarity', 'combined_key'], ascending=[False, True]).reset_index(drop=True)
    my_village_df = df[df['combined_key'] == my_village]
    return my_village_df
```

```python
my_village = '高雄市鼓山區桃源里'
my_village_df = find_my_village(my_village, votes_obtained)
my_village_similarity = my_village_df['cosine_similarity'].values[0]
my_village_rank = my_village_df.index[0]
n_rows = votes_obtained.shape[0]
print("{}的餘弦相似度為{:.4f}, 排名{}/{}".format(my_village, my_village_similarity, my_village_rank, n_rows))
my_village_df
```

    高雄市鼓山區桃源里的餘弦相似度為0.9985, 排名1714/7737

      combined_key
      soong
      han
      tsai
      cosine_similarity

      1714
      高雄市鼓山區桃源里
      0.023419
      0.370023
      0.606557
      0.998506

## Feeling movivated?

## Start with the most practical one: Python

- Procedural programming with Python
- Object-oriented programming with Python
- Using Python libraries

## Start with the most practical one: R

- Procedural programming with R
- Functional programming with R
- Using R libraries

## Resources I've used when learning Python

- [Introducing Python](https://www.amazon.com/Introducing-Python-Modern-Computing-Packages/dp/1449359361)
- [A Whirlwind Tour of Python](https://jakevdp.github.io/WhirlwindTourOfPython/index.html)
- [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/)

## Resources I've used when learning R

- [The Art of R Programming](https://www.amazon.com/Art-Programming-Statistical-Software-Design/dp/1593273843)
- [Advanced R](https://adv-r.hadley.nz/)
- [R for Data Science](https://r4ds.had.co.nz/)
- [Data Science Specialization](https://www.coursera.org/specializations/jhu-data-science)
- [Statistics with R Specialization](https://www.coursera.org/specializations/statistics)

## Learning resources from me

- [數據交點](https://www.datainpoint.com)
- [Substack](https://datainpoint.substack.com/about)

## Phew, that is a lot to catch up...

You do not have to finish every course or book from end to end.

Source: <https://giphy.com/>


```python
from LOTlib3.Grammar import Grammar
from math import log
from LOTlib3.Hypotheses.LOTHypothesis import LOTHypothesis
from LOTlib3.DataAndObjects import FunctionData
from LOTlib3.TopN import TopN
from LOTlib3.Samplers.MetropolisHastings import MetropolisHastingsSampler
from LOTlib3.Eval import primitive
```

```python
import numpy as np
import pandas as pd

import os
import json
import time
import copy

from IPython.display import clear_output
from IPython.display import HTML

import matplotlib.pyplot as plt
from matplotlib import animation
from matplotlib import colors

from PIL import Image, ImageDraw
from IPython.display import Image as Im
from itertools import permutations, combinations
from sympy.utilities.iterables import multiset_permutations

# from skimage.segmentation import flood, flood_fill

%matplotlib inline
```

```python
# define solver
class ARCSolver:
    def __init__(self, task_filename):
        # load task and extract input and output pairs
        self.task_filename = task_filename
        self.task = self.load_task(task_filename)
        self.train_inputs, self.train_outputs, self.test_inputs, self.test_outputs = \
            self.extract_io_pairs()
        self.test_pred = np.zeros((5, 5))
        self.test_pred_height, self.test_pred_width = self.test_pred.shape

        self.solved = False  # have we solved the task yet?
        self.selected_colour = 0
        self.clipboard = None
        self.description = ''

        # variables for plotting
        self.cmap = colors.ListedColormap(
            ['#000000', '#0074D9','#FF4136','#2ECC40','#FFDC00',
             '#AAAAAA', '#F012BE', '#FF851B', '#7FDBFF', '#870C25'])
        self.colour_to_num = {'black': 0, 'blue': 1, 'red': 2, 'green': 3, 'yellow': 4,
                             'grey': 5, 'magenta': 6, 'orange': 7, 'light_blue': 8,
                             'maroon': 9}
        self.num_to_colour = {0: 'black', 1: 'blue', 2: 'red', 3: 'green', 4: 'yellow',
                             5: 'grey', 6: 'magneta', 7: 'orange', 8: 'light_blue',
                             9: 'maroon'}

    def load_task(self, task_filename):
        with open(task_filename, 'r') as f:
            task = json.load(f)
        return task

    def plot_task(self):
        """
        Plots the first train and test pairs of a specified task,
        using same color scheme as the ARC app
        """
        norm = colors.Normalize(vmin=0, vmax=9)
        n_train = len(self.task['train'])
        fig, axs = plt.subplots(n_train+1, 2, figsize=(10, 10))
        for i in range(n_train):
            axs[i, 0].imshow(self.task['train'][i]['input'], cmap=self.cmap, norm=norm)
            axs[i, 0].axis('off')
            axs[i, 0].set_title('Train Input')
            axs[i, 1].imshow(self.task['train'][i]['output'], cmap=self.cmap, norm=norm)
            axs[i, 1].axis('off')
            axs[i, 1].set_title('Train Output')
        axs[n_train, 0].imshow(self.task['test'][0]['input'], cmap=self.cmap, norm=norm)
        axs[n_train, 0].axis('off')
        axs[n_train, 0].set_title('Test Input')
        axs[n_train, 1].imshow(self.task['test'][0]['output'], cmap=self.cmap, norm=norm)
        axs[n_train, 1].axis('off')
        axs[n_train, 1].set_title('Test Output')
        plt.tight_layout()
        plt.show()

    def plot_grid(self, grid):
        """
        Plots a single grid
        """
        #plt.clf()

        #plt.draw()
        #display(plt)

    def plot_grids(self, grids):
        """
        Plots a list of grids
        """
        n_grids = len(grids)
        norm = colors.Normalize(vmin=0, vmax=9)
        fig, axs = plt.subplots(1, n_grids, figsize=(6, 6), squeeze=False)
        for i in range(n_grids):
            axs[0, i].imshow(grids[i], cmap=self.cmap, norm=norm)
            axs[0, i].axis('off')
        plt.tight_layout()
        plt.show()

    def extract_io_pairs(self):
        train = self.task['train']
        test = self.task['test']
        n_train = len(train)
        n_test = len(test)

        train_inputs = [train[i]['input'] for i in range(n_train)]
        train_inputs = [np.asarray(i) for i in train_inputs]
        train_outputs = [train[i]['output'] for i in range(n_train)]
        train_outputs = [np.asarray(i) for i in train_outputs]
        test_inputs = [test[i]['input'] for i in range(n_test)]
        test_inputs = [np.asarray(i) for i in test_inputs]
        test_outputs = [test[i]['output'] for i in range(n_test)]
        test_outputs = [np.asarray(i) for i in test_outputs]

        return train_inputs, train_outputs, test_inputs, test_outputs

```

```python
colorMap = {0:"black",1:"blue",2:"red", 3:"green",4:"yellow",5:"grey",6:"magenta",7:"orange",8:"cyan",9:"brown"}
```

```python

```

```python
# intializing basic hypothesis, will edit later
class ARCHypothesis(LOTHypothesis):
    def __init__(self, **kwargs):
        LOTHypothesis.__init__(self, grammar=grammar, display="lambda grid: %s", **kwargs)

    def __call__(self, *args):
        try:
            # try to do it from the superclass
            return LOTHypothesis.__call__(self, *args)
        except ZeroDivisionError:
            # and if we get an error, return nan
            return float("nan")

    def compute_single_likelihood(self, datum):
        # for each correct pixel, add log(1), for each incorrect, add log(exp(-b))
        ll = 0
        input_grid = self(*datum.input)
        output_grid = datum.output
        # assuming input and output are the same size for now
        height, width = len(input_grid[:,0]), len(input_grid[0,:])
        for i in range(height):
            for j in range(width):
                if input_grid[i][j] == output_grid[i][j]:
                    ll += log(1)
#                 elif input_grid[i][j] == 0 and output_grid[i][j] != 0:
#                     ll += log(0.1)
                else:
                    ll += log(0.05)
        # print(self.value, ll)
        return ll

    def display_hypothesis(self):
        return self.value
```

```python
# initializing grammar
grammar = Grammar(start='GRID')

# Boolean
grammar.add_rule('BOOLEAN', True, None, 1.0)
grammar.add_rule('BOOLEAN', False, None, 1.0)
grammar.add_rule('BOOLEAN', 'not %s', ['BOOLEAN'], 1.0)
grammar.add_rule('BOOLEAN', '%s or %s', ['BOOLEAN', 'BOOLEAN'], 1.0)
grammar.add_rule('BOOLEAN', '%s and %s', ['BOOLEAN', 'BOOLEAN'], 1.0)
grammar.add_rule('BOOLEAN', 'equals_object', ['OBJECT', 'OBJECT'], 1.0)
grammar.add_rule('BOOLEAN', 'equals_location', ['LOC', 'LOC'], 1.0)
grammar.add_rule('BOOLEAN', 'equals_color', ['COLOR', 'COLOR'], 1.0)
grammar.add_rule('BOOLEAN', 'equals_int', ['INT', 'INT'], 1.0)
grammar.add_rule('BOOLEAN', '%s > %s', ['INT', 'INT'], 1.0)
grammar.add_rule('BOOLEAN', '%s < %s', ['INT', 'INT'], 1.0)
# grammar.add_rule('BOOLEAN', 'is_touching', ['OBJECT', 'OBJECT'], 1.0)
# grammar.add_rule('BOOLEAN', 'has_color', ['OBJECT', 'COLOR'], 1.0)
# grammar.add_rule('BOOLEAN', 'is_color', ['LOC', 'COLOR'], 1.0)
# grammar.add_rule('BOOLEAN', 'is_size', ['OBJECT', 'INT'], 1.0)

# Integer
for n in range(10):
    grammar.add_rule('INT', n, None, 1.0)
grammar.add_rule('INT', 'object_height', ['OBJECT'], 1.0)
grammar.add_rule('INT', 'object_width', ['OBJECT'], 1.0)
grammar.add_rule('INT', 'num_objects', ['GRID'], 100.0)
grammar.add_rule('INT', 'num_grid_colors', ['GRID'], 1.0)
grammar.add_rule('INT', 'num_colors', ['OBJECT'], 1.0)
grammar.add_rule('INT', 'grid_height', ['GRID'], 1.0)
grammar.add_rule('INT', 'grid_width', ['GRID'], 1.0)

# Colors
for n in range(10):
    grammar.add_rule('COLOR', str(n), None, 1.0)
grammar.add_rule('COLOR', 'object_color', ['OBJECT'], 1.0)
grammar.add_rule('COLOR', 'background_color', ['GRID'], 1.0)
grammar.add_rule('COLOR', 'get_color', ['GRID','LOC'], 1.0)

# Location
for i in range(30):
    for j in range(30):
        grammar.add_rule('LOC', [i,j], None, 1.0)
grammar.add_rule('LOC', 'top_left', ['OBJECT'],1.0)
grammar.add_rule('LOC', 'top_right', ['OBJECT'],1.0)
grammar.add_rule('LOC', 'bottom_left', ['OBJECT'],1.0)
grammar.add_rule('LOC', 'bottom_right', ['OBJECT'],1.0)
grammar.add_rule('LOC', 'top_left_corner', ['GRID'],1.0)
grammar.add_rule('LOC', 'top_right_corner', ['GRID'],1.0)
grammar.add_rule('LOC', 'bottom_left_corner', ['GRID'],1.0)
grammar.add_rule('LOC', 'bottom_right_corner', ['GRID'],1.0)

# Dictionary

# List/Set

# SetB
# SetC
# AllColors(Grid)
# SetL
# SetO

# Objects
grammar.add_rule('OBJECT', 'object_by_color', ['GRID', 'COLOR'], 1.0)

# Grid
grammar.add_rule('GRID', 'grid', None, 5.0)
# grammar.add_rule('GRID', 'color_grid', ['GRID','COLOR'], 1.0)
grammar.add_rule('GRID', 'reflection_horizontal', ['GRID'], 1.0)
grammar.add_rule('GRID', 'reflection_vertical', ['GRID'], 1.0)
grammar.add_rule('GRID', 'rotation_90', ['GRID'], 1.0)
grammar.add_rule('GRID', 'rotation_180', ['GRID'], 1.0)
grammar.add_rule('GRID', 'rotation_270', ['GRID'], 1.0)
# grammar.add_rule('GRID', 'scale', ['GRID', 'INT'], 1.0)
# grammar.add_rule('GRID', 'repeat', ['GRID'], 1.0)
grammar.add_rule('GRID', 'gravity', ['GRID'], 1.0)
# grammar.add_rule('GRID', 'replace', ['OBJECT','OBJECT','GRID'], 1.0)
grammar.add_rule('GRID', 'logic_and', ['GRID','GRID'], 1.0)
grammar.add_rule('GRID', 'logic_or', ['GRID','GRID'], 1.0)
grammar.add_rule('GRID', 'logic_xor', ['GRID','GRID'], 1.0)
grammar.add_rule('GRID', 'move_object_vertical', ['GRID', 'OBJECT', 'INT'], 1.0)
grammar.add_rule('GRID', 'move_object_horizontal', ['GRID', 'OBJECT', 'INT'], 1.0)
# extend color
# connect color

```

    GRID -> move_object_horizontal['GRID', 'OBJECT', 'INT']	w/ p=1.0

```python
### Object Primitives ###
@primitive
def object_by_color(grid, color):
    obj = []
    coords = list(zip(np.where(grid==color)[0],np.where(grid==color)[1]))
    for coord in coords:
        obj.append([coord[0],coord[1],color])
    return obj

### Boolean Primitives ###
@primitive
def equals_object(obj1,obj2):
    return sorted(relative_positions(obj1)) == sorted(relative_positions(obj2))

@primitive
def equals_location(loc1,loc2):
    if loc1[0] == loc2[0] and loc1[1] == loc2[1]:
        return True
    return False

@primitive
def equals_color(col1,col2):
    if col1 == col2:
        return True
    return False

@primitive
def equals_int(int1,int2):
    if int1==int2:
        return True
    return False

### Integer Primitives ###
@primitive
def object_height(obj):
    return max(obj, key=lambda x: x[0])[0] - min(obj, key=lambda x: x[0])[0]

@primitive
def object_width(obj):
    return max(obj, key=lambda x: x[1])[1] - min(obj, key=lambda x: x[1])[1]

@primitive
def grid_height(grid):
    return len(grid[:,0])

@primitive
def grid_width(grid):
    return len(grid[0,:])

@primitive
def num_objects(grid):
    # for now just using objects by color, and assuming black is background
    obj_counter = 0
    for i in range(1,10):
        if object_by_color(grid,i) != []:
            obj_counter += 1
    return obj_counter

@primitive
def num_grid_colors(grid):
    # assuming there's always a background color
    return len(np.unique(grid)) - 1

@primitive
def num_colors(obj):
    return len(np.unique(obj))

### Color Primitives ###
@primitive
def object_color(obj):
    # return the color that appears the most times in the object
    clrs, counts = np.unique(obj[:,2], return_counts=True)
    return clrs[np.argmax(counts)]

@primitive
def background_color(grid):
    # return the color that appears the most times in the grid
    clrs, counts = np.unique(grid, return_counts=True)
    return clrs[np.argmax(counts)]

@primitive
def get_color(grid,loc):
    return grid[loc[0]][loc[1]]

### Location Primitives ###
@primitive
def top_left(obj):
    sorted_obj = sorted(obj, key=lambda x: x[0])
    return sorted_obj[0][:2]

@primitive
def top_right(obj):
    sorted_obj = sorted(obj, key=lambda x: x[1], reverse=True)
    sorted_obj = sorted(sorted_obj, key=lambda x: x[0])
    return sorted_obj[0][:2]

@primitive
def bottom_left(obj):
    sorted_obj = sorted(obj, key=lambda x: x[1])
    sorted_obj = sorted(sorted_obj, key=lambda x: x[0], reverse=True)
    return sorted_obj[0][:2]

@primitive
def bottom_right(obj):
    sorted_obj = sorted(obj, key=lambda x: x[0], reverse=True)
    sorted_obj = sorted(sorted_obj, key=lambda x: x[1], reverse=True)
    return sorted_obj[0][:2]

@primitive
def top_left_corner(grid):
    return [0,0]

@primitive
def top_right_corner(grid):
    return [0,grid_width(grid)]

@primitive
def bottom_left_corner(grid):
    return [grid_height(grid),0]

@primitive
def bottom_right_corner(grid):
    return [grid_height(grid),grid_width(grid)]

# Grid primitives
@primitive
def color_grid(grid, color):
    for i in range(grid_height(grid)):
        for j in range(grid_width(grid)):
            grid[i][j] = color
    return grid

@primitive
def reflection_vertical(grid):
    return np.flipud(grid)

@primitive
def reflection_horizontal(grid):
    return np.fliplr(grid)

@primitive
def rotation_90(grid):
    return np.rot90(grid,1)

@primitive
def rotation_180(grid):
    return np.rot90(grid,2)

@primitive
def rotation_270(grid):
    return np.rot90(grid,3)

@primitive
def gravity(grid):
    background_col = background_color(grid)
    grid_copy = copy.deepcopy(grid)
    grid_copy = color_grid(grid_copy, background_col)
    for i in range(grid_width(grid)):
        col = grid[:,i]
        grav = col[col!=background_col]
        for k in range(1,len(grav)+1):
            grid_copy[-k,i] = grav[-k]
    return grid_copy

@primitive
def logic_and(grid1,grid2):
    # assuming background is black for now
    # if they're not the same size, arbitrarily return the first
    if grid_height(grid1) != grid_height(grid2) or grid_width(grid1) != grid_width(grid2):
        return grid1
    else:
        return np.logical_and(grid1,grid2)

@primitive
def logic_or(grid1,grid2):
    # assuming background is black for now
    # if they're not the same size, arbitrarily return the first
    if grid_height(grid1) != grid_height(grid2) or grid_width(grid1) != grid_width(grid2):
        return grid1
    else:
        return np.logical_or(grid1,grid2)

@primitive
def logic_xor(grid1,grid2):
    # assuming background is black for now
    # if they're not the same size, arbitrarily return the first
    if grid_height(grid1) != grid_height(grid2) or grid_width(grid1) != grid_width(grid2):
        return grid1
    else:
        return np.logical_xor(grid1,grid2)

# @primitive
# def replace(obj1,obj2,grid):
#     # repace obj1 by obj2
#     back_col = background_color(grid)
#     h1,w1 = object_height(obj1), object_width(obj1)
#     h2,w2 = object_height(obj2), object_width(obj2)
#     if h1 != h2 or w1 != w2:
#         return grid
#     else:
#         obj2_copy = np.ones((h2,w2))*back_col
#         obj2_rel_pos = relative_positions(obj2)
#         for pos in obj2_rel_pos:
#             obj2_copy[pos[0]][pos[1]] = pos[2]

@primitive
def move_object_vertical(grid, obj, int_):
    h,w = grid_height(grid), grid_width(grid)
    grid_copy = copy.deepcopy(grid)
    back_col = background_color(grid)
    for coord in obj:
        dy = coord[0]+int_
        if dy > h or dy < 0:
            return grid
        grid_copy[coord[0]][coord[1]]= back_col
        grid_copy[dy][coord[1]] = coord[2]
    return grid_copy

@primitive
def move_object_horizontal(grid, obj, int_):
    h,w = grid_height(grid), grid_width(grid)
    grid_copy = copy.deepcopy(grid)
    back_col =  background_color(grid)
    for coord in obj:
        dx = coord[1]+int_
        if dx > w or dx < 0:
            return grid
        grid_copy[coord[0]][coord[1]]= back_col
        grid_copy[coord[0]][dx] = coord[2]
    return grid_copy

# Helper functions
def relative_positions(obj):
    dx, dy = top_left(obj)[0], top_left(obj)[1]
    rel_obj = []
    for coord in obj:
        rel_obj.append([coord[0]-dx,coord[1]-dy, coord[2]])
    return rel_obj
```

```python

```

```python
training_path = "/Users/aysjajohnson/Desktop/ARC-master/data/training/"
solver = ARCSolver(task_filename=os.path.join(training_path, '0ca9ddb6.json'))
# solver = ARCSolver(task_filename=os.path.join(training_path, '0a938d79.json'))
solver.plot_grids(solver.train_inputs)
solver.plot_grids(solver.train_outputs)
```

```python
inputs = solver.train_inputs
outputs = solver.train_outputs
grid1 = inputs[0]
grid2 = outputs[0]
grid3 = outputs[1]
```

```python

```

```python
h0 = ARCHypothesis()
```

```python
for i in range(10):
    h0 = ARCHypothesis()
    print(h0)
    print(h0(grid1))
```

```python
h0(grid1)
```

    array([[0, 0, 0, 0, 0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0, 0, 0, 0, 0],
           [0, 0, 2, 0, 0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0, 0, 1, 0, 0],
           [0, 0, 0, 0, 0, 0, 0, 0, 0],
           [0, 0, 0, 0, 0, 0, 0, 0, 0]])

```python

```


# Python 预缓存的 Property 修饰器简单实现

> 创建时间：2021-03-19

这份简短笔记，我们会讨论预缓存 Python Property (类属性) 的简单实现。

在运行程序时，特别是对于耗时但不耗内存、以后需要经常取用的计算，我们会希望找个内存或硬盘空间储存起来。为了方便起见，我们只讨论借用内存的方法。

假设现在的问题是，我们要计算 $c = a+b, d = a^2$。为了调用的便利，$a, b$ 两个变量 (作为常数) 作为 property。

但麻烦之处在于，$a, b$ 的值并不容易求，求完之后对内存的消耗却又不大。(嘛先不要追问为什么求个 1+1 要这么复杂)

$$
\begin{align}
a &= \sum_{n = 1}^{\infty} \frac{1}{2^n} \simeq \sum_{n = 1}^{1000} \frac{1}{2^n} \\
b &= \int_0^1 3 x^2 \, \mathrm{d} x \simeq \sum_{n = 0}^{5000} \frac{3 n^2}{5000^3}
\end{align}
$$

同时，我们也不清楚末端用户是否需要求 $c$ (需要同时计算 $a$, $b$)，还是需要求 $d$ (只需要计算 $a$ 即可)。

```python
def get_a():
    a = 0
    for n in range(1, 1001):
        a += 1 / 2**n
    return a
```

```python
def get_b():
    b = 0
    for n in range(5001):
        b += 3 * n**2 / 5000**3
    return b
```

这篇文档讨论四种做法。第一种做法简单但低效；第二、三种做法代码较复杂；第四种代码简单且不会产生多余的计算。作者倾向使用 [第三种](#偷懒的做法：将赋值函数嵌入-getter-函数) 与 [第四种](#改进的做法：缩减隐含变量的声明) 做法。

## 即时调用 property 定义方法

最简单粗暴的方法是需要 $a, b$ 时就现场计算。在第一次调用 $a, b$ 时固然需要耗时的计算，但第二次调用仍然会相当费时。

```python
class Dummy:

    @property
    def a(self):
        return get_a()

    @property
    def b(self):
        return get_b()

    @property
    def c(self):
        return self.a + self.b

    @property
    def d(self):
        return self.a**2
```

```python
dum = Dummy()
dum.c
```

    2.00030002

```python
%%timeit -n 50
dum.c
```

    2.41 ms ± 172 µs per loop (mean ± std. dev. of 7 runs, 50 loops each)

## 一般的 property 做法

为了避免多余的计算，一般的方法是，需要首先在 `__init__` 中声明两个隐含变量 `_a`, `_b` 以保存结果。在使用 `a`, `b` 两个 property 之前，先要使用 setter 函数作 $a, b$ 的计算并分别保存到 `_a`, `_b` 中；随后再用 getter 函数调用它们。

```python
class General:

    def __init__(self):
        self._a = NotImplemented
        self._b = NotImplemented

    @property
    def a(self):
        return self._a

    @a.setter
    def a(self, val):
        self._a = val

    @property
    def b(self):
        return self._a

    @b.setter
    def b(self, val):
        self._b = val

    @property
    def c(self):
        return self.a + self.b

    @property
    def d(self):
        return self.a**2
```

如果没有预先使用 setter 函数，就会碰到下面这种尴尬的情况：

```python
gen = General()
gen.c
```

因此，正确的调用方式是

```python
gen = General()
gen.a, gen.b = get_a(), get_b()
gen.c, gen.d
```

    (2.0, 1.0)

上面的步骤是耗时的，但随后当要调用 `a`, `b` 变量时，就会快捷很多：

```python
%%timeit -n 50
gen.c
```

    265 ns ± 28 ns per loop (mean ± std. dev. of 7 runs, 50 loops each)

但这里有一个问题：$a, b$ 的值实际上可以看作常数；如果末端用户真正希望得到的是 $d = a^2$ 而非 $c = a + b$，那么实际上用户不需要 $b$，自然也就不需要对其花时间赋值了。决定是否要对 $b$ 赋值的任务由此交给末端用户，这会造成一些困扰。

## 偷懒的做法：将赋值函数嵌入 getter 函数

如果这个任务交给程序编写者，那么一种最简单的实现方式是把赋值函数嵌入到 getter 函数中：

- 如果 $a$ 的值已经被计算过，那么就从缓存空间 `_a` 取出该值；

- 如果 $a$ 被调用前没有被计算过，那么就计算该值并放入缓存 `_a`。

```python
class Improved:

    def __init__(self):
        self._a = NotImplemented
        self._b = NotImplemented

    @property
    def a(self):
        if self._a is NotImplemented:
            self._a = get_a()
        return self._a

    @property
    def b(self):
        if self._b is NotImplemented:
            self._b = get_b()
        return self._b

    @property
    def c(self):
        return self.a + self.b

    @property
    def d(self):
        return self.a**2
```

如果末端用户只需要求 $d = a^2$，那么耗费时间的关键步就只有 $a$ 的计算；缓存空间 `_b` 就是空的。

```python
imp = Improved()
print(imp.d)
print(imp._b)
```

    1.0
    NotImplemented

同时，以后再需要调用 $d$ 时，$a$ 的值也不会再被计算第二次。

当然，这种做法的弊端是，用户原则上无权限更改 $a, b$ 的值 (通过更改隐含变量 `_a`, `_b` 是可能的，但这违背了 PEP8 的程序规范)。

## 改进的做法：缩减隐含变量的声明

但上面的定义仍然有很多冗余。对于每个 property，我们总要声明隐含变量、调用时判断是否缓存空间存在。这两步可以通过改编在 property 修饰器内部增加一段代码方便地实现。这个修饰器我们命名为 `cached_property`：

```python
def cached_property(f):
    def wrap(*args, **kwargs):
        self = args[0]                                                    # self
        _f = "_" + f.__name__                                             # _a
        if not hasattr(self, _f) or getattr(self, _f) is NotImplemented:  # if self._a is NotImplemented:
            setattr(self, _f, f(*args))                                   # self._a = get_a()
        return getattr(self, _f)                                          # return self._a
    return property(wrap)                                                 # make this wrap a property
```

这样之后，不仅代码量减少很多 (调用方式与最简单的 `Dummy` 完全一致)，同时也避免多余重复的计算。

```python
class Advanced:

    @cached_property
    def a(self):
        return get_a()

    @cached_property
    def b(self):
        return get_b()

    @property
    def c(self):
        return self.a + self.b

    @property
    def d(self):
        return self.a**2
```

```python
adv = Advanced()
print(adv.d)
print(hasattr(adv, "_b"))
```

    1.0
    False