```
!pip install qiskit
```

    Collecting Qiskit
      Downloading https://files.pythonhosted.org/packages/df/02/16212b2c01f2652c094bb865f2786d4e7a6206472898ee12137d98a85aca/qiskit-0.23.5.tar.gz
    Collecting qiskit-terra==0.16.4
      Downloading https://files.pythonhosted.org/packages/56/e2/c256863ec8fab2162bf17d7e9c8564186a4636b4f24b5ea5d38a014b809e/qiskit_terra-0.16.4-cp36-cp36m-manylinux2010_x86_64.whl (8.5MB)
         |████████████████████████████████| 8.5MB 4.1MB/s
    Collecting qiskit-aer==0.7.4
      Downloading https://files.pythonhosted.org/packages/c0/e0/9b28d7bd52cb6b2bc16b8e98b97e96dc7d22f5c40f91b529487b31526c8c/qiskit_aer-0.7.4-cp36-cp36m-manylinux2010_x86_64.whl (17.6MB)
         |████████████████████████████████| 17.6MB 255kB/s
    Collecting qiskit-ibmq-provider==0.11.1
      Downloading https://files.pythonhosted.org/packages/32/b9/f99bec4fdc4dec234d8a85a8da378750441a203c614be35353f5e8738316/qiskit_ibmq_provider-0.11.1-py3-none-any.whl (195kB)
         |████████████████████████████████| 204kB 50.6MB/s
    Collecting qiskit-ignis==0.5.2
      Downloading https://files.pythonhosted.org/packages/b2/1d/d5df8cf48cf20a26deb6e0535f020cd2ad625cb27457abb0ab39c2104e38/qiskit_ignis-0.5.2-py3-none-any.whl (203kB)
         |████████████████████████████████| 204kB 57.0MB/s
    Collecting qiskit-aqua==0.8.2
      Downloading https://files.pythonhosted.org/packages/0e/82/528741676960a7fd1c75151639601826118f2729ad53034fa3b39290c056/qiskit_aqua-0.8.2-py3-none-any.whl (2.1MB)
         |████████████████████████████████| 2.1MB 51.9MB/s
    Collecting ply>=3.10
      Downloading https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49kB)
         |████████████████████████████████| 51kB 7.7MB/s
    Collecting python-constraint>=1.4
      Downloading https://files.pythonhosted.org/packages/37/8b/5f1bc2734ca611943e1d6733ee244238679f6410a10cd45ede55a61a8402/python-constraint-1.4.0.tar.bz2
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.4->Qiskit) (2.8.1)
    Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.4->Qiskit) (2.6.0)
    Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.4->Qiskit) (2.5)
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.4->Qiskit) (5.4.8)
    Requirement already satisfied: scipy>=1.4 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.4->Qiskit) (1.4.1)
    Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.4->Qiskit) (1.19.5)
    Collecting fastjsonschema>=2.10
      Downloading https://files.pythonhosted.org/packages/89/1c/8be51fa42aadc1c1611a52b866e1a5a1032a504f24789cf140b4e6d7c940/fastjsonschema-2.15.0-py3-none-any.whl
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.4->Qiskit) (0.3.3)
    Collecting retworkx>=0.5.0
      Downloading https://files.pythonhosted.org/packages/b7/21/5d4498cedb23927a31173176975e2162e0d6a2b80846ef254dd5500e10da/retworkx-0.7.2-cp36-cp36m-manylinux2010_x86_64.whl (1.3MB)
         |████████████████████████████████| 1.3MB 33.5MB/s
    Collecting sympy>=1.3
      Downloading https://files.pythonhosted.org/packages/ff/69/b16fc81b939d3efdd0b552f2e3e54f7fa1423d0c320cced2e69e675dde26/sympy-1.7.1-py3-none-any.whl (5.9MB)
         |████████████████████████████████| 5.9MB 52.6MB/s
    Collecting contextvars>=2.4; python_version < "3.7"
      Downloading https://files.pythonhosted.org/packages/83/96/55b82d9f13763be9d672622e1b8106c85acb83edd7cc2fa5bc67cd9877e9/contextvars-2.4.tar.gz
    Collecting pybind11>=2.4
      Downloading https://files.pythonhosted.org/packages/8d/43/7339dbabbc2793718d59703aace4166f53c29ee1c202f6ff5bf8a26c4d91/pybind11-2.6.2-py2.py3-none-any.whl (191kB)
         |████████████████████████████████| 194kB 56.8MB/s
    Requirement already satisfied: cython>=0.27.1 in /usr/local/lib/python3.6/dist-packages (from qiskit-aer==0.7.4->Qiskit) (0.29.21)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from qiskit-ibmq-provider==0.11.1->Qiskit) (1.24.3)
    Collecting websockets>=8
      Downloading https://files.pythonhosted.org/packages/bb/d9/856af84843912e2853b1b6e898ac8b802989fcf9ecf8e8445a1da263bf3b/websockets-8.1-cp36-cp36m-manylinux2010_x86_64.whl (78kB)
         |████████████████████████████████| 81kB 9.7MB/s
    Collecting requests-ntlm>=1.1.0
      Downloading https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.6/dist-packages (from qiskit-ibmq-provider==0.11.1->Qiskit) (2.23.0)
    Requirement already satisfied: nest-asyncio!=1.1.0,>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from qiskit-ibmq-provider==0.11.1->Qiskit) (1.5.1)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.6/dist-packages (from qiskit-ignis==0.5.2->Qiskit) (53.0.0)
    Requirement already satisfied: scikit-learn>=0.17 in /usr/local/lib/python3.6/dist-packages (from qiskit-ignis==0.5.2->Qiskit) (0.22.2.post1)
    Collecting quandl
      Downloading https://files.pythonhosted.org/packages/c2/58/9f0e69d836045e3865d263e9ed49f42b23a58526fdabb30f74c430baee3f/Quandl-3.6.0-py2.py3-none-any.whl
    Collecting dlx
      Downloading https://files.pythonhosted.org/packages/54/c0/b8fb5bb727e983b6f5251433ef941b48f38c65bb0bd6ec509e9185bcd406/dlx-1.0.4.tar.gz
    Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.8.2->Qiskit) (2.10.0)
    Collecting docplex==2.15.194
      Downloading https://files.pythonhosted.org/packages/9b/76/ebf5b75a25c4357c9a4758cae7fea778dd7f282b83c2f22e150dc7e3c852/docplex-2.15.194.tar.gz (582kB)
         |████████████████████████████████| 583kB 47.6MB/s
    Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.8.2->Qiskit) (1.1.5)
    Requirement already satisfied: dataclasses; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.8.2->Qiskit) (0.8)
    Collecting yfinance
      Downloading https://files.pythonhosted.org/packages/7a/e8/b9d7104d3a4bf39924799067592d9e59119fcfc900a425a12e80a3123ec8/yfinance-0.1.55.tar.gz
    Requirement already satisfied: fastdtw in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.8.2->Qiskit) (0.3.4)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.8.0->qiskit-terra==0.16.4->Qiskit) (1.15.0)
    Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=2.2->qiskit-terra==0.16.4->Qiskit) (4.4.2)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy>=1.3->qiskit-terra==0.16.4->Qiskit) (1.1.0)
    Collecting immutables>=0.9
      Downloading https://files.pythonhosted.org/packages/4a/52/e64a14a99c509cbdfe0405e9f076aef0331cb9548a3efa1d5bacd524978a/immutables-0.15-cp36-cp36m-manylinux1_x86_64.whl (100kB)
         |████████████████████████████████| 102kB 11.7MB/s
    Collecting ntlm-auth>=1.0.2
      Downloading https://files.pythonhosted.org/packages/ff/84/97c550164b54942b0e908c31ef09d9469f3ba4cd7332a671e2125732f63b/ntlm_auth-1.5.0-py2.py3-none-any.whl
    Collecting cryptography>=1.3
      Downloading https://files.pythonhosted.org/packages/2f/61/6f4765822d1f3878c913755bd99212f1ab0b9fa808fc2197ace0790f4cdb/cryptography-3.4.5-cp36-abi3-manylinux2014_x86_64.whl (3.2MB)
         |████████████████████████████████| 3.2MB 48.0MB/s
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.11.1->Qiskit) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.11.1->Qiskit) (2020.12.5)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.11.1->Qiskit) (2.10)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.17->qiskit-ignis==0.5.2->Qiskit) (1.0.0)
    Requirement already satisfied: more-itertools in /usr/local/lib/python3.6/dist-packages (from quandl->qiskit-aqua==0.8.2->Qiskit) (8.7.0)
    Collecting inflection>=0.3.1
      Downloading https://files.pythonhosted.org/packages/59/91/aa6bde563e0085a02a435aa99b49ef75b0a4b062635e606dab23ce18d720/inflection-0.5.1-py2.py3-none-any.whl
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->qiskit-aqua==0.8.2->Qiskit) (2018.9)
    Requirement already satisfied: multitasking>=0.0.7 in /usr/local/lib/python3.6/dist-packages (from yfinance->qiskit-aqua==0.8.2->Qiskit) (0.0.9)
    Collecting lxml>=4.5.1
      Downloading https://files.pythonhosted.org/packages/bd/78/56a7c88a57d0d14945472535d0df9fb4bbad7d34ede658ec7961635c790e/lxml-4.6.2-cp36-cp36m-manylinux1_x86_64.whl (5.5MB)
         |████████████████████████████████| 5.5MB 40.0MB/s
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.6/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.11.1->Qiskit) (1.14.4)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.11.1->Qiskit) (2.20)
    Building wheels for collected packages: Qiskit, python-constraint, contextvars, dlx, docplex, yfinance
      Building wheel for Qiskit (setup.py) ... done
      Created wheel for Qiskit: filename=qiskit-0.23.5-cp36-none-any.whl size=2882 sha256=2c6d6d271feca706c5447006bc74a751cfd02ad1b489fce08656a749212b9b81
      Stored in directory: /root/.cache/pip/wheels/0a/a7/8f/2720a308d16a991c6eba41517e8764158ea2c7856424112017
      Building wheel for python-constraint (setup.py) ... done
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24079 sha256=61f4f640a4b59ad68a2e41a3798525fe796ab96059122b752544256d00798c16
      Stored in directory: /root/.cache/pip/wheels/34/31/15/7b070b25d0a549d20ce2e9fe6d727471c2c61ef904720fd40c
      Building wheel for contextvars (setup.py) ... done
      Created wheel for contextvars: filename=contextvars-2.4-cp36-none-any.whl size=7667 sha256=f3c80b63366c5af8a6217a5077190d20ffb0618e23da32fbc806212621c26dd5
      Stored in directory: /root/.cache/pip/wheels/a5/7d/68/1ebae2668bda2228686e3c1cf16f2c2384cea6e9334ad5f6de
      Building wheel for dlx (setup.py) ... done
      Created wheel for dlx: filename=dlx-1.0.4-cp36-none-any.whl size=5711 sha256=14d754d31aefad0192e5d227d58de5b5a7681501bbc1535a1317c70af2a076dc
      Stored in directory: /root/.cache/pip/wheels/bb/ba/15/fdd0deb104df3254912998150ba9245668db06b00af5912d1a
      Building wheel for docplex (setup.py) ... done
      Created wheel for docplex: filename=docplex-2.15.194-cp36-none-any.whl size=645116 sha256=3cd2ad4d3771a539990fd71894a6b18564fc453e2ca6e5641082406da05be101
      Stored in directory: /root/.cache/pip/wheels/ce/fa/5d/8f4d5fed1b8edb72e453cb2ac6fb75c6776b49d7174eb70457
      Building wheel for yfinance (setup.py) ... done
      Created wheel for yfinance: filename=yfinance-0.1.55-py2.py3-none-any.whl size=22616 sha256=56825c7945bc28b878fd48409f5a361d70468174863ece7988efbe0f7eeae545
      Stored in directory: /root/.cache/pip/wheels/04/98/cc/2702a4242d60bdc14f48b4557c427ded1fe92aedf257d4565c
    Successfully built Qiskit python-constraint contextvars dlx docplex yfinance
    Installing collected packages: ply, python-constraint, fastjsonschema, retworkx, sympy, immutables, contextvars, qiskit-terra, pybind11, qiskit-aer, websockets, ntlm-auth, cryptography, requests-ntlm, qiskit-ibmq-provider, qiskit-ignis, inflection, quandl, dlx, docplex, lxml, yfinance, qiskit-aqua, Qiskit
      Found existing installation: sympy 1.1.1
        Uninstalling sympy-1.1.1:
          Successfully uninstalled sympy-1.1.1
      Found existing installation: lxml 4.2.6
        Uninstalling lxml-4.2.6:
          Successfully uninstalled lxml-4.2.6
    Successfully installed Qiskit-0.23.5 contextvars-2.4 cryptography-3.4.5 dlx-1.0.4 docplex-2.15.194 fastjsonschema-2.15.0 immutables-0.15 inflection-0.5.1 lxml-4.6.2 ntlm-auth-1.5.0 ply-3.11 pybind11-2.6.2 python-constraint-1.4.0 qiskit-aer-0.7.4 qiskit-aqua-0.8.2 qiskit-ibmq-provider-0.11.1 qiskit-ignis-0.5.2 qiskit-terra-0.16.4 quandl-3.6.0 requests-ntlm-1.1.0 retworkx-0.7.2 sympy-1.7.1 websockets-8.1 yfinance-0.1.55

```

```


# Merge

**Prerequisites**

- [Reshape](https://datascience.quantecon.org/reshape.html)

**Outcomes**

- Know the different pandas routines for combining datasets
- Know when to use `pd.concat` vs `pd.merge` vs `pd.join`
- Be able to apply the three main combining routines

**Data**

- WDI data on GDP components, population, and square miles of countries
- Book ratings: 6,000,000 ratings for the 10,000 most rated books on
  [Goodreads](https://www.goodreads.com/)
- Details for all delayed US domestic flights in November 2016,
  obtained from the [Bureau of Transportation
  Statistics](https://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236&DB_Short_Name=On-Time)

## Outline

- [Merge](#Merge)
  - [Combining Datasets](#Combining-Datasets)
  - [`pd.concat`](#`pd.concat`)
  - [`pd.merge`](#`pd.merge`)
  - [Arguments to `merge`](#Arguments-to-`merge`)
  - [`df.join`](#`df.join`)
  - [Case Study](#Case-Study)
  - [Extra Example: Airline Delays](#Extra-Example:-Airline-Delays)
  - [Visualizing Merge Operations](#Visualizing-Merge-Operations)
  - [Exercises](#Exercises)

```python
# Uncomment following line to install on colab
! pip install qeds
```

    Requirement already satisfied: qeds in c:\users\asus\anaconda3\lib\site-packages (0.6.2)
    Requirement already satisfied: pyarrow in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.16.0)
    Requirement already satisfied: scipy in c:\users\asus\anaconda3\lib\site-packages (from qeds) (1.3.1)
    Requirement already satisfied: requests in c:\users\asus\anaconda3\lib\site-packages (from qeds) (2.22.0)
    Requirement already satisfied: seaborn in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.9.0)
    Requirement already satisfied: pandas in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.25.1)
    Requirement already satisfied: quandl in c:\users\asus\anaconda3\lib\site-packages (from qeds) (3.5.0)
    Requirement already satisfied: matplotlib in c:\users\asus\anaconda3\lib\site-packages (from qeds) (3.1.1)
    Requirement already satisfied: scikit-learn in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.21.3)
    Requirement already satisfied: statsmodels in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.10.1)
    Requirement already satisfied: pandas-datareader in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.8.1)
    Requirement already satisfied: openpyxl in c:\users\asus\anaconda3\lib\site-packages (from qeds) (3.0.0)
    Requirement already satisfied: plotly in c:\users\asus\anaconda3\lib\site-packages (from qeds) (4.5.4)
    Requirement already satisfied: numpy in c:\users\asus\anaconda3\lib\site-packages (from qeds) (1.16.5)
    Requirement already satisfied: quantecon in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.4.6)
    Requirement already satisfied: six>=1.0.0 in c:\users\asus\anaconda3\lib\site-packages (from pyarrow->qeds) (1.12.0)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (1.24.2)
    Requirement already satisfied: certifi>=2017.4.17 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (2019.9.11)
    Requirement already satisfied: idna<2.9,>=2.5 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (2.8)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (3.0.4)
    Requirement already satisfied: python-dateutil>=2.6.1 in c:\users\asus\anaconda3\lib\site-packages (from pandas->qeds) (2.8.0)
    Requirement already satisfied: pytz>=2017.2 in c:\users\asus\anaconda3\lib\site-packages (from pandas->qeds) (2019.3)
    Requirement already satisfied: inflection>=0.3.1 in c:\users\asus\anaconda3\lib\site-packages (from quandl->qeds) (0.3.1)
    Requirement already satisfied: more-itertools in c:\users\asus\anaconda3\lib\site-packages (from quandl->qeds) (7.2.0)
    Requirement already satisfied: cycler>=0.10 in c:\users\asus\anaconda3\lib\site-packages (from matplotlib->qeds) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\asus\anaconda3\lib\site-packages (from matplotlib->qeds) (1.1.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\users\asus\anaconda3\lib\site-packages (from matplotlib->qeds) (2.4.2)
    Requirement already satisfied: joblib>=0.11 in c:\users\asus\anaconda3\lib\site-packages (from scikit-learn->qeds) (0.13.2)
    Requirement already satisfied: patsy>=0.4.0 in c:\users\asus\anaconda3\lib\site-packages (from statsmodels->qeds) (0.5.1)
    Requirement already satisfied: lxml in c:\users\asus\anaconda3\lib\site-packages (from pandas-datareader->qeds) (4.4.1)
    Requirement already satisfied: jdcal in c:\users\asus\anaconda3\lib\site-packages (from openpyxl->qeds) (1.4.1)
    Requirement already satisfied: et-xmlfile in c:\users\asus\anaconda3\lib\site-packages (from openpyxl->qeds) (1.0.1)
    Requirement already satisfied: retrying>=1.3.3 in c:\users\asus\anaconda3\lib\site-packages (from plotly->qeds) (1.3.3)
    Requirement already satisfied: sympy in c:\users\asus\anaconda3\lib\site-packages (from quantecon->qeds) (1.4)
    Requirement already satisfied: numba>=0.38 in c:\users\asus\anaconda3\lib\site-packages (from quantecon->qeds) (0.45.1)
    Requirement already satisfied: setuptools in c:\users\asus\anaconda3\lib\site-packages (from kiwisolver>=1.0.1->matplotlib->qeds) (41.4.0)
    Requirement already satisfied: mpmath>=0.19 in c:\users\asus\anaconda3\lib\site-packages (from sympy->quantecon->qeds) (1.1.0)
    Requirement already satisfied: llvmlite>=0.29.0dev0 in c:\users\asus\anaconda3\lib\site-packages (from numba>=0.38->quantecon->qeds) (0.29.0)

## Combining Datasets

Often, we will want perform joint analysis on data from different sources.

For example, when analyzing the regional sales for a company, we might
want to include industry aggregates or demographic information for each
region.

Or perhaps we are working with product-level data, have a list of
product groups in a separate dataset, and want to compute aggregate
statistics for each group.

```python
import pandas as pd
import qeds
%matplotlib inline
# activate plot theme
import qeds
qeds.themes.mpl_style();

from IPython.display import display
```

```python
# from WDI. Units trillions of 2010 USD
url = "https://datascience.quantecon.org/assets/data/wdi_data.csv"
wdi = pd.read_csv(url).set_index(["country", "year"])
wdi.info()

wdi2017 = wdi.xs(2017, level="year")
wdi2017
```

    <class 'pandas.core.frame.DataFrame'>
    MultiIndex: 72 entries, (Canada, 2017) to (United States, 2000)
    Data columns (total 5 columns):
    GovExpend      72 non-null float64
    Consumption    72 non-null float64
    Exports        72 non-null float64
    Imports        72 non-null float64
    GDP            72 non-null float64
    dtypes: float64(5)
    memory usage: 3.2+ KB

      GovExpend
      Consumption
      Exports
      Imports
      GDP

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627

```python
wdi2016_17 = wdi.loc[pd.IndexSlice[:, [2016, 2017]],: ]
wdi2016_17
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP

      country
      year

      Canada
      2017
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164

      2016
      0.364899
      1.058426
      0.576394
      0.575775
      1.814016

      Germany
      2017
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870

      2016
      0.734014
      2.075615
      1.844949
      1.589495
      3.801859

      United Kingdom
      2017
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704

      2016
      0.550596
      1.772348
      0.816792
      0.901494
      2.768241

      United States
      2017
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627

      2016
      2.407981
      11.722133
      2.219937
      2.936004
      16.972348

```python
# Data from https://www.nationmaster.com/country-info/stats/Geography/Land-area/Square-miles
# units -- millions of square miles
sq_miles = pd.Series({
   "United States": 3.8,
   "Canada": 3.8,
   "Germany": 0.137,
   "United Kingdom": 0.0936,
   "Russia": 6.6,
}, name="sq_miles").to_frame()
sq_miles.index.name = "country"
sq_miles
```

      sq_miles

      country

      United States
      3.8000

      Canada
      3.8000

      Germany
      0.1370

      United Kingdom
      0.0936

      Russia
      6.6000

```python
# from WDI. Units millions of people
pop_url = "https://datascience.quantecon.org/assets/data/wdi_population.csv"
pop = pd.read_csv(pop_url).set_index(["country", "year"])
pop.info()
pop.head(10)
```

    <class 'pandas.core.frame.DataFrame'>
    MultiIndex: 72 entries, (Canada, 2017) to (United States, 2000)
    Data columns (total 1 columns):
    Population    72 non-null float64
    dtypes: float64(1)
    memory usage: 1005.0+ bytes

      Population

      country
      year

      Canada
      2017
      36.540268

      2016
      36.109487

      2015
      35.702908

      2014
      35.437435

      2013
      35.082954

      2012
      34.714222

      2011
      34.339328

      2010
      34.004889

      2009
      33.628895

      2008
      33.247118

Suppose that we were asked to compute a number of statistics with the data above:

- As a measure of land usage or productivity, what is Consumption per square mile?
- What is GDP per capita (per person) for each country in each year? How about
  Consumption per person?
- What is the population density of each country? How much does it change over time?

Notice that to answer any of the questions from above, we will have to use data
from more than one of our DataFrames.

In this lecture, we will learn many techniques for combining datasets that
originate from different sources, careful to ensure that data is properly
aligned.

In pandas three main methods can combine datasets:

1. `pd.concat([dfs...])`
1. `pd.merge(df1, df2)`
1. `df1.join(df2)`

We’ll look at each one.

## `pd.concat`

The `pd.concat` function is used to stack two or more DataFrames
together.

An example of when you might want to do this is if you have monthly data
in separate files on your computer and would like to have 1 year of data
in a single DataFrame.

The first argument to `pd.concat` is a list of DataFrames to be
stitched together.

The other commonly used argument is named `axis`.

As we have seen before, many pandas functions have an `axis` argument
that specifies whether a particular operation should happen down rows
(`axis=0`) or along columns (`axis=1`).

In the context of `pd.concat`, setting `axis=0` (the default case)
will stack DataFrames on top of one another while `axis=1` stacks them
side by side.

We’ll look at each case separately.

### `axis=0`

When we call `pd.concat` and set `axis=0`, the list of DataFrames
passed in the first argument will be stacked on top of one another.

Let’s try it out here.

```python
# equivalent to pd.concat([wdi2017, sq_miles]) -- axis=0 is default
pd.concat([wdi2017, sq_miles], axis=0)
```

    C:\Users\asus\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
    of pandas will change to not sort by default.

    To accept the future behavior, pass 'sort=False'.

    To retain the current behavior and silence the warning, pass 'sort=True'.

      Consumption
      Exports
      GDP
      GovExpend
      Imports
      sq_miles

      country

      Canada
      1.095475
      0.582831
      1.868164
      0.372665
      0.600031
      NaN

      Germany
      2.112009
      1.930563
      3.883870
      0.745579
      1.666348
      NaN

      United Kingdom
      1.809154
      0.862629
      2.818704
      0.549538
      0.933145
      NaN

      United States
      12.019266
      2.287071
      17.348627
      2.405743
      3.069954
      NaN

      United States
      NaN
      NaN
      NaN
      NaN
      NaN
      3.8000

      Canada
      NaN
      NaN
      NaN
      NaN
      NaN
      3.8000

      Germany
      NaN
      NaN
      NaN
      NaN
      NaN
      0.1370

      United Kingdom
      NaN
      NaN
      NaN
      NaN
      NaN
      0.0936

      Russia
      NaN
      NaN
      NaN
      NaN
      NaN
      6.6000

Notice a few things:

-

The number of rows in the output is the total number

  of rows in all inputs. The labels are all from the original
  DataFrames.

- The column labels are all the distinct column labels from all the inputs.
- For columns that appeared only in one input, the value for all row labels
  originating from a different input is equal to `NaN` (marked as missing).

### `axis=1`

In this example, concatenating by stacking
side-by-side makes more sense.

We accomplish this by passing `axis=1` to `pd.concat`:

```python
pd.concat([wdi2017, sq_miles], axis=1)
```

    C:\Users\asus\Anaconda3\lib\site-packages\ipykernel_launcher.py:1: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
    of pandas will change to not sort by default.

    To accept the future behavior, pass 'sort=False'.

    To retain the current behavior and silence the warning, pass 'sort=True'.

      """Entry point for launching an IPython kernel.

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      Russia
      NaN
      NaN
      NaN
      NaN
      NaN
      6.6000

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

Notice here that

- The index entries are all unique index entries that appeared in any DataFrame.
- The column labels are all column labels from the inputs.
- As `wdi2017` didn’t have a `Russia` row, the value for all of its columns
  is `NaN`.

Now we can answer one of our questions from above: What is
Consumption per square mile?

```python
temp = pd.concat([wdi2017, sq_miles], axis=1)
temp["Consumption"] / temp["sq_miles"]
```

    C:\Users\asus\Anaconda3\lib\site-packages\ipykernel_launcher.py:1: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
    of pandas will change to not sort by default.

    To accept the future behavior, pass 'sort=False'.

    To retain the current behavior and silence the warning, pass 'sort=True'.

      """Entry point for launching an IPython kernel.

    Canada             0.288283
    Germany           15.416124
    Russia                  NaN
    United Kingdom    19.328569
    United States      3.162965
    dtype: float64

## `pd.merge`

`pd.merge` operates on two DataFrames at a time and is primarily used
to bring columns from one DataFrame into another, *aligning* data based
on one or more “key” columns.

This is a somewhat difficult concept to grasp by reading, so let’s look at some
examples.

```python
pd.merge(wdi2017, sq_miles, on="country")
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

The output here looks very similar to what we saw with `concat` and
`axis=1`, except that the row for `Russia` does not appear.

We will talk more about why this happened soon.

For now, let’s look at a slightly more intriguing example:

```python
pd.merge(wdi2016_17, sq_miles, on="country")
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Canada
      0.364899
      1.058426
      0.576394
      0.575775
      1.814016
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      Germany
      0.734014
      2.075615
      1.844949
      1.589495
      3.801859
      0.1370

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United Kingdom
      0.550596
      1.772348
      0.816792
      0.901494
      2.768241
      0.0936

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

      United States
      2.407981
      11.722133
      2.219937
      2.936004
      16.972348
      3.8000

Here’s how we think about what happened:

- The data in `wdi2016_17` is copied over exactly as is.
- Because `country` was on the index for both DataFrames, it is on the
  index of the output.
- We lost the year on the index – we’ll work on getting it back below.
- The additional column in `sq_miles` was added to column labels for the
  output.
- The data from the `sq_miles` column was added to the output by looking up
  rows where the `country` in the two DataFrames lined up.
  -  Note that all the countries appeared twice, and the data in `sq_miles` was repeated. This is because `wdi2016_17` had two rows for each country.
  -  Also note that because `Russia` did not appear in `wdi2016_17`, the value `sq_miles.loc["Russia"]` (i.e. `6.6`) is not used the output.

How do we get the year back?

We must first call `reset_index` on `wdi2016_17` so
that in the first step when all columns are copied over, `year` is included.

```python
pd.merge(wdi2016_17.reset_index(), sq_miles, on="country")
```

      country
      year
      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      0
      Canada
      2017
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      1
      Canada
      2016
      0.364899
      1.058426
      0.576394
      0.575775
      1.814016
      3.8000

      2
      Germany
      2017
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      3
      Germany
      2016
      0.734014
      2.075615
      1.844949
      1.589495
      3.801859
      0.1370

      4
      United Kingdom
      2017
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      5
      United Kingdom
      2016
      0.550596
      1.772348
      0.816792
      0.901494
      2.768241
      0.0936

      6
      United States
      2017
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

      7
      United States
      2016
      2.407981
      11.722133
      2.219937
      2.936004
      16.972348
      3.8000

### Multiple Columns

Sometimes, we need to merge multiple columns.

For example our `pop` and `wdi2016_17` DataFrames both have observations
organized by country and year.

To properly merge these datasets, we would need to align the data by
both country and year.

We pass a list to the `on` argument to accomplish this:

```python
pd.merge(wdi2016_17, pop, on=["country", "year"])
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      Population

      country
      year

      Canada
      2017
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      36.540268

      2016
      0.364899
      1.058426
      0.576394
      0.575775
      1.814016
      36.109487

      Germany
      2017
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      82.657002

      2016
      0.734014
      2.075615
      1.844949
      1.589495
      3.801859
      82.348669

      United Kingdom
      2017
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      66.058859

      2016
      0.550596
      1.772348
      0.816792
      0.901494
      2.768241
      65.595565

      United States
      2017
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      325.147121

      2016
      2.407981
      11.722133
      2.219937
      2.936004
      16.972348
      323.071342

Now, we can answer more of our questions from above: What is GDP per capita (per
person) for each country in each year? How about Consumption per person?

```python
wdi_pop = pd.merge(wdi2016_17, pop, on=["country", "year"])
wdi_pop["GDP"] / wdi_pop["Population"]
```

    country         year
    Canada          2017    0.051126
                    2016    0.050237
    Germany         2017    0.046988
                    2016    0.046168
    United Kingdom  2017    0.042670
                    2016    0.042202
    United States   2017    0.053356
                    2016    0.052534
    dtype: float64

```python
wdi_pop["Consumption"] / wdi_pop["Population"]
```

    country         year
    Canada          2017    0.029980
                    2016    0.029312
    Germany         2017    0.025551
                    2016    0.025205
    United Kingdom  2017    0.027387
                    2016    0.027019
    United States   2017    0.036966
                    2016    0.036283
    dtype: float64

## Arguments to `merge`

The `pd.merge` function can take many optional arguments.

We’ll talk about a few of the most commonly-used ones here and refer you
to the
[documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.merge.html#pandas.merge)
for more details.

We’ll follow the pandas convention and refer to the first argument to
`pd.merge` as `left` and call the second `right`.

### `on`

We have already seen this one used before, but we want to point out that on
is optional.

If nothing is given for this argument, pandas will use **all** columns
in `left` and `right` with the same name.

In our example, `country` is the only column that appears in both
DataFrames, so it is used for `on` if we don’t pass anything.

The following two are equivalent.

```python
pd.merge(wdi2017, sq_miles, on="country")
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

```python
# if we move index back to columns, the `on` is un-necessary
pd.merge(wdi2017.reset_index(), sq_miles.reset_index())
```

      country
      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      0
      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      1
      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      2
      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      3
      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

### `left_on`, `right_on`

Above, we used the `on` argument to identify a column in both `left`
and `right` that was used to align data.

Sometimes, both DataFrames don’t have the same name for this column.

In that case, we use the `left_on` and `right_on` arguments, passing
the proper column name(s) to align the data.

We’ll show you an example below, but it is somewhat silly as our
DataFrames do both have the `country` column.

```python
pd.merge(wdi2017, sq_miles, left_on="country", right_on="country")
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

### `left_index`, `right_index`

Sometimes, as in our example, the key used to align data is actually in the
index instead of one of the columns.

In this case, we can use the `left_index` or `right_index` arguments.

We should only set these values to a boolean (`True` or `False`).

Let’s practice with this.

```python
pd.merge(wdi2017, sq_miles, left_on="country", right_index=True)
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

### `how`

The `how` is perhaps the most powerful, but most conceptually
difficult of the arguments we will cover.

This argument controls which values from the key column(s) appear in the
output.

The 4 possible options for this argument are summarized in
the image below.

In words, we have:

- `left`: Default and what we described above. It uses
  the keys from the `left` DataFrame.
- `right`: Output will contain all keys from `right`.
- `inner`: The output will only contain keys that appear in *both*
  `left` and `right`.
- `outer`: The output will contain any key found in either `left`
  or `right`.

In addition to the above, we will use the following two DataFrames to
illustrate the `how` option.

```python
wdi2017_no_US = wdi2017.drop("United States")
wdi2017_no_US
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704

```python
sq_miles_no_germany = sq_miles.drop("Germany")
sq_miles_no_germany
```

      sq_miles

      country

      United States
      3.8000

      Canada
      3.8000

      United Kingdom
      0.0936

      Russia
      6.6000

Now, let’s see all the possible `how` options.

```python
# default
pd.merge(wdi2017, sq_miles, on="country", how="left")
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

```python
# notice ``Russia`` is included
pd.merge(wdi2017, sq_miles, on="country", how="right")
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

      Russia
      NaN
      NaN
      NaN
      NaN
      NaN
      6.6000

```python
# notice no United States or Russia
pd.merge(wdi2017_no_US, sq_miles, on="country", how="inner")
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

```python
# includes all 5, even though they don't all appear in either DataFrame
pd.merge(wdi2017_no_US, sq_miles_no_germany, on="country", how="outer")
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      NaN

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United States
      NaN
      NaN
      NaN
      NaN
      NaN
      3.8000

      Russia
      NaN
      NaN
      NaN
      NaN
      NaN
      6.6000

### `df.merge(df2)`

Note that the DataFrame type has a `merge` *method*.

It is the same as the function we have been working with, but passes the
DataFrame before the period as `left`.

Thus `df.merge(other)` is equivalent to `pd.merge(df, other)`.

```python
wdi2017.merge(sq_miles, on="country", how="right")
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

      Russia
      NaN
      NaN
      NaN
      NaN
      NaN
      6.6000

## `df.join`

The `join` method for a DataFrame is very similar to the `merge`
method described above, but only allows you to use the index of the
`right` DataFrame as the join key.

Thus, `left.join(right, on="country")` is equivalent to calling
`pd.merge(left, right, left_on="country", right_index=True)`.

The implementation of the `join` method calls `merge` internally,
but sets the `left_on` and `right_index` arguments for you.

You can do anything with `df.join` that you can do with
`df.merge`, but df.join` is more convenient to use if the keys of `right`
are in the index.

```python
wdi2017.join(sq_miles, on="country")
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

```python
wdi2017.merge(sq_miles, left_on="country", right_index=True)
```

      GovExpend
      Consumption
      Exports
      Imports
      GDP
      sq_miles

      country

      Canada
      0.372665
      1.095475
      0.582831
      0.600031
      1.868164
      3.8000

      Germany
      0.745579
      2.112009
      1.930563
      1.666348
      3.883870
      0.1370

      United Kingdom
      0.549538
      1.809154
      0.862629
      0.933145
      2.818704
      0.0936

      United States
      2.405743
      12.019266
      2.287071
      3.069954
      17.348627
      3.8000

## Case Study

Let’s put these tools to practice by loading some real datasets and
seeing how these functions can be applied.

We’ll analyze ratings of books from the website [Goodreads](https://www.goodreads.com/).

We accessed the data [here](https://github.com/zygmuntz/goodbooks-10k).

Let’s load it up.

```python
ratings = qeds.data.load("goodreads_ratings")
display(ratings.head())
ratings.info()
```

      user_id
      book_id
      rating

      0
      1
      258
      5

      1
      2
      4081
      4

      2
      2
      260
      5

      3
      2
      9296
      5

      4
      2
      2318
      3

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 5976479 entries, 0 to 5976478
    Data columns (total 3 columns):
    user_id    int64
    book_id    int64
    rating     int64
    dtypes: int64(3)
    memory usage: 136.8 MB

We can already do some interesting things with just the ratings data.

Let’s see how many ratings of each number are in our dataset.

```python
ratings["rating"].value_counts().sort_index().plot(kind="bar");
```

Let’s also see how many users have rated `N` books, for all `N`
possible.

To do this, we will use `value_counts` twice (can you think of why?).

We will see a more flexible way of performing similar grouped operations in
a future lecture.

```python
users_by_n = (
    ratings["user_id"]
    .value_counts()  # Series. Index: user_id, value: n ratings by user
    .value_counts()  # Series. Index: n_ratings by user, value: N_users with this many ratings
    .sort_index()    # Sort our Series by the index (number of ratings)
    .reset_index()   # Dataframe with columns `index` (from above) and `user_id`
    .rename(columns={"index": "N_ratings", "user_id": "N_users"})
)
users_by_n.head(10)
```

      N_ratings
      N_users

      0
      19
      1

      1
      20
      1

      2
      21
      3

      3
      22
      13

      4
      23
      5

      5
      24
      11

      6
      25
      13

      7
      26
      23

      8
      27
      34

      9
      28
      26

Let’s look at some statistics on that dataset.

```python
users_by_n.describe()
```

      N_ratings
      N_users

      count
      181.00000
      181.000000

      mean
      109.01105
      295.160221

      std
      52.41342
      309.461848

      min
      19.00000
      1.000000

      25%
      64.00000
      40.000000

      50%
      109.00000
      158.000000

      75%
      154.00000
      538.000000

      max
      200.00000
      964.000000

We can see the same data visually in a box plot.

```python
users_by_n.plot(kind="box", subplots=True)
```

Let’s practice applying the want operator…

**Want**: Determine whether a relationship between the number of
ratings a user has written and the distribution of the ratings exists. (Maybe we
are an author hoping to inflate our ratings and wonder if we should
target “more experienced” Goodreads users, or focus on newcomers.)

Let’s start from the result and work our way backwards:

1. We can answer our question if we have two similar DataFrames:
  - All ratings by the `N` (e.g. 5) users with the most ratings
  - All ratings by the `N` users with the least number of
    ratings
1. To get that, we will need to extract rows of `ratings` with
  `user_id` associated with the `N` most and least prolific raters
1. For that, we need the most and least active `user_id`s
1. To get that info, we need a count of how many ratings each user left.
  - We can get that with `df["user_id"].value_counts()`, so let’s
    start there.

```python
# step 4
n_ratings = ratings["user_id"].value_counts()
n_ratings.head()
```

    30944    200
    12874    200
    52036    199
    28158    199
    12381    199
    Name: user_id, dtype: int64

```python
# step 3
N = 5
most_prolific_users = n_ratings.nlargest(5).index.tolist()
least_prolific_users = n_ratings.nsmallest(5).index.tolist()
```

```python
# step 2
active_ratings = ratings.loc[ratings["user_id"].isin(most_prolific_users), :]
inactive_ratings = ratings.loc[ratings["user_id"].isin(least_prolific_users), :]
```

```python
# step 1 -- get the answer!
active_ratings["rating"].value_counts().sort_index().plot(
    kind="bar", title="Distribution of ratings by most active users"
)
```

```python
inactive_ratings["rating"].value_counts().sort_index().plot(
    kind="bar", title="Distribution of ratings by least active users"
)
```

Nice! From the picture above, the new users look much more
likely to leave 5 star ratings than more experienced users.

### Book Data

We know what you are probably thinking: “Isn’t this a lecture on merging?
Why are we only using one dataset?”

We hear you.

Let’s also load a dataset containing information on the actual books.

```python
books = qeds.data.load("goodreads_books")
# we only need a few of the columns
books = books[["book_id", "authors", "title"]]
print("shape: ", books.shape)
print("dtypes:\n", books.dtypes, sep="")
books.head()
```

    shape:  (10000, 3)
    dtypes:
    book_id     int64
    authors    object
    title      object
    dtype: object

      book_id
      authors
      title

      0
      1
      Suzanne Collins
      The Hunger Games (The Hunger Games, #1)

      1
      2
      J.K. Rowling, Mary GrandPré
      Harry Potter and the Sorcerer's Stone (Harry P...

      2
      3
      Stephenie Meyer
      Twilight (Twilight, #1)

      3
      4
      Harper Lee
      To Kill a Mockingbird

      4
      5
      F. Scott Fitzgerald
      The Great Gatsby

We could do similar interesting things with just the books dataset,
but we will skip it for now and merge them together.

```python
rated_books = pd.merge(ratings, books)
```

Now, let’s see which books have been most often rated.

```python
most_rated_books_id = rated_books["book_id"].value_counts().nlargest(10).index
most_rated_books = rated_books.loc[rated_books["book_id"].isin(most_rated_books_id), :]
list(most_rated_books["title"].unique())
```

    ['Harry Potter and the Prisoner of Azkaban (Harry Potter, #3)',
     "Harry Potter and the Sorcerer's Stone (Harry Potter, #1)",
     'Harry Potter and the Chamber of Secrets (Harry Potter, #2)',
     'The Great Gatsby',
     'To Kill a Mockingbird',
     'The Hobbit',
     'Twilight (Twilight, #1)',
     'The Hunger Games (The Hunger Games, #1)',
     'Catching Fire (The Hunger Games, #2)',
     'Mockingjay (The Hunger Games, #3)']

Let’s use our `pivot_table` knowledge to compute the average rating
for each of these books.

```python
most_rated_books.pivot_table(values="rating", index="title")
```

      rating

      title

      Catching Fire (The Hunger Games, #2)
      4.133422

      Harry Potter and the Chamber of Secrets (Harry Potter, #2)
      4.229418

      Harry Potter and the Prisoner of Azkaban (Harry Potter, #3)
      4.418732

      Harry Potter and the Sorcerer's Stone (Harry Potter, #1)
      4.351350

      Mockingjay (The Hunger Games, #3)
      3.853131

      The Great Gatsby
      3.772224

      The Hobbit
      4.148477

      The Hunger Games (The Hunger Games, #1)
      4.279707

      To Kill a Mockingbird
      4.329369

      Twilight (Twilight, #1)
      3.214341

These ratings seem surprisingly low, given that they are the most often
rated books on Goodreads.

I wonder what the bottom of the distribution looks like…

Let’s compute the average number of ratings for each book in our sample.

```python
average_ratings = (
    rated_books
    .pivot_table(values="rating", index="title")
    .sort_values(by="rating", ascending=False)
)
average_ratings.head(10)
```

      rating

      title

      The Complete Calvin and Hobbes
      4.829876

      ESV Study Bible
      4.818182

      Attack of the Deranged Mutant Killer Monster Snow Goons
      4.768707

      The Indispensable Calvin and Hobbes
      4.766355

      The Revenge of the Baby-Sat
      4.761364

      There's Treasure Everywhere: A Calvin and Hobbes Collection
      4.760456

      The Authoritative Calvin and Hobbes: A Calvin and Hobbes Treasury
      4.757202

      It's a Magical World: A Calvin and Hobbes Collection
      4.747396

      Harry Potter Boxed Set, Books 1-5 (Harry Potter, #1-5)
      4.736842

      The Calvin and Hobbes Tenth Anniversary Book
      4.728528

What does the overall distribution of average ratings look like?

```python
# plot a kernel density estimate of average ratings
average_ratings.plot.density(xlim=(1, 5))

# or a histogram
average_ratings.plot.hist(bins=30, xlim=(1, 5))
```

It looks like most books have an average rating of just below 4.

## Extra Example: Airline Delays

Let’s look at one more example.

This time, we will use a dataset from the [Bureau of Transportation
Statistics](https://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236&DB_Short_Name=On-Time)
that describes the cause of all US domestic flight delays
in November 2016:

```python
air_perf = qeds.load("airline_performance_dec16")[["CRSDepTime", "Carrier", "CarrierDelay", "ArrDelay"]]
air_perf.info()
air_perf.head()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 460949 entries, 0 to 460948
    Data columns (total 4 columns):
    CRSDepTime      460949 non-null datetime64[ns]
    Carrier         460949 non-null object
    CarrierDelay    460949 non-null float64
    ArrDelay        452229 non-null float64
    dtypes: datetime64[ns](1), float64(2), object(1)
    memory usage: 14.1+ MB

      CRSDepTime
      Carrier
      CarrierDelay
      ArrDelay

      0
      2016-12-18 15:58:00
      AA
      0.0
      20.0

      1
      2016-12-19 15:58:00
      AA
      0.0
      20.0

      2
      2016-12-20 15:58:00
      AA
      0.0
      -3.0

      3
      2016-12-21 15:58:00
      AA
      0.0
      -10.0

      4
      2016-12-22 15:58:00
      AA
      0.0
      -8.0

The `Carrier` column identifies the airline and the `CarrierDelay`
reports the number of minutes of the total delay assigned as the
“carrier’s fault”.

**Want**: Determine which airlines, on average, contribute most to
delays.

We can do this using `pivot_table`:

```python
avg_delays = (
    air_perf
    .pivot_table(index="Carrier", values="CarrierDelay", aggfunc="mean")
    .sort_values("CarrierDelay")
    .nlargest(10, "CarrierDelay")
)
avg_delays
```

      CarrierDelay

      Carrier

      F9
      7.856566

      EV
      7.125663

      OO
      6.705469

      B6
      5.588006

      DL
      4.674957

      HA
      4.577753

      UA
      4.368148

      NK
      4.166264

      AA
      4.073358

      VX
      3.342923

The one issue with this dataset is that we don’t know what all those two
letter carrier codes are!

Thankfully, we have a second dataset that maps the two letter code
into the full airline name.

```python
carrier_code = qeds.load("airline_carrier_codes")
carrier_code.tail()
```

      Description

      Code

      ZW
      Air Wisconsin Airlines Corp (1994 - )

      ZX
      Airbc Ltd. (1990 - 2000)

      ZX
      Air Georgian (2002 - )

      ZY
      Atlantic Gulf Airlines (1985 - 1986)

      ZYZ
      Skyway Aviation Inc. (1960 - 2002)

Let’s merge these names so we know which airlines we should avoid
flying…

```python
avg_delays_w_code = avg_delays.join(carrier_code)
avg_delays_w_code.sort_values("CarrierDelay", ascending=False)
```

      CarrierDelay
      Description

      F9
      7.856566
      Frontier Airlines Inc. (1994 - )

      EV
      7.125663
      ExpressJet Airlines Inc. (2012 - )

      EV
      7.125663
      Atlantic Southeast Airlines (1993 - 2011)

      OO
      6.705469
      SkyWest Airlines Inc. (2003 - )

      B6
      5.588006
      JetBlue Airways (2000 - )

      DL
      4.674957
      Delta Air Lines Inc. (1960 - )

      HA
      4.577753
      Hawaiian Airlines Inc. (1960 - )

      UA
      4.368148
      United Air Lines Inc. (1960 - )

      NK
      4.166264
      Spirit Air Lines (1992 - )

      AA
      4.073358
      American Airlines Inc. (1960 - )

      VX
      3.342923
      Virgin America (2007 - )

      VX
      3.342923
      Aces Airlines (1992 - 2003)

Based on that information, which airlines would you avoid near
the holidays?

## Visualizing Merge Operations

As we did in the [reshape lecture](https://datascience.quantecon.org/reshape.html), we will visualize the
various merge operations using artificial DataFrames.

First, we create some dummy DataFrames.

```python
dfL = pd.DataFrame(
    {"Key": ["A", "B", "A", "C"], "C1":[1, 2, 3, 4], "C2": [10, 20, 30, 40]},
    index=["L1", "L2", "L3", "L4"]
)[["Key", "C1", "C2"]]

print("This is dfL: ")
display(dfL)

dfR = pd.DataFrame(
    {"Key": ["A", "B", "C", "D"], "C3": [100, 200, 300, 400]},
    index=["R1", "R2", "R3", "R4"]
)[["Key", "C3"]]

print("This is dfR:")
display(dfR)
```

    This is dfL:

      Key
      C1
      C2

      L1
      A
      1
      10

      L2
      B
      2
      20

      L3
      A
      3
      30

      L4
      C
      4
      40

    This is dfR:

      Key
      C3

      R1
      A
      100

      R2
      B
      200

      R3
      C
      300

      R4
      D
      400

### `pd.concat`

Recall that calling `pd.concat(..., axis=0)` will stack DataFrames on top of
one another:

```python
pd.concat([dfL, dfR], axis=0)
```

    C:\Users\asus\Anaconda3\lib\site-packages\ipykernel_launcher.py:1: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
    of pandas will change to not sort by default.

    To accept the future behavior, pass 'sort=False'.

    To retain the current behavior and silence the warning, pass 'sort=True'.

      """Entry point for launching an IPython kernel.

      C1
      C2
      C3
      Key

      L1
      1.0
      10.0
      NaN
      A

      L2
      2.0
      20.0
      NaN
      B

      L3
      3.0
      30.0
      NaN
      A

      L4
      4.0
      40.0
      NaN
      C

      R1
      NaN
      NaN
      100.0
      A

      R2
      NaN
      NaN
      200.0
      B

      R3
      NaN
      NaN
      300.0
      C

      R4
      NaN
      NaN
      400.0
      D

Here’s how we might visualize that.

We can also set `axis=1` to stack side by side.

```python
pd.concat([dfL, dfR], axis=1)
```

    C:\Users\asus\Anaconda3\lib\site-packages\ipykernel_launcher.py:1: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
    of pandas will change to not sort by default.

    To accept the future behavior, pass 'sort=False'.

    To retain the current behavior and silence the warning, pass 'sort=True'.

      """Entry point for launching an IPython kernel.

      Key
      C1
      C2
      Key
      C3

      L1
      A
      1.0
      10.0
      NaN
      NaN

      L2
      B
      2.0
      20.0
      NaN
      NaN

      L3
      A
      3.0
      30.0
      NaN
      NaN

      L4
      C
      4.0
      40.0
      NaN
      NaN

      R1
      NaN
      NaN
      NaN
      A
      100.0

      R2
      NaN
      NaN
      NaN
      B
      200.0

      R3
      NaN
      NaN
      NaN
      C
      300.0

      R4
      NaN
      NaN
      NaN
      D
      400.0

Here’s how we might visualize that.

### `pd.merge`

The animation below shows a visualization of what happens when we call

```python
pd.merge(dfL, dfR, on="Key")
```

      Key
      C1
      C2
      C3

      0
      A
      1
      10
      100

      1
      A
      3
      30
      100

      2
      B
      2
      20
      200

      3
      C
      4
      40
      300

Now, let’s focus on what happens when we set `how="right"`.

Pay special attention to what happens when filling the output value for
the key `A`.

```python
pd.merge(dfL, dfR, on="Key", how="right")
```

      Key
      C1
      C2
      C3

      0
      A
      1.0
      10.0
      100

      1
      A
      3.0
      30.0
      100

      2
      B
      2.0
      20.0
      200

      3
      C
      4.0
      40.0
      300

      4
      D
      NaN
      NaN
      400


```python
import numpy as np
import sympy as sp
import compiler
class Primiaries:
    def __init__(self, x):
        self.data_type = type(x).__name__
        self.data = x

    def get_mean(self):
        if self.data_type == "dict":
            x = self.data["x"]
            f = self.data["f"]
            if len(x) != len(f): raise Exception("The length of `x` and `f` should be same")
            n = 0
            sum_xf = 0
            for i in range(len(x)):
                n += f[i]
                sum_xf += f[i] * x[i]
            return sum_xf/n
    def get_median(self):
        if self.data_type == "dict":
            x = self.data["x"]
            f = self.data["f"]
            if len(x) != len(f): raise Exception("The length of `x` and `f` should be same")
            cumulative = 0
            cumulative_list = []
            for frequency in f:
                cumulative += frequency
                cumulative_list.append(cumulative)
            to_minus = (cumulative+1)/2
            distance_list = []
            for value in f:
                distance = abs(value-to_minus)
                distance_list.append(distance)
            minimum_inices = [i for i, e in enumerate(distance_list) if e == min(distance_list)]
            return to_minus

data = {"x":[4,8,12,16,20,24,28],
        "f":[7,18,25,18,15,10,7]
}
my_class = Primiaries(data)
# my_class.get_median()
eq = "x^2+2x"
compiler
```

    <module 'compiler' from 'C:\\Softwares\\Python39\\lib\\site-packages\\compiler\\__init__.py'>

```python
!pip3 install compiler
```

    Collecting compiler
      Downloading compiler-0.2.0-py3-none-any.whl (5.6 kB)
    Installing collected packages: compiler
    Successfully installed compiler-0.2.0

```python

```


##### Copyright 2020 The Cirq Developers

```
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Shor's algorithm

    View on QuantumAI

    Run in Google Colab

    View source on GitHub

    Download notebook

This tutorial presents a pedagogical demonstration of Shor's algorithm. It is a modified and expanded version of [this Cirq example](https://github.com/quantumlib/Cirq/blob/master/examples/shor.py).

```
"""Install Cirq."""
try:
    import cirq
except ImportError:
    print("installing cirq...")
    !pip install --quiet cirq
    print("installed cirq.")
```

```
"""Imports for the notebook."""
import fractions
import math
import random

import numpy as np
import sympy
from typing import Callable, List, Optional, Sequence, Union

import cirq
```

# Order finding

Factoring an integer $n$ can be reduced to finding the period of the modular exponential function (to be defined). Finding this period can be accomplished (with high probability) by finding the order of a randomly chosen element of the multiplicative group modulo $n$.

Let $n$ be a positive integer and

$$ \mathbb{Z}_n := \{x \in \mathbb{Z}_+ : x < n \text{ and } \text{gcd}(x, n) = 1\} $$

be the multiplicative group modulo $n$.
Given $x \in \mathbb{Z}_n$, compute the smallest positive integer $r$ such that $x^r \text{ mod } n = 1$.

It can be shown from group/number theory that:

(1) Such an integer $r$ exists. (Note that $g^{|G|} = 1_G$ for any group $G$ with cardinality $|G|$ and element $g \in G$, but it's possible that $r < |G|$.)

(2) If $n = pq$ for primes $p$ and $q$, then $|\mathbb{Z}_n| = \phi(n) = (p - 1) (q - 1)$. (The function $\phi$ is called [Euler's totient function](https://en.wikipedia.org/wiki/Euler%27s_totient_function).)

(3) The modular exponential function

$$ f_x(z) := x^z \mod n $$

is periodic with period $r$ (the order of the element $x \in \mathbb{Z}_n$). That is, $f_x(z + r) = f_x(z)$.

(4) If we know the period of the modular exponential function, we can (with high probability) figure out $p$ and $q$ -- that is, factor $n$.

As a refresher, we can visualize the elements of some multiplicative groups $\mathbb{Z}_n$ for integers $n$ via the following simple function.

```
"""Function to compute the elements of Z_n."""
def multiplicative_group(n: int) -> List[int]:
    """Returns the multiplicative group modulo n.

    Args:
        n: Modulus of the multiplicative group.
    """
    assert n > 1
    group = [1]
    for x in range(2, n):
        if math.gcd(x, n) == 1:
            group.append(x)
    return group
```

For example, the multiplicative group modulo $n = 15$ is shown below.

```
"""Example of a multiplicative group."""
n = 15
print(f"The multiplicative group modulo n = {n} is:")
print(multiplicative_group(n))
```

One can check that this set of elements indeed forms a group (under ordinary multiplication).

## Classical order finding

A function for classically computing the order $r$ of an element $x \in \mathbb{Z}_n$ is provided below. This function simply computes the sequence

$$ x^2 \text{ mod } n $$
$$ x^3 \text{ mod } n $$
$$ x^4 \text{ mod } n $$
$$ \vdots $$

until an integer $r$ is found such that $x^r = 1 \text{ mod } n$. Since $|\mathbb{Z}_n| = \phi(n)$, this algorithm for order finding has time complexity $O(\phi(n))$ which is inefficient. (Roughly $O(2^{L / 2})$ where $L$ is the number of bits in $n$.)

```
"""Function for classically computing the order of an element of Z_n."""
def classical_order_finder(x: int, n: int) -> Optional[int]:
    """Computes smallest positive r such that x**r mod n == 1.

    Args:
        x: Integer whose order is to be computed, must be greater than one
           and belong to the multiplicative group of integers modulo n (which
           consists of positive integers relatively prime to n),
        n: Modulus of the multiplicative group.

    Returns:
        Smallest positive integer r such that x**r == 1 mod n.
        Always succeeds (and hence never returns None).

    Raises:
        ValueError when x is 1 or not an element of the multiplicative
        group of integers modulo n.
    """
    # Make sure x is both valid and in Z_n.
    if x < 2 or x >= n or math.gcd(x, n) > 1:
        raise ValueError(f"Invalid x={x} for modulus n={n}.")

    # Determine the order.
    r, y = 1, x
    while y != 1:
        y = (x * y) % n
        r += 1
    return r
```

An example of computing $r$ for a given $x \in \mathbb{Z}_n$ and given $n$ is shown in the code block below.

```
"""Example of (classically) computing the order of an element."""
n = 15  # The multiplicative group is [1, 2, 4, 7, 8, 11, 13, 14].
x = 8
r = classical_order_finder(x, n)

# Check that the order is indeed correct.
print(f"x^r mod n = {x}^{r} mod {n} = {x**r % n}")
```

The quantum part of Shor's algorithm is order finding, but done via a quantum circuit, which we'll discuss below.

## Quantum order finding

Quantum order finding is essentially quantum phase estimation with unitary $U$ that computes the modular exponential function $f_x(z)$ for some randomly chosen $x \in \mathbb{Z}_n$. The full details of how $U$ is computed in terms of elementary gates can be complex to unravel, especially on a first reading. In this tutorial, we'll use arithmetic operations in Cirq which can implement such a unitary $U$ without fully delving into the details of elementary gates.

Below we first show an example of a simple arithmetic operation in Cirq (addition) then discuss the operation we care about (modular exponentiation).

### Quantum arithmetic operations in Cirq

Here we discuss an example of defining an arithmetic operation in Cirq, namely modular addition. This operation adds the value of the input register into the target register. More specifically, this operation acts on two qubit registers as

$$ |a\rangle_i |b\rangle_t \mapsto |a\rangle_i |a + b \text{ mod } N_t \rangle_t . $$

Here, the subscripts $i$ and $t$ denote input and target register, respectively, and $N_t$ is the dimension of the target register.

To define this operation, called `Adder`, we inherit from `cirq.ArithmeticOperation` and override the four methods shown below. The main method is the `apply` method which defines the arithmetic. Here, we simply state the expression as $a + b$ instead of the more accurate $a + b \text{ mod } N_t$ above -- the `cirq.ArithmeticOperation` class is able to deduce what we mean by simply $a + b$ since the operation must be reversible.

```
"""Example of defining an arithmetic (quantum) operation in Cirq."""
class Adder(cirq.ArithmeticOperation):
    """Quantum addition."""
    def __init__(self, target_register, input_register):
        self.input_register = input_register
        self.target_register = target_register

    def registers(self):
        return self.target_register, self.input_register

    def with_registers(self, *new_registers):
        return Adder(*new_registers)

    def apply(self, target_value, input_value):
        return target_value + input_value
```

Now that we have the operation defined, we can use it in a circuit. The cell below creates two qubit registers, then sets the first register to be $|10\rangle$ (in binary) and the second register to be $|01\rangle$ (in binary) via $X$ gates. Then, we use the `Adder` operation, then measure all the qubits.

Since $10 + 01 = 11$ (in binary), we expect to measure $|11\rangle$ in the target register every time. Additionally, since we do not alter the input register, we expect to measure $|10\rangle$ in the input register every time. In short, the only bitstring we expect to measure is $1011$.

```
"""Example of using an Adder in a circuit."""
# Two qubit registers.
qreg1 = cirq.LineQubit.range(2)
qreg2 = cirq.LineQubit.range(2, 4)

# Define the circuit.
circ = cirq.Circuit(
    cirq.ops.X.on(qreg1[0]),
    cirq.ops.X.on(qreg2[1]),
    Adder(input_register=qreg1, target_register=qreg2),
    cirq.measure_each(*qreg1),
    cirq.measure_each(*qreg2)
)

# Display it.
print("Circuit:\n")
print(circ)

# Print the measurement outcomes.
print("\n\nMeasurement outcomes:\n")
print(cirq.sample(circ, repetitions=5).data)
```

In the output of this code block, we first see the circuit which shows the initial $X$ gates, the `Adder` operation, then the final measurements. Next, we see the measurement outcomes which are all the bitstring $1011$ as expected.

It is also possible to see the unitary of the adder operation, which we do below. Here, we set the target register to be two qubits in the zero state, i.e. $|00\rangle$. We specify the input register as the integer one which corresponds to the qubit register $|01\rangle$.

```
"""Example of the unitary of an Adder operation."""
cirq.unitary(
    Adder(target_register=cirq.LineQubit.range(2),
          input_register=1)
).real
```

We can understand this unitary as follows. The $i$th column of the unitary is the state $|i + 1 \text{ mod } 4\rangle$. For example, if we look at the $0$th column of the unitary, we see the state $|i + 1 \text{ mod } 4\rangle = |0 + 1 \text{ mod } 4\rangle = |1\rangle$. If we look at the $1$st column of the unitary, we see the state $|i + 1 \text{ mod } 4\rangle = |1 + 1 \text{ mod } 4\rangle = |2\rangle$. Similarly for the last two columns.

### Modular exponential arithmetic operation

We can define the modular exponential arithmetic operation in a similar way to the simple addition arithmetic operation, shown below. For the purposes of understanding Shor's algorithm, the most important part of the following code block is the `apply` method which defines the arithmetic operation.

```
"""Defines the modular exponential operation used in Shor's algorithm."""
class ModularExp(cirq.ArithmeticOperation):
    """Quantum modular exponentiation.

    This class represents the unitary which multiplies base raised to exponent
    into the target modulo the given modulus. More precisely, it represents the
    unitary V which computes modular exponentiation x**e mod n:

        V|y⟩|e⟩ = |y * x**e mod n⟩ |e⟩     0 <= y < n
        V|y⟩|e⟩ = |y⟩ |e⟩                  n <= y

    where y is the target register, e is the exponent register, x is the base
    and n is the modulus. Consequently,

        V|y⟩|e⟩ = (U**e|y)|e⟩

    where U is the unitary defined as

        U|y⟩ = |y * x mod n⟩      0 <= y < n
        U|y⟩ = |y⟩                n <= y
    """
    def __init__(
        self,
        target: Sequence[cirq.Qid],
        exponent: Union[int, Sequence[cirq.Qid]],
        base: int,
        modulus: int
    ) -> None:
        if len(target) < modulus.bit_length():
            raise ValueError(f'Register with {len(target)} qubits is too small '
                             f'for modulus {modulus}')
        self.target = target
        self.exponent = exponent
        self.base = base
        self.modulus = modulus

    def registers(self) -> Sequence[Union[int, Sequence[cirq.Qid]]]:
        return self.target, self.exponent, self.base, self.modulus

    def with_registers(
            self,
            *new_registers: Union[int, Sequence['cirq.Qid']],
    ) -> cirq.ArithmeticOperation:
        if len(new_registers) != 4:
            raise ValueError(f'Expected 4 registers (target, exponent, base, '
                             f'modulus), but got {len(new_registers)}')
        target, exponent, base, modulus = new_registers
        if not isinstance(target, Sequence):
            raise ValueError(
                f'Target must be a qubit register, got {type(target)}')
        if not isinstance(base, int):
            raise ValueError(
                f'Base must be a classical constant, got {type(base)}')
        if not isinstance(modulus, int):
            raise ValueError(
                f'Modulus must be a classical constant, got {type(modulus)}')
        return ModularExp(target, exponent, base, modulus)

    def apply(self, *register_values: int) -> int:
        assert len(register_values) == 4
        target, exponent, base, modulus = register_values
        if target >= modulus:
            return target
        return (target * base**exponent) % modulus

    def _circuit_diagram_info_(
            self,
            args: cirq.CircuitDiagramInfoArgs,
    ) -> cirq.CircuitDiagramInfo:
        assert args.known_qubits is not None
        wire_symbols: List[str] = []
        t, e = 0, 0
        for qubit in args.known_qubits:
            if qubit in self.target:
                if t == 0:
                    if isinstance(self.exponent, Sequence):
                        e_str = 'e'
                    else:
                        e_str = str(self.exponent)
                    wire_symbols.append(
                        f'ModularExp(t*{self.base}**{e_str} % {self.modulus})')
                else:
                    wire_symbols.append('t' + str(t))
                t += 1
            if isinstance(self.exponent, Sequence) and qubit in self.exponent:
                wire_symbols.append('e' + str(e))
                e += 1
        return cirq.CircuitDiagramInfo(wire_symbols=tuple(wire_symbols))
```

In the `apply` method, we see that we evaluate `(target * base**exponent) % modulus`. The `target` and the `exponent` depend on the values of the respective qubit registers, and the `base` and `modulus` are constant -- namely, the `modulus` is $n$ and the `base` is some $x \in \mathbb{Z}_n$.

The total number of qubits we will use is $3 (L + 1)$ where $L$ is the number of bits needed to store the integer $n$ to factor. The size of the unitary which implements the modular exponential is thus $4^{3(L + 1)}$. For a modest $n = 15$, the unitary requires storing $2^{30}$ floating point numbers in memory which is out of reach of most current standard laptops.

```
"""Create the target and exponent registers for phase estimation,
and see the number of qubits needed for Shor's algorithm.
"""
n = 15
L = n.bit_length()

# The target register has L qubits.
target = cirq.LineQubit.range(L)

# The exponent register has 2L + 3 qubits.
exponent = cirq.LineQubit.range(L, 3 * L + 3)

# Display the total number of qubits to factor this n.
print(f"To factor n = {n} which has L = {L} bits, we need 3L + 3 = {3 * L + 3} qubits.")
```

As with the simple adder operation, this modular exponential operation has a unitary which we can display (memory permitting) as follows.

```
"""See (part of) the unitary for a modular exponential operation."""
# Pick some element of the multiplicative group modulo n.
x = 5

# Display (part of) the unitary. Uncomment if n is small enough.
# cirq.unitary(ModularExp(target, exponent, x, n))
```

## Using the modular exponentional operation in a circuit

The quantum part of Shor's algorithm is just phase estimation with the unitary $U$ corresponding to the modular exponential operation. The following cell defines a function which creates the circuit for Shor's algorithm using the `ModularExp` operation we defined above.

```
"""Function to make the quantum circuit for order finding."""
def make_order_finding_circuit(x: int, n: int) -> cirq.Circuit:
    """Returns quantum circuit which computes the order of x modulo n.

    The circuit uses Quantum Phase Estimation to compute an eigenvalue of
    the unitary

        U|y⟩ = |y * x mod n⟩      0 <= y < n
        U|y⟩ = |y⟩                n <= y

    Args:
        x: positive integer whose order modulo n is to be found
        n: modulus relative to which the order of x is to be found

    Returns:
        Quantum circuit for finding the order of x modulo n
    """
    L = n.bit_length()
    target = cirq.LineQubit.range(L)
    exponent = cirq.LineQubit.range(L, 3 * L + 3)
    return cirq.Circuit(
        cirq.X(target[L - 1]),
        cirq.H.on_each(*exponent),
        ModularExp(target, exponent, x, n),
        cirq.qft(*exponent, inverse=True),
        cirq.measure(*exponent, key='exponent'),
    )
```

Using this function, we can visualize the circuit for a given $x$ and $n$ as follows.

```
"""Example of the quantum circuit for period finding."""
n = 15
x = 7
circuit = make_order_finding_circuit(x, n)
print(circuit)
```

As previously described, we put the exponent register into an equal superposition via Hadamard gates. The $X$ gate on the last qubit in the target register is used for phase kickback. The modular exponential operation performs the sequence of controlled unitaries in phase estimation, then we apply the inverse quantum Fourier transform to the exponent register and measure to read out the result.

To illustrate the measurement results, we can sample from a smaller circuit. (Note that in practice we would never run Shor's algorithm with $n = 6$ because it is even. This is just an example to illustrate the measurement outcomes.)

```
"""Measuring Shor's period finding circuit."""
circuit = make_order_finding_circuit(x=5, n=6)
res = cirq.sample(circuit, repetitions=8)

print("Raw measurements:")
print(res)

print("\nInteger in exponent register:")
print(res.data)
```

We interpret each measured bitstring as an integer, but what do these integers tell us? In the next section we look at how to classically post-process to interpret them.

## Classical post-processing

The integer we measure is close to $s / r$ where $r$ is the order of $x \in \mathbb{Z}_n$ and $0 \le s < r$ is an integer. We use the continued fractions algorithm to determine $r$ from $s / r$ then return it if the order finding circuit succeeded, else we return `None`.

```
def process_measurement(result: cirq.TrialResult, x: int, n: int) -> Optional[int]:
    """Interprets the output of the order finding circuit.

    Specifically, it determines s/r such that exp(2πis/r) is an eigenvalue
    of the unitary

        U|y⟩ = |xy mod n⟩  0 <= y < n
        U|y⟩ = |y⟩         n <= y

    then computes r (by continued fractions) if possible, and returns it.

    Args:
        result: trial result obtained by sampling the output of the
            circuit built by make_order_finding_circuit

    Returns:
        r, the order of x modulo n or None.
    """
    # Read the output integer of the exponent register.
    exponent_as_integer = result.data["exponent"][0]
    exponent_num_bits = result.measurements["exponent"].shape[1]
    eigenphase = float(exponent_as_integer / 2**exponent_num_bits)

    # Run the continued fractions algorithm to determine f = s / r.
    f = fractions.Fraction.from_float(eigenphase).limit_denominator(n)

    # If the numerator is zero, the order finder failed.
    if f.numerator == 0:
        return None

    # Else, return the denominator if it is valid.
    r = f.denominator
    if x**r % n != 1:
        return None
    return r
```

The next code block shows an example of creating an order finding circuit, executing it, then using the classical postprocessing function to determine the order. Recall that the quantum part of the algorithm succeeds with some probability. If the order is `None`, try re-running the cell a few times.

```
"""Example of the classical post-processing."""
# Set n and x here
n = 6
x = 5

print(f"Finding the order of x = {x} modulo n = {n}\n")
measurement = cirq.sample(circuit, repetitions=1)
print("Raw measurements:")
print(measurement)

print("\nInteger in exponent register:")
print(measurement.data)

r = process_measurement(measurement, x, n)
print("\nOrder r =", r)
if r is not None:
    print(f"x^r mod n = {x}^{r} mod {n} = {x**r % n}")
```

You should see that the order of $x = 5$ in $\mathbb{Z}_6$ is $r = 2$. Indeed, $5^2 \text{ mod } 6 = 25 \text{ mod } 6 = 1$.

## Quantum order finder

We can now define a streamlined function for the quantum version of order finding using the functions we have previously written. The quantum order finder below creates the circuit, executes it, and processes the measurement result.

```
def quantum_order_finder(x: int, n: int) -> Optional[int]:
    """Computes smallest positive r such that x**r mod n == 1.

    Args:
        x: integer whose order is to be computed, must be greater than one
           and belong to the multiplicative group of integers modulo n (which
           consists of positive integers relatively prime to n),
        n: modulus of the multiplicative group.
    """
    # Check that the integer x is a valid element of the multiplicative group
    # modulo n.
    if x < 2 or n <= x or math.gcd(x, n) > 1:
        raise ValueError(f'Invalid x={x} for modulus n={n}.')

    # Create the order finding circuit.
    circuit = make_order_finding_circuit(x, n)

    # Sample from the order finding circuit.
    measurement = cirq.sample(circuit)

    # Return the processed measurement result.
    return process_measurement(measurement, x, n)
```

This completes our quantum implementation of an order finder, and the quantum part of Shor's algorithm.

# The complete factoring algorithm

We can use this quantum order finder (or the classical order finder) to complete Shor's algorithm. In the following code block, we add a few pre-processing steps which:

(1) Check if $n$ is even,

(2) Check if $n$ is prime,

(3) Check if $n$ is a prime power,

all of which can be done efficiently with a classical computer. Additionally, we add the last necessary post-processing step which uses the order $r$ to compute a non-trivial factor $p$ of $n$. This is achieved by computing $y = x^{r / 2} \text{ mod } n$ (assuming $r$ is even), then computing $p = \text{gcd}(y - 1, n)$.

```
"""Functions for factoring from start to finish."""
def find_factor_of_prime_power(n: int) -> Optional[int]:
    """Returns non-trivial factor of n if n is a prime power, else None."""
    for k in range(2, math.floor(math.log2(n)) + 1):
        c = math.pow(n, 1 / k)
        c1 = math.floor(c)
        if c1**k == n:
            return c1
        c2 = math.ceil(c)
        if c2**k == n:
            return c2
    return None

def find_factor(
    n: int,
    order_finder: Callable[[int, int], Optional[int]] = quantum_order_finder,
    max_attempts: int = 30
) -> Optional[int]:
    """Returns a non-trivial factor of composite integer n.

    Args:
        n: Integer to factor.
        order_finder: Function for finding the order of elements of the
            multiplicative group of integers modulo n.
        max_attempts: number of random x's to try, also an upper limit
            on the number of order_finder invocations.

    Returns:
        Non-trivial factor of n or None if no such factor was found.
        Factor k of n is trivial if it is 1 or n.
    """
    # If the number is prime, there are no non-trivial factors.
    if sympy.isprime(n):
        print("n is prime!")
        return None

    # If the number is even, two is a non-trivial factor.
    if n % 2 == 0:
        return 2

    # If n is a prime power, we can find a non-trivial factor efficiently.
    c = find_factor_of_prime_power(n)
    if c is not None:
        return c

    for _ in range(max_attempts):
        # Choose a random number between 2 and n - 1.
        x = random.randint(2, n - 1)

        # Most likely x and n will be relatively prime.
        c = math.gcd(x, n)

        # If x and n are not relatively prime, we got lucky and found
        # a non-trivial factor.
        if 1 < c < n:
            return c

        # Compute the order r of x modulo n using the order finder.
        r = order_finder(x, n)

        # If the order finder failed, try again.
        if r is None:
            continue

        # If the order r is even, try again.
        if r % 2 != 0:
            continue

        # Compute the non-trivial factor.
        y = x**(r // 2) % n
        assert 1 < y < n
        c = math.gcd(y - 1, n)
        if 1 < c < n:
            return c

    print(f"Failed to find a non-trivial factor in {max_attempts} attempts.")
    return None
```

The function `find_factor` uses the `quantum_order_finder` by default, in which case it is executing Shor's algorithm. As previously mentioned, due to the large memory requirements for classically simulating this circuit, we cannot run Shor's algorithm for $n \ge 15$. However, we can use the classical order finder as a substitute.

```
"""Example of factoring via Shor's algorithm (order finding)."""
# Number to factor
n = 184573

# Attempt to find a factor
p = find_factor(n, order_finder=classical_order_finder)
q = n // p

print("Factoring n = pq =", n)
print("p =", p)
print("q =", q)
```

```
"""Check the answer is correct."""
p * q == n
```