```python
!pip install mxnet
!pip install gluonnlp pandas tqdm
!pip install sentencepiece
!pip install transformers==3.0.2
!pip install torch
```

    Collecting mxnet
      Downloading mxnet-1.9.0-py3-none-manylinux2014_x86_64.whl (47.3 MB)
    [K     |████████████████████████████████| 47.3 MB 1.8 MB/s 
    [?25hCollecting graphviz<0.9.0,>=0.8.1
      Downloading graphviz-0.8.4-py2.py3-none-any.whl (16 kB)
    Requirement already satisfied: numpy<2.0.0,>1.16.0 in /usr/local/lib/python3.7/dist-packages (from mxnet) (1.21.5)
    Requirement already satisfied: requests<3,>=2.20.0 in /usr/local/lib/python3.7/dist-packages (from mxnet) (2.23.0)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet) (1.24.3)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet) (2021.10.8)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet) (3.0.4)
    Installing collected packages: graphviz, mxnet
      Attempting uninstall: graphviz
        Found existing installation: graphviz 0.10.1
        Uninstalling graphviz-0.10.1:
          Successfully uninstalled graphviz-0.10.1
    Successfully installed graphviz-0.8.4 mxnet-1.9.0
    Collecting gluonnlp
      Downloading gluonnlp-0.10.0.tar.gz (344 kB)
    [K     |████████████████████████████████| 344 kB 11.1 MB/s 
    [?25hRequirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (1.3.5)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (4.63.0)
    Requirement already satisfied: numpy>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from gluonnlp) (1.21.5)
    Requirement already satisfied: cython in /usr/local/lib/python3.7/dist-packages (from gluonnlp) (0.29.28)
    Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from gluonnlp) (21.3)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas) (2.8.2)
    Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas) (2018.9)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)
    Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->gluonnlp) (3.0.7)
    Building wheels for collected packages: gluonnlp
      Building wheel for gluonnlp (setup.py) ... [?25l[?25hdone
      Created wheel for gluonnlp: filename=gluonnlp-0.10.0-cp37-cp37m-linux_x86_64.whl size=595740 sha256=87beb3c2084de50a6c64b79314d4490cdf589e47ca61da0cd664106ec4d7faf8
      Stored in directory: /root/.cache/pip/wheels/be/b4/06/7f3fdfaf707e6b5e98b79c041e023acffbe395d78a527eae00
    Successfully built gluonnlp
    Installing collected packages: gluonnlp
    Successfully installed gluonnlp-0.10.0
    Collecting sentencepiece
      Downloading sentencepiece-0.1.96-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
    [K     |████████████████████████████████| 1.2 MB 13.0 MB/s 
    [?25hInstalling collected packages: sentencepiece
    Successfully installed sentencepiece-0.1.96
    Collecting transformers==3.0.2
      Downloading transformers-3.0.2-py3-none-any.whl (769 kB)
    [K     |████████████████████████████████| 769 kB 12.4 MB/s 
    [?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (4.63.0)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (2.23.0)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (1.21.5)
    Collecting tokenizers==0.8.1.rc1
      Downloading tokenizers-0.8.1rc1-cp37-cp37m-manylinux1_x86_64.whl (3.0 MB)
    [K     |████████████████████████████████| 3.0 MB 39.9 MB/s 
    [?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (3.6.0)
    Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (2019.12.20)
    Requirement already satisfied: sentencepiece!=0.1.92 in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (0.1.96)
    Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2) (21.3)
    Collecting sacremoses
      Downloading sacremoses-0.0.49-py3-none-any.whl (895 kB)
    [K     |████████████████████████████████| 895 kB 44.5 MB/s 
    [?25hRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers==3.0.2) (3.0.7)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.0.2) (1.24.3)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.0.2) (2021.10.8)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.0.2) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==3.0.2) (3.0.4)
    Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==3.0.2) (1.1.0)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==3.0.2) (1.15.0)
    Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==3.0.2) (7.1.2)
    Installing collected packages: tokenizers, sacremoses, transformers
    Successfully installed sacremoses-0.0.49 tokenizers-0.8.1rc1 transformers-3.0.2
    Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (1.10.0+cu111)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch) (3.10.0.2)



```python
!pip install qiskit
```

    Collecting qiskit
      Downloading qiskit-0.35.0.tar.gz (13 kB)
    Collecting qiskit-terra==0.20.0
      Downloading qiskit_terra-0.20.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB)
    [K     |████████████████████████████████| 6.5 MB 9.7 MB/s 
    [?25hCollecting qiskit-aer==0.10.3
      Downloading qiskit_aer-0.10.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (18.0 MB)
    [K     |████████████████████████████████| 18.0 MB 97 kB/s 
    [?25hCollecting qiskit-ibmq-provider==0.18.3
      Downloading qiskit_ibmq_provider-0.18.3-py3-none-any.whl (238 kB)
    [K     |████████████████████████████████| 238 kB 46.6 MB/s 
    [?25hCollecting qiskit-ignis==0.7.0
      Downloading qiskit_ignis-0.7.0-py3-none-any.whl (200 kB)
    [K     |████████████████████████████████| 200 kB 49.4 MB/s 
    [?25hRequirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.3->qiskit) (1.21.5)
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.3->qiskit) (1.4.1)
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.23.0)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (1.24.3)
    Collecting requests-ntlm>=1.1.0
      Downloading requests_ntlm-1.1.0-py2.py3-none-any.whl (5.7 kB)
    Collecting websocket-client>=1.0.1
      Downloading websocket_client-1.3.2-py3-none-any.whl (54 kB)
    [K     |████████████████████████████████| 54 kB 2.4 MB/s 
    [?25hRequirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.8.2)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (57.4.0)
    Collecting retworkx>=0.8.0
      Downloading retworkx-0.11.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
    [K     |████████████████████████████████| 1.6 MB 42.4 MB/s 
    [?25hRequirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.0->qiskit) (1.7.1)
    Collecting python-constraint>=1.4
      Downloading python-constraint-1.4.0.tar.bz2 (18 kB)
    Collecting ply>=3.10
      Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
    [K     |████████████████████████████████| 49 kB 5.0 MB/s 
    [?25hRequirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.0->qiskit) (0.3.4)
    Collecting stevedore>=3.0.0
      Downloading stevedore-3.5.0-py3-none-any.whl (49 kB)
    [K     |████████████████████████████████| 49 kB 4.3 MB/s 
    [?25hCollecting tweedledum<2.0,>=1.1
      Downloading tweedledum-1.1.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (943 kB)
    [K     |████████████████████████████████| 943 kB 35.0 MB/s 
    [?25hCollecting symengine>=0.9
      Downloading symengine-0.9.2-cp37-cp37m-manylinux2010_x86_64.whl (37.5 MB)
    [K     |████████████████████████████████| 37.5 MB 23.2 MB/s 
    [?25hCollecting scipy>=1.0
      Downloading scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
    [K     |████████████████████████████████| 38.1 MB 1.4 MB/s 
    [?25hRequirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.0->qiskit) (5.4.8)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2021.10.8)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2.10)
    Collecting ntlm-auth>=1.0.2
      Downloading ntlm_auth-1.5.0-py2.py3-none-any.whl (29 kB)
    Collecting cryptography>=1.3
      Downloading cryptography-36.0.2-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)
    [K     |████████████████████████████████| 3.6 MB 37.9 MB/s 
    [?25hRequirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (2.21)
    Collecting pbr!=2.1.0,>=2.0.0
      Downloading pbr-5.8.1-py2.py3-none-any.whl (113 kB)
    [K     |████████████████████████████████| 113 kB 46.8 MB/s 
    [?25hRequirement already satisfied: importlib-metadata>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.20.0->qiskit) (4.11.3)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.20.0->qiskit) (3.7.0)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.20.0->qiskit) (3.10.0.2)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra==0.20.0->qiskit) (1.2.1)
    Building wheels for collected packages: qiskit, python-constraint
      Building wheel for qiskit (setup.py) ... [?25l[?25hdone
      Created wheel for qiskit: filename=qiskit-0.35.0-py3-none-any.whl size=11839 sha256=36064059005bd9da7d98f6c7762e1e377c80ddee196703322ca9ade0961bad06
      Stored in directory: /root/.cache/pip/wheels/1e/0a/42/93e5cc11795fb0a85fcf6686eac2791def0c9a1cad0a774045
      Building wheel for python-constraint (setup.py) ... [?25l[?25hdone
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24081 sha256=bfc2683bcb2127e2d3dce03dda60659bd97b1eaed7b24a401cdf4c4e577f7fe4
      Stored in directory: /root/.cache/pip/wheels/07/27/db/1222c80eb1e431f3d2199c12569cb1cac60f562a451fe30479
    Successfully built qiskit python-constraint
    Installing collected packages: pbr, tweedledum, symengine, stevedore, scipy, retworkx, python-constraint, ply, ntlm-auth, cryptography, websocket-client, requests-ntlm, qiskit-terra, qiskit-ignis, qiskit-ibmq-provider, qiskit-aer, qiskit
      Attempting uninstall: scipy
        Found existing installation: scipy 1.4.1
        Uninstalling scipy-1.4.1:
          Successfully uninstalled scipy-1.4.1
    [31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.[0m
    Successfully installed cryptography-36.0.2 ntlm-auth-1.5.0 pbr-5.8.1 ply-3.11 python-constraint-1.4.0 qiskit-0.35.0 qiskit-aer-0.10.3 qiskit-ibmq-provider-0.18.3 qiskit-ignis-0.7.0 qiskit-terra-0.20.0 requests-ntlm-1.1.0 retworkx-0.11.0 scipy-1.7.3 stevedore-3.5.0 symengine-0.9.2 tweedledum-1.1.1 websocket-client-1.3.2



```python
from google.colab import drive
drive.mount('/content/drive')
```

    Mounted at /content/drive



```python
!pip install git+https://git@github.com/SKTBrain/KoBERT.git@master
```

    Collecting git+https://****@github.com/SKTBrain/KoBERT.git@master
      Cloning https://****@github.com/SKTBrain/KoBERT.git (to revision master) to /tmp/pip-req-build-yvpq2g0m
      Running command git clone -q 'https://****@github.com/SKTBrain/KoBERT.git' /tmp/pip-req-build-yvpq2g0m
    Collecting boto3
      Downloading boto3-1.21.34-py3-none-any.whl (132 kB)
    [K     |████████████████████████████████| 132 kB 11.7 MB/s 
    [?25hRequirement already satisfied: gluonnlp>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from kobert==0.2.3) (0.10.0)
    Requirement already satisfied: mxnet>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from kobert==0.2.3) (1.9.0)
    Collecting onnxruntime==1.8.0
      Downloading onnxruntime-1.8.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB)
    [K     |████████████████████████████████| 4.5 MB 48.2 MB/s 
    [?25hRequirement already satisfied: sentencepiece>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from kobert==0.2.3) (0.1.96)
    Requirement already satisfied: torch>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from kobert==0.2.3) (1.10.0+cu111)
    Collecting transformers>=4.8.1
      Downloading transformers-4.17.0-py3-none-any.whl (3.8 MB)
    [K     |████████████████████████████████| 3.8 MB 46.4 MB/s 
    [?25hRequirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from onnxruntime==1.8.0->kobert==0.2.3) (3.17.3)
    Requirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.7/dist-packages (from onnxruntime==1.8.0->kobert==0.2.3) (1.21.5)
    Requirement already satisfied: flatbuffers in /usr/local/lib/python3.7/dist-packages (from onnxruntime==1.8.0->kobert==0.2.3) (2.0)
    Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from gluonnlp>=0.6.0->kobert==0.2.3) (21.3)
    Requirement already satisfied: cython in /usr/local/lib/python3.7/dist-packages (from gluonnlp>=0.6.0->kobert==0.2.3) (0.29.28)
    Requirement already satisfied: graphviz<0.9.0,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from mxnet>=1.4.0->kobert==0.2.3) (0.8.4)
    Requirement already satisfied: requests<3,>=2.20.0 in /usr/local/lib/python3.7/dist-packages (from mxnet>=1.4.0->kobert==0.2.3) (2.23.0)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet>=1.4.0->kobert==0.2.3) (2021.10.8)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet>=1.4.0->kobert==0.2.3) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet>=1.4.0->kobert==0.2.3) (1.24.3)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.20.0->mxnet>=1.4.0->kobert==0.2.3) (3.0.4)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=1.7.0->kobert==0.2.3) (3.10.0.2)
    Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.8.1->kobert==0.2.3) (4.63.0)
    Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.8.1->kobert==0.2.3) (2019.12.20)
    Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers>=4.8.1->kobert==0.2.3) (4.11.3)
    Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers>=4.8.1->kobert==0.2.3) (0.0.49)
    Collecting huggingface-hub<1.0,>=0.1.0
      Downloading huggingface_hub-0.5.0-py3-none-any.whl (77 kB)
    [K     |████████████████████████████████| 77 kB 7.0 MB/s 
    [?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers>=4.8.1->kobert==0.2.3) (3.6.0)
    Collecting tokenizers!=0.11.3,>=0.11.1
      Downloading tokenizers-0.11.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB)
    [K     |████████████████████████████████| 6.5 MB 41.4 MB/s 
    [?25hCollecting pyyaml>=5.1
      Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
    [K     |████████████████████████████████| 596 kB 40.8 MB/s 
    [?25hRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->gluonnlp>=0.6.0->kobert==0.2.3) (3.0.7)
    Collecting s3transfer<0.6.0,>=0.5.0
      Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
    [K     |████████████████████████████████| 79 kB 6.6 MB/s 
    [?25hCollecting botocore<1.25.0,>=1.24.34
      Downloading botocore-1.24.34-py3-none-any.whl (8.6 MB)
    [K     |████████████████████████████████| 8.6 MB 31.8 MB/s 
    [?25hCollecting jmespath<2.0.0,>=0.7.1
      Downloading jmespath-1.0.0-py3-none-any.whl (23 kB)
    Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
      Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
    [K     |████████████████████████████████| 127 kB 43.5 MB/s 
    [?25hRequirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.25.0,>=1.24.34->boto3->kobert==0.2.3) (2.8.2)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.25.0,>=1.24.34->boto3->kobert==0.2.3) (1.15.0)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers>=4.8.1->kobert==0.2.3) (3.7.0)
    Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=4.8.1->kobert==0.2.3) (7.1.2)
    Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=4.8.1->kobert==0.2.3) (1.1.0)
    Building wheels for collected packages: kobert
      Building wheel for kobert (setup.py) ... [?25l[?25hdone
      Created wheel for kobert: filename=kobert-0.2.3-py3-none-any.whl size=15674 sha256=5c2a9cb5fffed3a25a3387568d42af92b341c69a5e53c0576b51e9654db64de3
      Stored in directory: /tmp/pip-ephem-wheel-cache-gon6p60e/wheels/d3/68/ca/334747dfb038313b49cf71f84832a33372f3470d9ddfd051c0
    Successfully built kobert
    Installing collected packages: urllib3, jmespath, pyyaml, botocore, tokenizers, s3transfer, huggingface-hub, transformers, onnxruntime, boto3, kobert
      Attempting uninstall: urllib3
        Found existing installation: urllib3 1.24.3
        Uninstalling urllib3-1.24.3:
          Successfully uninstalled urllib3-1.24.3
      Attempting uninstall: pyyaml
        Found existing installation: PyYAML 3.13
        Uninstalling PyYAML-3.13:
          Successfully uninstalled PyYAML-3.13
      Attempting uninstall: tokenizers
        Found existing installation: tokenizers 0.8.1rc1
        Uninstalling tokenizers-0.8.1rc1:
          Successfully uninstalled tokenizers-0.8.1rc1
      Attempting uninstall: transformers
        Found existing installation: transformers 3.0.2
        Uninstalling transformers-3.0.2:
          Successfully uninstalled transformers-3.0.2
    [31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.[0m
    Successfully installed boto3-1.21.34 botocore-1.24.34 huggingface-hub-0.5.0 jmespath-1.0.0 kobert-0.2.3 onnxruntime-1.8.0 pyyaml-6.0 s3transfer-0.5.2 tokenizers-0.11.6 transformers-4.17.0 urllib3-1.25.11



```python
import torch
from torch import nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader
import gluonnlp as nlp
import numpy as np
from tqdm import tqdm, tqdm_notebook
```


```python
from kobert.utils import get_tokenizer
from kobert.pytorch_kobert import get_pytorch_kobert_model
```


```python
from transformers import AdamW
from transformers.optimization import get_cosine_schedule_with_warmup
```


```python
# pytorch 에서 cpu 사용 선택
# cpu 환경에서 모델 구동시
device = torch.device("cuda:0")
```


```python
# 사전 훈련된 kobert model 불러옴
bertmodel, vocab = get_pytorch_kobert_model()
```

    /content/.cache/kobert_v1.zip[██████████████████████████████████████████████████]
    /content/.cache/kobert_news_wiki_ko_cased-1087f8699e.spiece[██████████████████████████████████████████████████]



```python
# 학습에 사용할 데이터 셋 클래스 선언
class BERTDataset(Dataset):
    def __init__(self, dataset, sent_idx, label_idx, bert_tokenizer, max_len,
                 pad, pair):
        transform = nlp.data.BERTSentenceTransform(
            bert_tokenizer, max_seq_length=max_len, pad=pad, pair=pair)

        self.sentences = [transform([i[sent_idx]]) for i in dataset]
        self.labels = [np.int32(i[label_idx]) for i in dataset]

    def __getitem__(self, i):
        return (self.sentences[i] + (self.labels[i], ))

    def __len__(self):
        return (len(self.labels))
```


```python
# 모델 파라미터 설정
# 토큰의 최대 길이라고 생각
max_len = 64
# 몇 개의 샘플들을 예측해보고 가중치를 업데이트 할 지 설정
# 아래와 같이 배치 사이즈가 64인 경우 데이터 64개 마다 예측한 것을 실제 값과 비교한다
batch_size = 64
warmup_ratio = 0.1
# epoch 횟수는 모델이 전체 데이터셋을 훈련시킬 횟수를 의미한다.
num_epochs = 10
max_grad_norm = 1
log_interval = 200
# learning_rate 값이 너무 크면 원하는 값에 도달하기 힘들고, 너무 작으면 학습기간이 오래 걸린다.
learning_rate =  5e-5
```


```python
# 분류에 사용할 모델 클래스 선언
class BERTClassifier(nn.Module):
    def __init__(self,
                 bert,
                 hidden_size = 768,
                 # num_classes는 카테고리의 개수를 의미한다. (현재 데이터셋의 경우 3개의 분류로 데이터셋이 구성되어 있음)
                 num_classes=3,
                 dr_rate=None,
                 params=None):
        super(BERTClassifier, self).__init__()
        self.bert = bert
        self.dr_rate = dr_rate
                 
        self.classifier = nn.Linear(hidden_size , num_classes)
        if dr_rate:
            self.dropout = nn.Dropout(p=dr_rate)
    
    def gen_attention_mask(self, token_ids, valid_length):
        attention_mask = torch.zeros_like(token_ids)
        for i, v in enumerate(valid_length):
            attention_mask[i][:v] = 1
        return attention_mask.float()

    def forward(self, token_ids, valid_length, segment_ids):
        attention_mask = self.gen_attention_mask(token_ids, valid_length)
        
        _, pooler = self.bert(input_ids = token_ids, token_type_ids = segment_ids.long(), attention_mask = attention_mask.float().to(token_ids.device))
        if self.dr_rate:
            out = self.dropout(pooler)
        return self.classifier(out)
```


```python
# to(Device)는 위의 torch.device("cuda:0")를 의미하며 
# GPU에서 학습된 모델을 GPU로 불러올 때 사용한다.
# 또한 GPU로 학습된 모델에 데이터를 제공할 때도 to(Device)를 붙여줘야 함.
# bert모델 불러오기
model = BERTClassifier(bertmodel,  dr_rate=0.5).to(device)
```


```python
# optimizer와 schedule 설정
no_decay = ['bias', 'LayerNorm.weight']
optimizer_grouped_parameters = [
    {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
    {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
]
```


```python
# optimizer = 전체 데이터 셋의 실제 결과와 모델이 예측한 값 간의 차이가 효율적으로 좁혀질 수 있도록 최적화해주는 역할
# transform에서 제공하는 AdamW optimizer 사용
optimizer = AdamW(optimizer_grouped_parameters, lr=learning_rate)
# 예측값과 실제값의 오차인 손실함수는 pytorch에서 제공하는 다중분류를 위한 대표적인 손실함수인 torch.nn.CrossEntropyLoss 사용
loss_fn = nn.CrossEntropyLoss()
```

    /usr/local/lib/python3.7/dist-packages/transformers/optimization.py:309: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
      FutureWarning,



```python
# 전체 모델 불러오기(cpu 환경으로 사용할 때)
sentimentModel = torch.load('/content/drive/MyDrive/Colab Notebooks/jejuRecommand/jeju_model.pt')

# dict 불러오기
# model1.load_state_dict(torch.load('7emotions_model_state_dict.pt'))  

# 지정한 값 불러오기
# checkpoint = torch.load('7emotions_all.tar')   
# model1.load_state_dict(checkpoint['model'])
# optimizer.load_state_dict(checkpoint['optimizer'])
```


```python
# 앞에 클래스를 선언해줘야 모델 실행 가능

# kobert에서 vocab을 통해서 토큰화 진행
tokenizer = get_tokenizer()
tok = nlp.data.BERTSPTokenizer(tokenizer, vocab, lower=False)
sources = ['공천포','휴애리 자연생활공원','성산일출봉(UNESCO 세계자연유산)','본태박물관','제주올레 14코스',
           '바이나흐튼 크리스마스 박물관','세계자동차 & 피아노박물관','한국야구명예전당(야구박물관)','초콜릿박물관']
# 테스트 함수 
def predict(predict_sentence):

    data = [predict_sentence, '0']
    dataset_another = [data]

    another_test = BERTDataset(dataset_another, 0, 1, tok, max_len, True, False)
    test_dataloader = torch.utils.data.DataLoader(another_test, batch_size=batch_size, num_workers=5)
    
    # 평가 모드
    sentimentModel.eval()

    for batch_id, (token_ids, valid_length, segment_ids, label) in enumerate(test_dataloader):
        token_ids = token_ids.long().to(device)
        segment_ids = segment_ids.long().to(device)

        valid_length= valid_length
        label = label.long().to(device)

        out = sentimentModel(token_ids, valid_length, segment_ids)

        test_eval=[]

        # 학습위해 변환한 분류값 초기 분류값으로 변환
        for i in out:
            logits=i
            logits = logits.detach().cpu().numpy()

            test_eval.append(sources[np.argmax(logits)])

        print(test_eval[0])
```

    using cached model. /content/.cache/kobert_news_wiki_ko_cased-1087f8699e.spiece



```python
sentence = input("검색문장: ")
predict(sentence)
```

    검색문장: 박물관 추천


    /usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 5 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
      cpuset_checked))


    초콜릿박물관


ipynb
e16aacb4b9c6fddfdca27a57ebb0dc6870cbd6ee
Jupyter Notebook
2022-03-19T13:21:22.000Z
2022-03-09T02:45:17.000Z
6acf792ac8dc8cfc266e9837f25944cb3b7dfa16
MIT
dldydrhkd/jeju_tourist_recommand
analysis/jeju-chatbot-model-Import.ipynb
6acf792ac8dc8cfc266e9837f25944cb3b7dfa16
2022-03-23T13:52:26.000Z
2022-03-17T13:14:53.000Z
MIT
dldydrhkd/jeju_tourist_recommand
analysis/jeju-chatbot-model-Import.ipynb
6acf792ac8dc8cfc266e9837f25944cb3b7dfa16
MIT
dldydrhkd/jeju_tourist_recommand
analysis/jeju-chatbot-model-Import.ipynb