{"action":"created","issue":{"url":"https://api.github.com/repos/rkcosmos/deepcut/issues/42","repository_url":"https://api.github.com/repos/rkcosmos/deepcut","labels_url":"https://api.github.com/repos/rkcosmos/deepcut/issues/42/labels{/name}","comments_url":"https://api.github.com/repos/rkcosmos/deepcut/issues/42/comments","events_url":"https://api.github.com/repos/rkcosmos/deepcut/issues/42/events","html_url":"https://github.com/rkcosmos/deepcut/issues/42","id":365358366,"node_id":"MDU6SXNzdWUzNjUzNTgzNjY=","number":42,"title":"I have a few question about DeepcutTokenizer","user":{"login":"Zylinks","id":31320324,"node_id":"MDQ6VXNlcjMxMzIwMzI0","avatar_url":"https://avatars3.githubusercontent.com/u/31320324?v=4","gravatar_id":"","url":"https://api.github.com/users/Zylinks","html_url":"https://github.com/Zylinks","followers_url":"https://api.github.com/users/Zylinks/followers","following_url":"https://api.github.com/users/Zylinks/following{/other_user}","gists_url":"https://api.github.com/users/Zylinks/gists{/gist_id}","starred_url":"https://api.github.com/users/Zylinks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zylinks/subscriptions","organizations_url":"https://api.github.com/users/Zylinks/orgs","repos_url":"https://api.github.com/users/Zylinks/repos","events_url":"https://api.github.com/users/Zylinks/events{/privacy}","received_events_url":"https://api.github.com/users/Zylinks/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-01T08:33:59Z","updated_at":"2018-10-01T19:56:15Z","closed_at":null,"author_association":"NONE","body":"CountVectorier have methods transform but DeepcutTokenizer don't have this methods. \r\nin the future deepcut will implement this methods or not.\r\nbecause i have to change CountVectorier to DeepTokenizer.\r\nbecause it have to transform test data before predict.\r\n// I am not good with English grammar. if you not understand i can explain in Thai lang thx\r\n![42943831_332569077309557_2723233342697766912_n](https://user-images.githubusercontent.com/31320324/46277926-f3233980-c58e-11e8-8d13-de5b3bffc544.png)\r\n"},"comment":{"url":"https://api.github.com/repos/rkcosmos/deepcut/issues/comments/426041378","html_url":"https://github.com/rkcosmos/deepcut/issues/42#issuecomment-426041378","issue_url":"https://api.github.com/repos/rkcosmos/deepcut/issues/42","id":426041378,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNjA0MTM3OA==","user":{"login":"titipata","id":8295838,"node_id":"MDQ6VXNlcjgyOTU4Mzg=","avatar_url":"https://avatars0.githubusercontent.com/u/8295838?v=4","gravatar_id":"","url":"https://api.github.com/users/titipata","html_url":"https://github.com/titipata","followers_url":"https://api.github.com/users/titipata/followers","following_url":"https://api.github.com/users/titipata/following{/other_user}","gists_url":"https://api.github.com/users/titipata/gists{/gist_id}","starred_url":"https://api.github.com/users/titipata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/titipata/subscriptions","organizations_url":"https://api.github.com/users/titipata/orgs","repos_url":"https://api.github.com/users/titipata/repos","events_url":"https://api.github.com/users/titipata/events{/privacy}","received_events_url":"https://api.github.com/users/titipata/received_events","type":"User","site_admin":false},"created_at":"2018-10-01T19:56:15Z","updated_at":"2018-10-01T19:56:15Z","author_association":"COLLABORATOR","body":"Hi @Zylinks, Iet's say `['ฉันบินได้', 'ฉันกินข้าว', 'ฉันอยากบิน']` is your training set and  `['ฉันหิว', 'เขาไม่มี']` is your validation set. You can use `DeepcutTokenizer` like the following:\r\n\r\n```python\r\nfrom deepcut import DeepcutTokenizer\r\ntokenizer = DeepcutTokenizer(ngram_range=(1,1),\r\n                             max_df=1.0, min_df=0.0)\r\nX = tokenizer.fit_tranform(['ฉันบินได้', 'ฉันกินข้าว', 'ฉันอยากบิน'])\r\n```\r\n\r\nIf you see the vocabulary, it will look something like:\r\n\r\n```python\r\ntokenizer.vocabulary_\r\n>> {'ได้': 0, 'บิน': 1, 'ฉัน': 2, 'อยาก': 3, 'ข้าว': 4, 'กิน': 5}\r\n```\r\n\r\nThen, when you apply on the validation dataset, you will see that we will have `0` for the second row since words in `เขาไม่มี` does not exist in the training data.\r\n\r\n```python\r\nX_test = tokenizer.transform(['ฉันหิว', 'เขาไม่มี'])\r\nX_test.todense()\r\n\r\n>> matrix([[0., 0., 1., 0., 0., 0.],\r\n           [0., 0., 0., 0., 0., 0.]])\r\n```"}}
{"actor":{"display_login":"titipata"}}