{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/text/issues/289","repository_url":"https://api.github.com/repos/pytorch/text","labels_url":"https://api.github.com/repos/pytorch/text/issues/289/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/text/issues/289/comments","events_url":"https://api.github.com/repos/pytorch/text/issues/289/events","html_url":"https://github.com/pytorch/text/issues/289","id":314907009,"node_id":"MDU6SXNzdWUzMTQ5MDcwMDk=","number":289,"title":"Unicode Error in Using IWSLT dataset: TypeError: write() argument 1 must be unicode, not str","user":{"login":"ustctf","id":10680608,"node_id":"MDQ6VXNlcjEwNjgwNjA4","avatar_url":"https://avatars3.githubusercontent.com/u/10680608?v=4","gravatar_id":"","url":"https://api.github.com/users/ustctf","html_url":"https://github.com/ustctf","followers_url":"https://api.github.com/users/ustctf/followers","following_url":"https://api.github.com/users/ustctf/following{/other_user}","gists_url":"https://api.github.com/users/ustctf/gists{/gist_id}","starred_url":"https://api.github.com/users/ustctf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ustctf/subscriptions","organizations_url":"https://api.github.com/users/ustctf/orgs","repos_url":"https://api.github.com/users/ustctf/repos","events_url":"https://api.github.com/users/ustctf/events{/privacy}","received_events_url":"https://api.github.com/users/ustctf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-04-17T05:08:39Z","updated_at":"2018-07-31T22:50:24Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nI'm using the IWSLT translation dataset in torchtext. However I found the following encoding errors. The code snippet is:\r\n\r\nMAX_LEN = 100\r\ntrain, val, test = datasets.IWSLT.splits(\r\n        exts=('.de', '.en'), fields=(SRC, TGT),\r\n        filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and\r\n            len(vars(x)['trg']) <= MAX_LEN)\r\n\r\nIndeed in an third-party implementation of Transformer using pytorch: https://github.com/harvardnlp/annotated-transformer/blob/master/The%20Annotated%20Transformer.ipynb\r\n\r\nThe error is:\r\n\r\n.data/iwslt/de-en/IWSLT16.TED.tst2011.de-en.en.xml\r\nTraceback (most recent call last):\r\n  File \"The+Annotated+Transformer.py\", line 773, in <module>\r\n    filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and\r\n  File \"/usr/anaconda2/lib/python2.7/site-packages/torchtext/datasets/translation.py\", line 140, in splits\r\n    cls.clean(path)\r\n  File \"/usr/anaconda2/lib/python2.7/site-packages/torchtext/datasets/translation.py\", line 160, in clean\r\n    fd_txt.write(e.text.strip() + '\\n')\r\nTypeError: write() argument 1 must be unicode, not str\r\n\r\nWould you help to check the issue? Thanks!\r\n\r\nSystems config: Ubuntu 14.04, python 2.7.\r\n\r\nBest,\r\nFei"},"comment":{"url":"https://api.github.com/repos/pytorch/text/issues/comments/409393173","html_url":"https://github.com/pytorch/text/issues/289#issuecomment-409393173","issue_url":"https://api.github.com/repos/pytorch/text/issues/289","id":409393173,"node_id":"MDEyOklzc3VlQ29tbWVudDQwOTM5MzE3Mw==","user":{"login":"suwangcompling","id":16518665,"node_id":"MDQ6VXNlcjE2NTE4NjY1","avatar_url":"https://avatars1.githubusercontent.com/u/16518665?v=4","gravatar_id":"","url":"https://api.github.com/users/suwangcompling","html_url":"https://github.com/suwangcompling","followers_url":"https://api.github.com/users/suwangcompling/followers","following_url":"https://api.github.com/users/suwangcompling/following{/other_user}","gists_url":"https://api.github.com/users/suwangcompling/gists{/gist_id}","starred_url":"https://api.github.com/users/suwangcompling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suwangcompling/subscriptions","organizations_url":"https://api.github.com/users/suwangcompling/orgs","repos_url":"https://api.github.com/users/suwangcompling/repos","events_url":"https://api.github.com/users/suwangcompling/events{/privacy}","received_events_url":"https://api.github.com/users/suwangcompling/received_events","type":"User","site_admin":false},"created_at":"2018-07-31T22:50:24Z","updated_at":"2018-07-31T22:50:24Z","author_association":"NONE","body":"I have exactly the same issue. The full code is from a tutorial, as follows:\r\n```\r\n# For data loading.\r\nfrom torchtext import data, datasets\r\n\r\nif True:\r\n    import spacy\r\n    spacy_de = spacy.load('de')\r\n    spacy_en = spacy.load('en')\r\n\r\n    def tokenize_de(text):\r\n        return [tok.text for tok in spacy_de.tokenizer(text)]\r\n\r\n    def tokenize_en(text):\r\n        return [tok.text for tok in spacy_en.tokenizer(text)]\r\n\r\n    BOS_WORD = u'<s>'\r\n    EOS_WORD = u'</s>'\r\n    BLANK_WORD = u\"<blank>\"\r\n    SRC = data.Field(tokenize=tokenize_de, pad_token=BLANK_WORD)\r\n    TGT = data.Field(tokenize=tokenize_en, init_token = BOS_WORD, \r\n                     eos_token = EOS_WORD, pad_token=BLANK_WORD)\r\n\r\n    MAX_LEN = 100\r\n    \r\n    train, val, test = datasets.IWSLT.splits(\r\n        exts=('.de', '.en'), fields=(SRC, TGT), \r\n        filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and \r\n            len(vars(x)['trg']) <= MAX_LEN)\r\n    MIN_FREQ = 2\r\n    SRC.build_vocab(train.src, min_freq=MIN_FREQ)\r\n    TGT.build_vocab(train.trg, min_freq=MIN_FREQ)\r\n```\r\nThe error that results is\r\n```\r\n.data/iwslt/de-en/IWSLT16.TED.tst2013.de-en.en.xml\r\n---------------------------------------------------------------------------\r\nTypeError                                 Traceback (most recent call last)\r\n<ipython-input-51-d787c2f8b289> in <module>()\r\n     24     train, val, test = datasets.IWSLT.splits(\r\n     25         exts=('.de', '.en'), fields=(SRC, TGT),\r\n---> 26         filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and\r\n     27             len(vars(x)['trg']) <= MAX_LEN)\r\n     28     MIN_FREQ = 2\r\n\r\n/usr/local/lib/python2.7/dist-packages/torchtext/datasets/translation.pyc in splits(cls, exts, fields, root, train, validation, test, **kwargs)\r\n    138 \r\n    139         if not os.path.exists(os.path.join(path, train) + exts[0]):\r\n--> 140             cls.clean(path)\r\n    141 \r\n    142         train_data = None if train is None else cls(\r\n\r\n/usr/local/lib/python2.7/dist-packages/torchtext/datasets/translation.pyc in clean(path)\r\n    158                 for doc in root.findall('doc'):\r\n    159                     for e in doc.findall('seg'):\r\n--> 160                         fd_txt.write(e.text.strip() + '\\n')\r\n    161 \r\n    162         xml_tags = ['<url', '<keywords', '<talkid', '<description',\r\n\r\nTypeError: write() argument 1 must be unicode, not str\r\n```"}}
{"actor":{"display_login":"suwangcompling"}}