{"action":"created","issue":{"url":"https://api.github.com/repos/IndicoDataSolutions/finetune/issues/134","repository_url":"https://api.github.com/repos/IndicoDataSolutions/finetune","labels_url":"https://api.github.com/repos/IndicoDataSolutions/finetune/issues/134/labels{/name}","comments_url":"https://api.github.com/repos/IndicoDataSolutions/finetune/issues/134/comments","events_url":"https://api.github.com/repos/IndicoDataSolutions/finetune/issues/134/events","html_url":"https://github.com/IndicoDataSolutions/finetune/issues/134","id":358003867,"node_id":"MDU6SXNzdWUzNTgwMDM4Njc=","number":134,"title":"Cannot specify max_length more than 512","user":{"login":"thinline72","id":8447843,"node_id":"MDQ6VXNlcjg0NDc4NDM=","avatar_url":"https://avatars0.githubusercontent.com/u/8447843?v=4","gravatar_id":"","url":"https://api.github.com/users/thinline72","html_url":"https://github.com/thinline72","followers_url":"https://api.github.com/users/thinline72/followers","following_url":"https://api.github.com/users/thinline72/following{/other_user}","gists_url":"https://api.github.com/users/thinline72/gists{/gist_id}","starred_url":"https://api.github.com/users/thinline72/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thinline72/subscriptions","organizations_url":"https://api.github.com/users/thinline72/orgs","repos_url":"https://api.github.com/users/thinline72/repos","events_url":"https://api.github.com/users/thinline72/events{/privacy}","received_events_url":"https://api.github.com/users/thinline72/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-09-07T10:06:08Z","updated_at":"2018-09-07T18:26:02Z","closed_at":"2018-09-07T18:26:02Z","author_association":"NONE","body":"Hello,\r\n\r\nI've tried to use max_length more than 512 to featurize text:\r\n```\r\nmodel = finetune.Classifier()\r\ntrn_X_q_vecs = model.featurize(trn_X_q, max_length=1000)\r\n```\r\n\r\nBut I got the following exception:\r\n```\r\n---------------------------------------------------------------------------\r\nValueError                                Traceback (most recent call last)\r\n<ipython-input-7-d3d9e8b820e5> in <module>()\r\n----> 1 trn_X_q_vecs = model.featurize(trn_X_q, max_length=1000)\r\n\r\n/opt/conda/lib/python3.6/site-packages/finetune/classifier.py in featurize(self, X, max_length)\r\n     24         :returns: np.array of features of shape (n_examples, embedding_size).\r\n     25         \"\"\"\r\n---> 26         return super().featurize(X, max_length=max_length)\r\n     27 \r\n     28     def predict(self, X, max_length=None):\r\n\r\n/opt/conda/lib/python3.6/site-packages/finetune/base.py in featurize(self, *args, **kwargs)\r\n    386         These features are the same that are fed into the target_model.\r\n    387         \"\"\"\r\n--> 388         return self._featurize(*args, **kwargs)\r\n    389 \r\n    390     @classmethod\r\n\r\n/opt/conda/lib/python3.6/site-packages/finetune/base.py in _featurize(self, Xs, max_length)\r\n    371             warnings.filterwarnings(\"ignore\")\r\n    372             max_length = max_length or self.config.max_length\r\n--> 373             for xmb, mmb in self._infer_prep(Xs, max_length=max_length):\r\n    374                 feature_batch = self.sess.run(self.features, {\r\n    375                     self.X: xmb,\r\n\r\n/opt/conda/lib/python3.6/site-packages/finetune/base.py in _infer_prep(self, Xs, max_length)\r\n    400     def _infer_prep(self, Xs, max_length=None):\r\n    401         max_length = max_length or self.config.max_length\r\n--> 402         arr_encoded = self._text_to_ids(Xs, max_length=max_length)\r\n    403         n_batch_train = self.config.batch_size * max(len(self.config.visible_gpus), 1)\r\n    404         self._build_model(n_updates_total=0, target_dim=self.target_dim, train=False)\r\n\r\n/opt/conda/lib/python3.6/site-packages/finetune/base.py in _text_to_ids(self, Xs, Y, max_length)\r\n    156         else:\r\n    157             encoder_out = self.encoder.encode_multi_input(Xs, Y=Y, max_length=max_length)\r\n--> 158             return self._array_format(encoder_out)\r\n    159 \r\n    160 \r\n\r\n/opt/conda/lib/python3.6/site-packages/finetune/base.py in _array_format(self, encoded_output)\r\n    421         for i, seq_length in enumerate(seq_lengths):\r\n    422             # BPE embedding\r\n--> 423             x[i, :seq_length, 0] = encoded_output.token_ids[i]\r\n    424             # masking: value of 1 means \"consider this in cross-entropy LM loss\"\r\n    425             mask[i, 1:seq_length] = 1\r\n\r\nValueError: cannot copy sequence with size 667 to array axis with dimension 512\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/IndicoDataSolutions/finetune/issues/comments/419526072","html_url":"https://github.com/IndicoDataSolutions/finetune/issues/134#issuecomment-419526072","issue_url":"https://api.github.com/repos/IndicoDataSolutions/finetune/issues/134","id":419526072,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTUyNjA3Mg==","user":{"login":"thinline72","id":8447843,"node_id":"MDQ6VXNlcjg0NDc4NDM=","avatar_url":"https://avatars0.githubusercontent.com/u/8447843?v=4","gravatar_id":"","url":"https://api.github.com/users/thinline72","html_url":"https://github.com/thinline72","followers_url":"https://api.github.com/users/thinline72/followers","following_url":"https://api.github.com/users/thinline72/following{/other_user}","gists_url":"https://api.github.com/users/thinline72/gists{/gist_id}","starred_url":"https://api.github.com/users/thinline72/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thinline72/subscriptions","organizations_url":"https://api.github.com/users/thinline72/orgs","repos_url":"https://api.github.com/users/thinline72/repos","events_url":"https://api.github.com/users/thinline72/events{/privacy}","received_events_url":"https://api.github.com/users/thinline72/received_events","type":"User","site_admin":false},"created_at":"2018-09-07T18:26:02Z","updated_at":"2018-09-07T18:26:02Z","author_association":"NONE","body":"@madisonmay Got it, thank you! I'll close the issue.\r\n\r\nJust last question. Why spacy tokenizer + bpe are used? Why not just bpe (or something like https://github.com/google/sentencepiece)? Is it something that was just inherited from original OpenAI model and preprocessing?"}}
{"actor":{"display_login":"thinline72"}}