{"action":"created","issue":{"url":"https://api.github.com/repos/baidu/Paddle/issues/369","repository_url":"https://api.github.com/repos/baidu/Paddle","labels_url":"https://api.github.com/repos/baidu/Paddle/issues/369/labels{/name}","comments_url":"https://api.github.com/repos/baidu/Paddle/issues/369/comments","events_url":"https://api.github.com/repos/baidu/Paddle/issues/369/events","html_url":"https://github.com/baidu/Paddle/issues/369","id":187636881,"number":369,"title":"Can't get beyond the first batch when feeding in a dense_vector_sequence","user":{"login":"alvations","id":1050316,"avatar_url":"https://avatars.githubusercontent.com/u/1050316?v=3","gravatar_id":"","url":"https://api.github.com/users/alvations","html_url":"https://github.com/alvations","followers_url":"https://api.github.com/users/alvations/followers","following_url":"https://api.github.com/users/alvations/following{/other_user}","gists_url":"https://api.github.com/users/alvations/gists{/gist_id}","starred_url":"https://api.github.com/users/alvations/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alvations/subscriptions","organizations_url":"https://api.github.com/users/alvations/orgs","repos_url":"https://api.github.com/users/alvations/repos","events_url":"https://api.github.com/users/alvations/events{/privacy}","received_events_url":"https://api.github.com/users/alvations/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"reyoung","id":728699,"avatar_url":"https://avatars.githubusercontent.com/u/728699?v=3","gravatar_id":"","url":"https://api.github.com/users/reyoung","html_url":"https://github.com/reyoung","followers_url":"https://api.github.com/users/reyoung/followers","following_url":"https://api.github.com/users/reyoung/following{/other_user}","gists_url":"https://api.github.com/users/reyoung/gists{/gist_id}","starred_url":"https://api.github.com/users/reyoung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reyoung/subscriptions","organizations_url":"https://api.github.com/users/reyoung/orgs","repos_url":"https://api.github.com/users/reyoung/repos","events_url":"https://api.github.com/users/reyoung/events{/privacy}","received_events_url":"https://api.github.com/users/reyoung/received_events","type":"User","site_admin":false},"assignees":[{"login":"reyoung","id":728699,"avatar_url":"https://avatars.githubusercontent.com/u/728699?v=3","gravatar_id":"","url":"https://api.github.com/users/reyoung","html_url":"https://github.com/reyoung","followers_url":"https://api.github.com/users/reyoung/followers","following_url":"https://api.github.com/users/reyoung/following{/other_user}","gists_url":"https://api.github.com/users/reyoung/gists{/gist_id}","starred_url":"https://api.github.com/users/reyoung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reyoung/subscriptions","organizations_url":"https://api.github.com/users/reyoung/orgs","repos_url":"https://api.github.com/users/reyoung/repos","events_url":"https://api.github.com/users/reyoung/events{/privacy}","received_events_url":"https://api.github.com/users/reyoung/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2016-11-07T06:47:00Z","updated_at":"2016-11-07T08:54:06Z","closed_at":null,"body":"I have managed to feed numpy objects into Paddle by using something like `np.array.tolist()`:\r\n\r\n```python\r\nfrom paddle.trainer.PyDataProvider2 import *\r\n\r\nimport numpy as np\r\n\r\nUNK_IDX = 2\r\nSTART = \"<s>\"\r\nEND = \"<e>\"\r\n\r\ndef _get_ids(s, dictionary):\r\n    words = s.strip().split()\r\n    return [dictionary[START]] + \\\r\n           [dictionary.get(w, UNK_IDX) for w in words] + \\\r\n           [dictionary[END]]\r\n\r\ndef hook(settings, src_dict, trg_dict, file_list, **kwargs):\r\n    # Some code ...\r\n    # A numpy matrix that corresponds to the src (row) and target (column) vocabulary\r\n    settings.thematrix = np.random.rand(len(src_dict), len(trg_dict))\r\n    # ...\r\n    settings.slots = [ integer_value_sequence(len(settings.src_dict)),\r\n                           dense_vector_sequence(len(setting.src_dict)),\r\n                            integer_value_sequence(len(settings.trg_dict))]\r\n    # ...\r\n\r\n@provider(init_hook=hook, pool_size=50000)\r\ndef process(settings, file_name):\r\n    # ...\r\n    for line in enumerate(f):\r\n        src_seq, trg_seq = line.strip().split('\\t')\r\n        src_ids = _get_ids(src_seq, settings.src_dict)\r\n        trg_ids = [settings.trg_dict.get(w, UNK_IDX)\r\n                           for w in trg_words]\r\n        trg_ids = [settings.trg_dict[START]] + trg_ids\r\n    yield src_ids , settings.thematrix[src_ids].tolist(), trg_ids\r\n```\r\n\r\nSomehow the vectors can't seem to get pass the first batch and Paddle throws this error:\r\n\r\n```bash\r\n~/Paddle/demo/rowrow$ bash train.sh \r\nI1104 18:59:42.636052 18632 Util.cpp:151] commandline: /home/ltan/Paddle/binary/bin/../opt/paddle/bin/paddle_trainer --config=train.conf --save_dir=/home/ltan/Paddle/demo/rowrow/model --use_gpu=true --num_passes=100 --show_parameter_stats_period=1000 --trainer_count=4 --log_period=10 --dot_period=5 \r\nI1104 18:59:46.503566 18632 Util.cpp:126] Calling runInitFunctions\r\nI1104 18:59:46.503810 18632 Util.cpp:139] Call runInitFunctions done.\r\n[WARNING 2016-11-04 18:59:46,847 default_decorators.py:40] please use keyword arguments in paddle config.\r\n[INFO 2016-11-04 18:59:46,856 networks.py:1125] The input order is [source_language_word, target_language_word, target_language_next_word]\r\n[INFO 2016-11-04 18:59:46,857 networks.py:1132] The output order is [__cost_0__]\r\nI1104 18:59:46.871026 18632 Trainer.cpp:170] trainer mode: Normal\r\nI1104 18:59:46.871906 18632 MultiGradientMachine.cpp:108] numLogicalDevices=1 numThreads=4 numDevices=4\r\nI1104 18:59:46.988584 18632 PyDataProvider2.cpp:247] loading dataprovider dataprovider::process\r\n[INFO 2016-11-04 18:59:46,990 dataprovider.py:15] src dict len : 45661\r\n[INFO 2016-11-04 18:59:47,316 dataprovider.py:26] trg dict len : 422\r\nI1104 18:59:47.347944 18632 PyDataProvider2.cpp:247] loading dataprovider dataprovider::process\r\n[INFO 2016-11-04 18:59:47,348 dataprovider.py:15] src dict len : 45661\r\n[INFO 2016-11-04 18:59:47,657 dataprovider.py:26] trg dict len : 422\r\nI1104 18:59:47.658279 18632 GradientMachine.cpp:134] Initing parameters..\r\nI1104 18:59:49.244287 18632 GradientMachine.cpp:141] Init parameters done.\r\nF1104 18:59:50.485621 18632 PythonUtil.h:213] Check failed: PySequence_Check(seq_) \r\n*** Check failure stack trace: ***\r\n    @     0x7f71f521adaa  (unknown)\r\n    @     0x7f71f521ace4  (unknown)\r\n    @     0x7f71f521a6e6  (unknown)\r\n    @     0x7f71f521d687  (unknown)\r\n    @           0x54dac9  paddle::DenseScanner::fill()\r\n    @           0x54f1d1  paddle::SequenceScanner::fill()\r\n    @           0x5543cc  paddle::PyDataProvider2::getNextBatchInternal()\r\n    @           0x5779b2  paddle::DataProvider::getNextBatch()\r\n    @           0x6a01f7  paddle::Trainer::trainOnePass()\r\n    @           0x6a3b57  paddle::Trainer::train()\r\n    @           0x53a2b3  main\r\n    @     0x7f71f4426f45  (unknown)\r\n    @           0x545ae5  (unknown)\r\n    @              (nil)  (unknown)\r\n/home/ltan/Paddle/binary/bin/paddle: line 81: 18632 Aborted                 (core dumped) ${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer ${@:2}\r\n```\r\n\r\nWithout `tolist()`, the dataprovider would support numpy array too but it throws the same error.\r\n\r\nMore details on http://stackoverflow.com/questions/40421248/why-is-paddle-throwing-errors-when-feeding-in-a-dense-vector-sequence-to-a-seqto and the data+code that i use to run `train.sh` is in https://github.com/alvations/rowrow .\r\n\r\n**What does the error means? Why is it not going pass the first batch?**\r\n\r\nAlthough the vectors I am using are random vectors, I hope to read a pickled file that loads a similar matrix that corresponds to the `src_dict` and `target_dict`. I would consider them to be dense since all cells in the matrix contains floating point values that ranges between `[0.3, 1.0]`.\r\n\r\n**Is there any example of how to load a `dense_vector_sequence` in Paddle?**"},"comment":{"url":"https://api.github.com/repos/baidu/Paddle/issues/comments/258778619","html_url":"https://github.com/baidu/Paddle/issues/369#issuecomment-258778619","issue_url":"https://api.github.com/repos/baidu/Paddle/issues/369","id":258778619,"user":{"login":"alvations","id":1050316,"avatar_url":"https://avatars.githubusercontent.com/u/1050316?v=3","gravatar_id":"","url":"https://api.github.com/users/alvations","html_url":"https://github.com/alvations","followers_url":"https://api.github.com/users/alvations/followers","following_url":"https://api.github.com/users/alvations/following{/other_user}","gists_url":"https://api.github.com/users/alvations/gists{/gist_id}","starred_url":"https://api.github.com/users/alvations/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alvations/subscriptions","organizations_url":"https://api.github.com/users/alvations/orgs","repos_url":"https://api.github.com/users/alvations/repos","events_url":"https://api.github.com/users/alvations/events{/privacy}","received_events_url":"https://api.github.com/users/alvations/received_events","type":"User","site_admin":false},"created_at":"2016-11-07T08:54:06Z","updated_at":"2016-11-07T08:54:06Z","body":"Should I close this issue?"}}
{"actor":{"display_login":"alvations"}}