{"action":"created","issue":{"url":"https://api.github.com/repos/reinforceio/tensorforce/issues/335","repository_url":"https://api.github.com/repos/reinforceio/tensorforce","labels_url":"https://api.github.com/repos/reinforceio/tensorforce/issues/335/labels{/name}","comments_url":"https://api.github.com/repos/reinforceio/tensorforce/issues/335/comments","events_url":"https://api.github.com/repos/reinforceio/tensorforce/issues/335/events","html_url":"https://github.com/reinforceio/tensorforce/issues/335","id":295891122,"number":335,"title":"tensorflow.python.framework.errors_impl.InvalidArgumentError: Must have updates.shape = indices.shape + params.shape[1:], got updates.shape [352], indices.shape [254], params.shape [100000]","user":{"login":"lefnire","id":195202,"avatar_url":"https://avatars3.githubusercontent.com/u/195202?v=4","gravatar_id":"","url":"https://api.github.com/users/lefnire","html_url":"https://github.com/lefnire","followers_url":"https://api.github.com/users/lefnire/followers","following_url":"https://api.github.com/users/lefnire/following{/other_user}","gists_url":"https://api.github.com/users/lefnire/gists{/gist_id}","starred_url":"https://api.github.com/users/lefnire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lefnire/subscriptions","organizations_url":"https://api.github.com/users/lefnire/orgs","repos_url":"https://api.github.com/users/lefnire/repos","events_url":"https://api.github.com/users/lefnire/events{/privacy}","received_events_url":"https://api.github.com/users/lefnire/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"AlexKuhnle","id":14937649,"avatar_url":"https://avatars2.githubusercontent.com/u/14937649?v=4","gravatar_id":"","url":"https://api.github.com/users/AlexKuhnle","html_url":"https://github.com/AlexKuhnle","followers_url":"https://api.github.com/users/AlexKuhnle/followers","following_url":"https://api.github.com/users/AlexKuhnle/following{/other_user}","gists_url":"https://api.github.com/users/AlexKuhnle/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexKuhnle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexKuhnle/subscriptions","organizations_url":"https://api.github.com/users/AlexKuhnle/orgs","repos_url":"https://api.github.com/users/AlexKuhnle/repos","events_url":"https://api.github.com/users/AlexKuhnle/events{/privacy}","received_events_url":"https://api.github.com/users/AlexKuhnle/received_events","type":"User","site_admin":false},"assignees":[{"login":"AlexKuhnle","id":14937649,"avatar_url":"https://avatars2.githubusercontent.com/u/14937649?v=4","gravatar_id":"","url":"https://api.github.com/users/AlexKuhnle","html_url":"https://github.com/AlexKuhnle","followers_url":"https://api.github.com/users/AlexKuhnle/followers","following_url":"https://api.github.com/users/AlexKuhnle/following{/other_user}","gists_url":"https://api.github.com/users/AlexKuhnle/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexKuhnle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexKuhnle/subscriptions","organizations_url":"https://api.github.com/users/AlexKuhnle/orgs","repos_url":"https://api.github.com/users/AlexKuhnle/repos","events_url":"https://api.github.com/users/AlexKuhnle/events{/privacy}","received_events_url":"https://api.github.com/users/AlexKuhnle/received_events","type":"User","site_admin":false},{"login":"michaelschaarschmidt","id":22558112,"avatar_url":"https://avatars1.githubusercontent.com/u/22558112?v=4","gravatar_id":"","url":"https://api.github.com/users/michaelschaarschmidt","html_url":"https://github.com/michaelschaarschmidt","followers_url":"https://api.github.com/users/michaelschaarschmidt/followers","following_url":"https://api.github.com/users/michaelschaarschmidt/following{/other_user}","gists_url":"https://api.github.com/users/michaelschaarschmidt/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelschaarschmidt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelschaarschmidt/subscriptions","organizations_url":"https://api.github.com/users/michaelschaarschmidt/orgs","repos_url":"https://api.github.com/users/michaelschaarschmidt/repos","events_url":"https://api.github.com/users/michaelschaarschmidt/events{/privacy}","received_events_url":"https://api.github.com/users/michaelschaarschmidt/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-02-09T14:36:21Z","updated_at":"2018-02-10T13:01:22Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Rebased #memory today starting at c2c9dddf6d99bfa14b8ffb65b507a6be50b0ad6e. I'm using PPO, `memory = {'type': 'latest', 'include_next_states': False, 'capacity': 100000}`, `update_mode = {'unit': 'episodes', 'batch_size': 8, 'frequency': 8}`. I'm using both multi-state and multi-action, [they're both dicts](https://github.com/lefnire/tforce_btc_trader/blob/master/btc_env.py#L144tL160) (thought I'd mention seeing `initialize/action-action` in the error). Single-thread runner, no distributed.\r\n\r\nLMK if nothing obvious pops up and I'll try to replicate w/ one in the examples dir.\r\n\r\n```shell\r\nTraceback (most recent call last):\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 1350, in _do_call\r\n    return fn(*args)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 1329, in _run_fn\r\n    status, run_metadata)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py\", line 473, in __exit__\r\n    c_api.TF_GetCode(self.status.status))\r\ntensorflow.python.framework.errors_impl.InvalidArgumentError: Must have updates.shape = indices.shape + params.shape[1:], got updates.shape [352], indices.shape [254], params.shape [100000]\r\n         [[Node: ppo/observe-timestep/store/ScatterUpdate_2 = ScatterUpdate[T=DT_INT32, Tindices=DT_INT32, _class=[\"loc:@ppo/initialize/latest/initialize/action-action\"], use_locking=true, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](ppo/initialize/latest/initialize/action-action, ppo/observe-timestep/store/mod/_347, ppo/StopGradient_2, ^ppo/observe\r\n-timestep/store/AssignSub)]]\r\n         [[Node: ppo/observe-timestep/store/ScatterUpdate_2/_384 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device_incarnation=1, tensor_name=\"edge_147_ppo/observe-timestep/store/ScatterUpdate_2\", tensor_type=DT_FLOAT, _device=\"/job:localhost/repl\r\nica:0/task:0/device:GPU:0\"]()]]\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"hypersearch.py\", line 809, in <module>\r\n    main()\r\n  File \"hypersearch.py\", line 805, in main\r\n    y_list=Y\r\n  File \"/home/lefnire/btcbot/gp.py\", line 218, in bayesian_optimisation2\r\n    loss_fn(next_sample)\r\n  File \"hypersearch.py\", line 756, in loss_fn\r\n    reward = hsearch.execute(vec2hypers(params))\r\n  File \"hypersearch.py\", line 582, in execute\r\n    env.train_and_test(agent, self.cli_args.n_steps, self.cli_args.n_tests, -1)\r\n  File \"/home/lefnire/btcbot/btc_env.py\", line 514, in train_and_test\r\n    runner.run(timesteps=timesteps_each)\r\n  File \"/home/lefnire/tensorforce/tensorforce/execution/runner.py\", line 131, in run\r\n    self.agent.observe(terminal=terminal, reward=reward)\r\n  File \"/home/lefnire/tensorforce/tensorforce/agents/agent.py\", line 216, in observe\r\n    reward=self.observe_reward\r\n  File \"/home/lefnire/tensorforce/tensorforce/models/model.py\", line 1188, in observe\r\n    episode = self.monitored_session.run(fetches=fetches, feed_dict=feed_dict)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py\", line 539, in run\r\n    run_metadata=run_metadata)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py\", line 1104, in run\r\n    raise six.reraise(*original_exc_info)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/six.py\", line 693, in reraise\r\n    raise value\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py\", line 1089, in run\r\n    return self._sess.run(*args, **kwargs)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py\", line 1161, in run\r\n    run_metadata=run_metadata)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py\", line 941, in run\r\n    return self._sess.run(*args, **kwargs)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 895, in run\r\n    run_metadata_ptr)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 1128, in _run\r\n    feed_dict_tensor, options, run_metadata)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 1344, in _do_run\r\n    options, run_metadata)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 1363, in _do_call\r\n    raise type(e)(node_def, op, message)\r\ntensorflow.python.framework.errors_impl.InvalidArgumentError: Must have updates.shape = indices.shape + params.shape[1:], got updates.shape [352], indices.shape [254], params.shape [100000]\r\n         [[Node: ppo/observe-timestep/store/ScatterUpdate_2 = ScatterUpdate[T=DT_INT32, Tindices=DT_INT32, _class=[\"loc:@ppo/initialize/latest/initialize/action-action\"], use_locking=true, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](ppo/initialize/latest/initialize/action-action, ppo/observe-timestep/store/mod/_347, ppo/StopGradient_2, ^ppo/observe\r\n-timestep/store/AssignSub)]]\r\n         [[Node: ppo/observe-timestep/store/ScatterUpdate_2/_384 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device_incarnation=1, tensor_name=\"edge_147_ppo/observe-timestep/store/ScatterUpdate_2\", tensor_type=DT_FLOAT, _device=\"/job:localhost/repl\r\nica:0/task:0/device:GPU:0\"]()]]\r\n\r\nCaused by op 'ppo/observe-timestep/store/ScatterUpdate_2', defined at:\r\n  File \"hypersearch.py\", line 809, in <module>\r\n    main()\r\n  File \"hypersearch.py\", line 805, in main\r\n    y_list=Y\r\n  File \"/home/lefnire/btcbot/gp.py\", line 218, in bayesian_optimisation2\r\n    loss_fn(next_sample)\r\n  File \"hypersearch.py\", line 756, in loss_fn\r\n    reward = hsearch.execute(vec2hypers(params))\r\n  File \"hypersearch.py\", line 579, in execute\r\n    **hydrated\r\n  File \"/home/lefnire/tensorforce/tensorforce/agents/ppo_agent.py\", line 176, in __init__\r\n    entropy_regularization=entropy_regularization\r\n  File \"/home/lefnire/tensorforce/tensorforce/agents/learning_agent.py\", line 132, in __init__\r\n    batching_capacity=batching_capacity\r\n  File \"/home/lefnire/tensorforce/tensorforce/agents/agent.py\", line 85, in __init__\r\n    self.model = self.initialize_model()\r\n  File \"/home/lefnire/tensorforce/tensorforce/agents/ppo_agent.py\", line 204, in initialize_model\r\n  File \"/home/lefnire/tensorforce/tensorforce/agents/ppo_agent.py\", line 204, in initialize_model\r\n    likelihood_ratio_clipping=self.likelihood_ratio_clipping\r\n  File \"/home/lefnire/tensorforce/tensorforce/models/pg_prob_ratio_model.py\", line 88, in __init__\r\n    gae_lambda=gae_lambda\r\n  File \"/home/lefnire/tensorforce/tensorforce/models/pg_model.py\", line 95, in __init__\r\n    requires_deterministic=False\r\n  File \"/home/lefnire/tensorforce/tensorforce/models/distribution_model.py\", line 86, in __init__\r\n    discount=discount\r\n  File \"/home/lefnire/tensorforce/tensorforce/models/memory_model.py\", line 80, in __init__\r\n    reward_preprocessing=reward_preprocessing\r\n  File \"/home/lefnire/tensorforce/tensorforce/models/model.py\", line 214, in __init__\r\n    self.setup()\r\n  File \"/home/lefnire/tensorforce/tensorforce/models/model.py\", line 355, in setup\r\n    deterministic=deterministic\r\n  File \"/home/lefnire/tensorforce/tensorforce/models/memory_model.py\", line 548, in create_operations\r\n    deterministic=deterministic\r\n  File \"/home/lefnire/tensorforce/tensorforce/models/model.py\", line 998, in create_operations\r\n    self.create_observe_operations(reward=reward, terminal=terminal)\r\n  File \"/home/lefnire/tensorforce/tensorforce/models/model.py\", line 980, in create_observe_operations\r\n    reward=reward\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/template.py\", line 283, in __call__\r\n    result = self._call_func(args, kwargs, check_for_new_variables=False)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/template.py\", line 222, in _call_func\r\n    result = self._func(*args, **kwargs)\r\n  File \"/home/lefnire/tensorforce/tensorforce/models/memory_model.py\", line 452, in tf_observe_timestep\r\n    reward=reward\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/template.py\", line 266, in __call__\r\n    return self._call_func(args, kwargs, check_for_new_variables=True)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/template.py\", line 222, in _call_func\r\n    result = self._func(*args, **kwargs)\r\n  File \"/home/lefnire/tensorforce/tensorforce/core/memories/queue.py\", line 175, in tf_store\r\n    updates=action\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py\", line 349, in scatter_update\r\n    use_locking=use_locking, name=name)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py\", line 920, in scatter_update\r\n    use_locking=use_locking, name=name)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py\", line 787, in _apply_op_helper\r\n    op_def=op_def)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 3160, in create_op\r\n    op_def=op_def)\r\n  File \"/home/lefnire/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 1625, in __init__\r\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\r\n\r\nInvalidArgumentError (see above for traceback): Must have updates.shape = indices.shape + params.shape[1:], got updates.shape [352], indices.shape [254], params.shape [100000]\r\n         [[Node: ppo/observe-timestep/store/ScatterUpdate_2 = ScatterUpdate[T=DT_INT32, Tindices=DT_INT32, _class=[\"loc:@ppo/initialize/latest/initialize/action-action\"], use_locking=true, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](ppo/initialize/latest/initialize/action-action, ppo/observe-timestep/store/mod/_347, ppo/StopGradient_2, ^ppo/observe\r\n-timestep/store/AssignSub)]]\r\n         [[Node: ppo/observe-timestep/store/ScatterUpdate_2/_384 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device_incarnation=1, tensor_name=\"edge_147_ppo/observe-timestep/store/ScatterUpdate_2\", tensor_type=DT_FLOAT, _device=\"/job:localhost/repl\r\nica:0/task:0/device:GPU:0\"]()]]\r\n```"},"comment":{"url":"https://api.github.com/repos/reinforceio/tensorforce/issues/comments/364650713","html_url":"https://github.com/reinforceio/tensorforce/issues/335#issuecomment-364650713","issue_url":"https://api.github.com/repos/reinforceio/tensorforce/issues/335","id":364650713,"user":{"login":"AlexKuhnle","id":14937649,"avatar_url":"https://avatars2.githubusercontent.com/u/14937649?v=4","gravatar_id":"","url":"https://api.github.com/users/AlexKuhnle","html_url":"https://github.com/AlexKuhnle","followers_url":"https://api.github.com/users/AlexKuhnle/followers","following_url":"https://api.github.com/users/AlexKuhnle/following{/other_user}","gists_url":"https://api.github.com/users/AlexKuhnle/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexKuhnle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexKuhnle/subscriptions","organizations_url":"https://api.github.com/users/AlexKuhnle/orgs","repos_url":"https://api.github.com/users/AlexKuhnle/repos","events_url":"https://api.github.com/users/AlexKuhnle/events{/privacy}","received_events_url":"https://api.github.com/users/AlexKuhnle/received_events","type":"User","site_admin":false},"created_at":"2018-02-10T13:01:22Z","updated_at":"2018-02-10T13:01:22Z","author_association":"OWNER","body":"That sounds unexpected indeed, so if you can provide an example to reproduce the message, that would be helpful. I encountered this problem in distributed mode in the last days, but in the end all tests and distributed seemed to work, and we seem to cover multi-state/action PPO. It's interesting, though, that the problem seems to happen for actions, not for states/internals (which are assigned before)."}}
{"actor":{"display_login":"AlexKuhnle"}}