{"action":"opened","issue":{"url":"https://api.github.com/repos/AppliedDataSciencePartners/WorldModels/issues/3","repository_url":"https://api.github.com/repos/AppliedDataSciencePartners/WorldModels","labels_url":"https://api.github.com/repos/AppliedDataSciencePartners/WorldModels/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/AppliedDataSciencePartners/WorldModels/issues/3/comments","events_url":"https://api.github.com/repos/AppliedDataSciencePartners/WorldModels/issues/3/events","html_url":"https://github.com/AppliedDataSciencePartners/WorldModels/issues/3","id":315664865,"number":3,"title":"Error training controller","user":{"login":"boucher","id":22065,"avatar_url":"https://avatars0.githubusercontent.com/u/22065?v=4","gravatar_id":"","url":"https://api.github.com/users/boucher","html_url":"https://github.com/boucher","followers_url":"https://api.github.com/users/boucher/followers","following_url":"https://api.github.com/users/boucher/following{/other_user}","gists_url":"https://api.github.com/users/boucher/gists{/gist_id}","starred_url":"https://api.github.com/users/boucher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/boucher/subscriptions","organizations_url":"https://api.github.com/users/boucher/orgs","repos_url":"https://api.github.com/users/boucher/repos","events_url":"https://api.github.com/users/boucher/events{/privacy}","received_events_url":"https://api.github.com/users/boucher/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-18T22:32:38Z","updated_at":"2018-04-18T22:32:38Z","closed_at":null,"author_association":"NONE","body":"```boucher@boucher-pc ~/Projects/WorldModels $ python 05_train_controller.py car_racing --num_worker 1 --num_worker_trial 2 --num_episode 16 --max_length 1000 --eval_steps 25\r\n/home/boucher/.virtualenvs/worldmodels/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\r\n  from ._conv import register_converters as _register_converters\r\nUsing TensorFlow backend.\r\n['mpirun', '-np', '2', '/home/boucher/.virtualenvs/worldmodels/bin/python', '05_train_controller.py', 'car_racing', '--num_worker', '1', '--num_worker_trial', '2', '--num_episode', '16', '--max_length', '1000', '--eval_steps', '25']\r\n/home/boucher/.virtualenvs/worldmodels/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\r\n  from ._conv import register_converters as _register_converters\r\n/home/boucher/.virtualenvs/worldmodels/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\r\n  from ._conv import register_converters as _register_converters\r\nUsing TensorFlow backend.\r\nUsing TensorFlow backend.\r\nassigning the rank and nworkers 1 0\r\nassigning the rank and nworkers 1 0\r\n2018-04-18 18:30:08.922363: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\r\n2018-04-18 18:30:08.927218: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\r\n2018-04-18 18:30:09.084787: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2018-04-18 18:30:09.085010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties: \r\nname: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683\r\npciBusID: 0000:01:00.0\r\ntotalMemory: 10.91GiB freeMemory: 9.47GiB\r\n2018-04-18 18:30:09.085020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0\r\n2018-04-18 18:30:09.085433: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2018-04-18 18:30:09.085774: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties: \r\nname: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683\r\npciBusID: 0000:01:00.0\r\ntotalMemory: 10.91GiB freeMemory: 9.47GiB\r\n2018-04-18 18:30:09.085783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0\r\n2018-04-18 18:30:09.224535: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2018-04-18 18:30:09.224562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917]      0 \r\n2018-04-18 18:30:09.224566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0:   N \r\n2018-04-18 18:30:09.224701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9110 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)\r\n2018-04-18 18:30:09.228563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2018-04-18 18:30:09.228578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917]      0 \r\n2018-04-18 18:30:09.228597: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0:   N \r\n2018-04-18 18:30:09.229018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 253 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)\r\nsize of model 867\r\nsize of model 867\r\n(1,2mirr1)-aCMA-ES (mu_w=1.0,w_1=100%) in dimension 867 (seed=205998, Wed Apr 18 18:30:09 2018)\r\n('process', 0, 'out of total ', 1, 'started')\r\n('training', 'car_racing')\r\n('population', 2)\r\n('num_worker', 1)\r\n('num_worker_trial', 2)\r\n('num_episode', 16)\r\n('max_length', 1000)\r\nWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\r\nTraceback (most recent call last):\r\n  File \"05_train_controller.py\", line 461, in <module>\r\n    main(args)\r\n  File \"05_train_controller.py\", line 410, in main\r\n    master()\r\n  File \"05_train_controller.py\", line 319, in master\r\n    send_packets_to_slaves(packet_list)\r\n  File \"05_train_controller.py\", line 233, in send_packets_to_slaves\r\n    assert len(packet_list) == num_worker-1\r\nAssertionError\r\n(1,2mirr1)-aCMA-ES (mu_w=1.0,w_1=100%) in dimension 867 (seed=250130, Wed Apr 18 18:30:09 2018)\r\n('process', 0, 'out of total ', 1, 'started')\r\n('training', 'car_racing')\r\n('population', 2)\r\n('num_worker', 1)\r\n('num_worker_trial', 2)\r\n('num_episode', 16)\r\n('max_length', 1000)\r\nWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\r\nTraceback (most recent call last):\r\n  File \"05_train_controller.py\", line 461, in <module>\r\n    main(args)\r\n  File \"05_train_controller.py\", line 410, in main\r\n    master()\r\n  File \"05_train_controller.py\", line 319, in master\r\n    send_packets_to_slaves(packet_list)\r\n  File \"05_train_controller.py\", line 233, in send_packets_to_slaves\r\n    assert len(packet_list) == num_worker-1\r\nAssertionError\r\nTraceback (most recent call last):\r\n  File \"05_train_controller.py\", line 460, in <module>\r\n    if \"parent\" == mpi_fork(args.num_worker+1): os.exit()\r\n  File \"05_train_controller.py\", line 429, in mpi_fork\r\n    subprocess.check_call([\"mpirun\", \"-np\", str(n), sys.executable] +['-u']+ sys.argv, env=env)\r\n  File \"/usr/lib/python3.5/subprocess.py\", line 581, in check_call\r\n    raise CalledProcessError(retcode, cmd)\r\nsubprocess.CalledProcessError: Command '['mpirun', '-np', '2', '/home/boucher/.virtualenvs/worldmodels/bin/python', '-u', '05_train_controller.py', 'car_racing', '--num_worker', '1', '--num_worker_trial', '2', '--num_episode', '16', '--max_length', '1000', '--eval_steps', '25']' returned non-zero exit status 1\r\n```\r\n\r\nNot exactly sure what the problem is here. I had to lower the number of workers to run on my gpu without running out of memory, but playing around with that number and the num_worker_trial I wasn't able to get it past this error."}}
{"actor":{"display_login":"boucher"}}