{"action":"created","issue":{"url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/740","repository_url":"https://api.github.com/repos/Unity-Technologies/ml-agents","labels_url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/740/labels{/name}","comments_url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/740/comments","events_url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/740/events","html_url":"https://github.com/Unity-Technologies/ml-agents/issues/740","id":322124991,"number":740,"title":"Is it possible to train a brain using Behavioral Cloning training so that the brain can get the idea of what it has to do then train using PPO so that it can get better and faster?","user":{"login":"djdoggy99","id":25078678,"avatar_url":"https://avatars2.githubusercontent.com/u/25078678?v=4","gravatar_id":"","url":"https://api.github.com/users/djdoggy99","html_url":"https://github.com/djdoggy99","followers_url":"https://api.github.com/users/djdoggy99/followers","following_url":"https://api.github.com/users/djdoggy99/following{/other_user}","gists_url":"https://api.github.com/users/djdoggy99/gists{/gist_id}","starred_url":"https://api.github.com/users/djdoggy99/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djdoggy99/subscriptions","organizations_url":"https://api.github.com/users/djdoggy99/orgs","repos_url":"https://api.github.com/users/djdoggy99/repos","events_url":"https://api.github.com/users/djdoggy99/events{/privacy}","received_events_url":"https://api.github.com/users/djdoggy99/received_events","type":"User","site_admin":false},"labels":[{"id":687381587,"url":"https://api.github.com/repos/Unity-Technologies/ml-agents/labels/help-wanted","name":"help-wanted","color":"33aa3f","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-10T23:57:33Z","updated_at":"2018-05-12T11:17:28Z","closed_at":null,"author_association":"NONE","body":"I am trying to train an ai to drive a car when using ppo the ai just sits at the starting position the whole time spamming all the different buttons i have an idea why and this is because i am using a check point system for the rewards therefore its not getting a reward i then used Behavioural Cloning training this worked and the ai was able to go forwards and turn the corner the only problem was it was slow and not very efficient. I am wondering is there a way to train the brain using ppo after it has already been trained using behavioural cloning."},"comment":{"url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/comments/388548097","html_url":"https://github.com/Unity-Technologies/ml-agents/issues/740#issuecomment-388548097","issue_url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/740","id":388548097,"user":{"login":"mbaske","id":38779068,"avatar_url":"https://avatars2.githubusercontent.com/u/38779068?v=4","gravatar_id":"","url":"https://api.github.com/users/mbaske","html_url":"https://github.com/mbaske","followers_url":"https://api.github.com/users/mbaske/followers","following_url":"https://api.github.com/users/mbaske/following{/other_user}","gists_url":"https://api.github.com/users/mbaske/gists{/gist_id}","starred_url":"https://api.github.com/users/mbaske/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbaske/subscriptions","organizations_url":"https://api.github.com/users/mbaske/orgs","repos_url":"https://api.github.com/users/mbaske/repos","events_url":"https://api.github.com/users/mbaske/events{/privacy}","received_events_url":"https://api.github.com/users/mbaske/received_events","type":"User","site_admin":false},"created_at":"2018-05-12T11:17:28Z","updated_at":"2018-05-12T11:17:28Z","author_association":"NONE","body":"Such a feature would be high on my wish list too. I know the point of RL is to not make assumptions about HOW an agent reaches its goals. Nevertheless, sometimes you'd like to implement some basic actions quickly, before training more complex and flexible behavior on top of that. A workaround I have experimented with, is to train a PPO agent with a heuristic one first. Assuming the two agents face identical situations, the PPO agent receives its observations from the heuristic one. After invoking actions on the PPO agent, I accumulate the deltas of both agents' observations and set that as a negative reward. Which eventually results in the PPO agent mimicking the heuristic's behavior. I can then reuse the PPO model for further training.\r\n\r\nHere's how my PPO agent is set up:\r\n1) get a reference to the heuristic agent\r\n2) In CollectObservations, AddVectorObs(heuristicAgent.observations)\r\n3) AgentAction\r\n4) SetReward(-abs(heuristicAgent.observations - ppoAgent.observations))\r\n"}}
{"actor":{"display_login":"mbaske"}}