{"action":"created","issue":{"url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/608","repository_url":"https://api.github.com/repos/Unity-Technologies/ml-agents","labels_url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/608/labels{/name}","comments_url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/608/comments","events_url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/608/events","html_url":"https://github.com/Unity-Technologies/ml-agents/issues/608","id":312785113,"number":608,"title":"Looking for general advise after a hack-week project using ml-agents","user":{"login":"ChrisCF","id":21068547,"avatar_url":"https://avatars1.githubusercontent.com/u/21068547?v=4","gravatar_id":"","url":"https://api.github.com/users/ChrisCF","html_url":"https://github.com/ChrisCF","followers_url":"https://api.github.com/users/ChrisCF/followers","following_url":"https://api.github.com/users/ChrisCF/following{/other_user}","gists_url":"https://api.github.com/users/ChrisCF/gists{/gist_id}","starred_url":"https://api.github.com/users/ChrisCF/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ChrisCF/subscriptions","organizations_url":"https://api.github.com/users/ChrisCF/orgs","repos_url":"https://api.github.com/users/ChrisCF/repos","events_url":"https://api.github.com/users/ChrisCF/events{/privacy}","received_events_url":"https://api.github.com/users/ChrisCF/received_events","type":"User","site_admin":false},"labels":[{"id":687381590,"url":"https://api.github.com/repos/Unity-Technologies/ml-agents/labels/discussion","name":"discussion","color":"d4c5f9","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-04-10T06:11:17Z","updated_at":"2018-04-18T01:59:12Z","closed_at":null,"author_association":"NONE","body":"We had a Hack week in our office just before Easter and my team tried out ml-agents in one of our games. This was our first adventure in machine learning so we were working off the documentation, a bit of theoretical knowledge from some other articles and presentations. We ended up with a number of (possibly quite naive) questions I was hoping someone might be able to help us out with.  **(skip to the last paragraph for the tldr; questions)** The middle section is the details of what we tried. (Note: the various imgur images are animated gifs that are too big to embed here).\r\n\r\nThe game we chose to try this on was Ninja Tobu. It's a 2D mobile game where the user controls a ninja by swiping across the screen to make the ninja jump. The aim is either to reach a goal at the end of the level or, in endless mode, to climb as high as possible through an infinite level. There are various spiked surfaces and NPCs it's not safe to land on. The sections of the infinite mode vary in numbers of hazards and there is a set of spikes below you that force you to keep climbing nearly constantly. It's not the simplest game we had available but we wanted to push things a bit.\r\n\r\nStep one was refactoring some of our code so we could try imitation learning and running multiple agents at once (we did this before our hack week started - tech debt reduction). \r\n\r\nThen at the start of the week we setup an initial set of observations, actions and rewards and tried letting the agent go at it in an empty level. We were hoping that, in this basic situation, it would fairly quickly get to the point where it could climb forever so we could start adding in some obstacles.\r\n\r\n**Observations**: Characters x/y velocity (normalised based on max possible character velocity) and 8 raycasts evenly spaced radiating out around the character each producing: x/y velocity (normalised) of thing hit, vector to the thing from the character (normalised relative to max raycast distance) and a 0/1 value for whether it was something the character could land on or not.\r\n![image](https://user-images.githubusercontent.com/21068547/38291208-bef9f7a0-3832-11e8-8624-95b311a75b75.png)\r\n\r\n**Rewards**: Penalty for dying of -1. Penalty for being lower than previous height -0.005. Reward proportional to height gained, 1 for gaining the max possible height for a movement.\r\n\r\n**Action Vector**:\r\nMouse down/up _(bool mouseButtonDown = mouseAction > 0.5f;)_\r\nX Screen Position (normalised 0-1)\r\nY Screen Position (normalised 0-1)\r\n\r\nWe basically gave the agent the exact input method a player would use. I was going to call this our first issue, but it did seem to be figuring it out, it was just taking a long time to do so.  The following gif is after ~4 hours (left side), ~8 hours (right side) https://imgur.com/MZkF9Wn\r\nYou might notice some mid air direction changes, this is because ninjas are allowed a single mid-air jump between each jump from a surface. The right image is actually the best double jumping we saw all week. But this rate of progress was a bit slow for our tastes. 8 hours and the agent wouldn't even have been capable of reaching the first obstacle in a normal level.\r\n\r\nSo we started working towards imitation learning and playing around with rewards in the first version while we were making those changes. We simplified the rewards. -1 for dying, +0.1 for moving and +0.01 for getting higher; and we tried more complex observations to see if they would help. None of these changes made a big difference to the first version.\r\n\r\nOnce we got imitation learning setup we immediately ran into more problems related to the agents actions.  https://imgur.com/9vnK600  The agent really wasn't getting the touch up/down. So we decided to modify the input system so that the agent was dealing with jump vectors. x/y normalised strength, anything under abs(0.2) was treated as no jump. 1 was the biggest jump possible. (Was trying to encode jump/no jump  information like this OK?) This worked much better and the agent was quickly able to understand how to jump. This worked for standalone reinforcement learning as well with the agent learning the basics of movement in a couple of minutes rather than over several hours. https://imgur.com/gfu9NP6 Progress! Given the faster training of the reinforcement method with these changes we ended up focusing on that, because, we thought that was more interesting than the imitation learning and it was going much faster with the changes. So it seemed viable.\r\n\r\nReinforcement learning now kept getting us rapidly stuck with the agent wall running and then, sitting stuck in corners.\r\n![wallbug](https://user-images.githubusercontent.com/21068547/38536883-a828a8a8-3cdf-11e8-8a74-50f86a9eb29f.gif)\r\n This turned out to be an issue with the game. It's possible to swipe upwards and towards a wall you are hanging on and do a tiny hop up the wall. The agent kept deciding that these micro-hops were the best way to climb, but when it hit an overhang it would get stuck and just sit there till the spikes caught up and killed it. Huge caveat here, our longest tests were probably only 2-3 hours, and a lot of the time we were stopping and making changes after 30 minutes when we started seeing the same wall hugging behaviour. Later experience suggested that we maybe weren't giving the agent enough time to overcome these situations.\r\n\r\nSo we fixed the wall hugging and this led to much better behaviour. We went through a \r\nday of short training runs, playing with hyper parameters and trying various combinations of observations and rewards again. I think this day was mostly wasted given how little time we were training each iteration for. At the end of that day things were looking like a bit of a bust, with presentations on projects due the next day. We setup several different agents running with different configs and left them overnight. In the morning after about 15 hours we had the closest thing to success we saw all week (for 2 of the setups). These particular setups had the simplified reward set with default hyper parameters running in simplified environments. One was a completely empty environment and one was using 3 easy level chunks with basic obstacles and no lethal hazards (**The best agent:** https://imgur.com/a/lyFSt)\r\nTensorboard: \r\n![image](https://user-images.githubusercontent.com/21068547/38291461-f0301aba-3833-11e8-9f43-df54017586cd.png)\r\n\r\nInterestingly, this best case agent became terrible about 30 minutes later and is you can see below it was generally all over the show.\r\n![allovertheplace](https://user-images.githubusercontent.com/21068547/38537424-af532006-3ce2-11e8-80fe-8010378407f3.png)\r\n\r\nThe things that stuck out for me as potential problem indicators (based off reading the ml-agents docs) were: Entropy increasing and info/policy loss magnitude averaging fairly similar levels (to my eyeball). (Plus the erratic cumulative reward and value estimate.) But after the time we'd already spent trying to tweak hyper parameters and seeing every change making things worse (except increasing the hidden_units: this seemed to produce the same results but faster) we weren't sure how to proceed to improve the training and we were out of time (we were only running 30 minute runs with modified hyper parameters. Perhaps not enough to see a positive difference?)\r\n\r\nSome other things we tried along the way:\r\nA normalised jump count. 0, 0.5, 1. Indicating whether the agent was resting, on it's first jump or on its 3rd jump. Should this have been encoded in a one-hot fashion? What if more jumps were allowed (there is a power-up in the full game that provides extra air jumps)\r\nStacking observations - This went very poorly for some reason we didn't have time to investigate. With a few sets of stacked observations the Agent ended up jumping once towards a wall and then just sitting there, very occasionally he'd jump back towards the start point after getting to the wall. Both courses of action are fatal in a few seconds.\r\nDoubling the number of ray casts. \r\nRecurrent, though we didn't expect that to work well given the use of a continuous action space (and it didn't).\r\nWe wanted to build up a curriculum and slowly add more difficult levels and features so the agent could learn to play the full game. But we ended up a fair way off getting to that point.\r\nWe didn't look at camera based observations because that seemed like it would be much more complicated and therefore slower to train(?)\r\n\r\n**Questions:**\r\n\r\n1. Are their any rules of thumb around how long training should take given the size of the observation and action spaces? And therefore were our shorter 30 minute runs a waste of time? \r\n2. The 15+ hour training run we did seems like a long time when it takes example projects much less time to train and some of them have more observations than we do.  Does this indicate a problem or is it due to the continuous action space or something else?\r\n3. Is there such a thing as a guide to troubleshooting for PPO training using tensorboard?  (Beyond the documentation provided in this repo)\r\n4. In the likely event that there isn't. Can anyone recommend good resources for more introductory level information on ML, preferably with a focus on game agents rather than image classification or other areas? Or is it all just a dark art? The docs were a good starting point, but our conclusion after a few days using ml-agents is that we need significantly more base knowledge to be able to get value out of it in the kind of real world use cases we have (Maybe our expectations are too high). We just can't tell if our fundamental observations and rewards are OK and the hyper parameters need work or the other way around or if something totally different is going on or if we are doing everything horribly wrong.\r\n5. Any suggestions about how to proceed based on the last tensorboard image (or any other info I've provided) would be great. We're still keen to improve things when we get time. "},"comment":{"url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/comments/382220963","html_url":"https://github.com/Unity-Technologies/ml-agents/issues/608#issuecomment-382220963","issue_url":"https://api.github.com/repos/Unity-Technologies/ml-agents/issues/608","id":382220963,"user":{"login":"MichaelTSakai","id":21374129,"avatar_url":"https://avatars1.githubusercontent.com/u/21374129?v=4","gravatar_id":"","url":"https://api.github.com/users/MichaelTSakai","html_url":"https://github.com/MichaelTSakai","followers_url":"https://api.github.com/users/MichaelTSakai/followers","following_url":"https://api.github.com/users/MichaelTSakai/following{/other_user}","gists_url":"https://api.github.com/users/MichaelTSakai/gists{/gist_id}","starred_url":"https://api.github.com/users/MichaelTSakai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MichaelTSakai/subscriptions","organizations_url":"https://api.github.com/users/MichaelTSakai/orgs","repos_url":"https://api.github.com/users/MichaelTSakai/repos","events_url":"https://api.github.com/users/MichaelTSakai/events{/privacy}","received_events_url":"https://api.github.com/users/MichaelTSakai/received_events","type":"User","site_admin":false},"created_at":"2018-04-18T01:59:12Z","updated_at":"2018-04-18T01:59:12Z","author_association":"NONE","body":"Hello ChrisCF, \r\n\r\nwhat did you do to tweak the entropy? I'm having the same issues where entropy is increasing and am looking for some advice. Did you increase it or decrease it?"}}
{"actor":{"display_login":"MichaelTSakai"}}