{"action":"opened","issue":{"url":"https://api.github.com/repos/openai/atari-py/issues/28","repository_url":"https://api.github.com/repos/openai/atari-py","labels_url":"https://api.github.com/repos/openai/atari-py/issues/28/labels{/name}","comments_url":"https://api.github.com/repos/openai/atari-py/issues/28/comments","events_url":"https://api.github.com/repos/openai/atari-py/issues/28/events","html_url":"https://github.com/openai/atari-py/issues/28","id":326703117,"number":28,"title":"The clone/restoreState() actually includes the pseudorandomness of Atari.","user":{"login":"YuhangSong","id":28839288,"avatar_url":"https://avatars1.githubusercontent.com/u/28839288?v=4","gravatar_id":"","url":"https://api.github.com/users/YuhangSong","html_url":"https://github.com/YuhangSong","followers_url":"https://api.github.com/users/YuhangSong/followers","following_url":"https://api.github.com/users/YuhangSong/following{/other_user}","gists_url":"https://api.github.com/users/YuhangSong/gists{/gist_id}","starred_url":"https://api.github.com/users/YuhangSong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/YuhangSong/subscriptions","organizations_url":"https://api.github.com/users/YuhangSong/orgs","repos_url":"https://api.github.com/users/YuhangSong/repos","events_url":"https://api.github.com/users/YuhangSong/events{/privacy}","received_events_url":"https://api.github.com/users/YuhangSong/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-26T02:32:35Z","updated_at":"2018-05-26T02:32:35Z","closed_at":null,"author_association":"NONE","body":"The document said that clone/restoreState() **doesn't include** pseudorandomness. However, the fact is that clone/restoreState() **doesn't include** the _pseudorandomness of repeat_action_, but it **includes** the _pseudorandomness of Atari_.\r\n\r\nThe code to verify above fact is as below:\r\n```\r\nimport numpy as np\r\nimport atari_py\r\n\r\ntest = 'loadROM'\r\n# test = 'restoreState'\r\n\r\nframe_skip = 4\r\nbunch = 200\r\nsequence = 500\r\n\r\ndef main():\r\n    result = {\r\n        'name':[],\r\n        'grouped_num':[],\r\n        'distribution':[],\r\n    }\r\n\r\n    # game_list = ['air_raid-n', 'alien', 'amidar', 'assault', 'asterix', 'asteroids', 'atlantis']\r\n    # game_list = ['bank_heist', 'battle_zone', 'beam_rider', 'berzerk-n', 'bowling', 'boxing', 'breakout', 'carnival-n']\r\n    # game_list = ['centipede', 'chopper_command', 'crazy_climber', 'demon_attack', 'double_dunk']\r\n    # game_list = ['elevator_action-n', 'enduro', 'fishing_derby', 'freeway', 'frostbite', 'gopher', 'gravitar']\r\n    # game_list = ['hero', 'ice_hockey', 'jamesbond', 'journey_escape-n', 'kangaroo', 'krull', 'kung_fu_master']\r\n    # game_list = ['montezuma_revenge-n', 'ms_pacman', 'name_this_game', 'phoenix-n', 'pitfall-n', 'pong', 'pooyan-n']\r\n    # game_list = ['private_eye', 'qbert', 'riverraid', 'road_runner', 'robotank', 'seaquest', 'skiing-n']\r\n    # game_list = ['solaris-n', 'space_invaders', 'star_gunner', 'tennis', 'time_pilot', 'tutankham', 'up_n_down']\r\n    # game_list = ['venture', 'video_pinball', 'wizard_of_wor', 'yars_revenge-n', 'zaxxon']\r\n\r\n    game_list = ['pong', 'assault','ms_pacman']\r\n\r\n    for game in game_list:\r\n\r\n        if '-n' in game:\r\n            '''games that are not in the nature DQN list'''\r\n            continue\r\n\r\n        game_path = atari_py.get_game_path(game)\r\n\r\n        env_father = atari_py.ALEInterface()\r\n        env_father.setFloat('repeat_action_probability'.encode('utf-8'), 0.0)\r\n        env_father.setInt(b'random_seed', 3)\r\n        env_father.loadROM(game_path)\r\n        env_father.reset_game()\r\n\r\n        if test in ['restoreState']:\r\n            state_after_reset = env_father.cloneState()\r\n\r\n        '''generate a sequence of actions'''\r\n        action_sequence = np.random.randint(\r\n            len(env_father.getMinimalActionSet()),\r\n            size = sequence,\r\n        )\r\n\r\n        bunch_obs = []\r\n        distribution = []\r\n        samples = []\r\n        for bunch_i in range(bunch):\r\n\r\n            env_temp = atari_py.ALEInterface()\r\n            env_temp.setFloat('repeat_action_probability'.encode('utf-8'), 0.0)\r\n            env_temp.setInt(b'random_seed', bunch_i)\r\n            if test in ['loadROM']:\r\n                env_temp.loadROM(game_path)\r\n                env_temp.reset_game()\r\n            elif test in ['restoreState']:\r\n                env_temp.loadROM(game_path) # restoreState without calling loadROM first will cause Segmentation fault (core dumped)\r\n                env_temp.restoreState(state_after_reset)\r\n\r\n            # just to make sure\r\n            env_temp.setFloat('repeat_action_probability'.encode('utf-8'), 0.0)\r\n            env_temp.setInt(b'random_seed', bunch_i)\r\n\r\n            for sequence_i in range(sequence):\r\n                for frame_skip_i in range(frame_skip):\r\n                    env_temp.act(\r\n                        env_father.getMinimalActionSet()[\r\n                            action_sequence[sequence_i]\r\n                        ]\r\n                    )\r\n                if env_temp.game_over():\r\n                    env_temp.reset_game()\r\n\r\n            obs = env_temp.getScreenRGB2()\r\n\r\n            samples += [obs]\r\n            found_at_bunch = -1\r\n            if_has_identical_one = False\r\n            max_value = 0\r\n            for bunch_obs_i in range(len(bunch_obs)):\r\n                obs_in_bunch = bunch_obs[bunch_obs_i]\r\n                max_value = np.max(\r\n                    np.abs(\r\n                        obs-obs_in_bunch\r\n                    )\r\n                )\r\n                if max_value < 1:\r\n                    found_at_bunch = bunch_obs_i\r\n                    if_has_identical_one = True\r\n                    distribution[found_at_bunch] += 1\r\n                    break\r\n\r\n            if if_has_identical_one is False:\r\n                bunch_obs += [obs]\r\n                distribution += [1]\r\n\r\n        grouped_num = len(bunch_obs)\r\n        print('game:{} grouped_num:{} distribution:{}'.format(\r\n            game,\r\n            grouped_num,\r\n            distribution,\r\n        ))\r\n\r\nif __name__ == \"__main__\":\r\n    main()\r\n```\r\n\r\nSpecifically, when set ```test = 'loadROM'```, it tests the stochasticity produced by\r\n```\r\nsetInt(b'random_seed')\r\nloadROM()\r\n```\r\nThe results are:\r\n```\r\ngame:pong grouped_num:1 distribution:[200]\r\ngame:assault grouped_num:57 distribution:[4, 4, 6, 3, 8, 9, 2, 5, 3, 2, 5, 5, 5, 4, 5, 4, 2, 4, 6, 3, 1, 4, 1, 5, 5, 1, 2, 1, 3, 6, 3, 4, 2, 3, 5, 3, 4, 6, 3, 5, 4, 3, 2, 2, 3, 4, 2, 1, 3, 4, 3, 1, 2, 2, 5, 2, 1]\r\ngame:ms_pacman grouped_num:1 distribution:[200]\r\n```\r\n\r\nWhen set ```test = 'restoreState'```, it tests the stochasticity produced by ```clone/restoreState()```. The results are:\r\n```\r\ngame:pong grouped_num:1 distribution:[200]\r\ngame:assault grouped_num:1 distribution:[200]\r\ngame:ms_pacman grouped_num:1 distribution:[200]\r\n```\r\n\r\nAs you can see, we set repeat_action_probability to zero all the time since we do not care about this kind of stochasticity. The results shows that ```clone/restoreState()``` actually includes the pseudorandomness of Atari. Thus, we actually **do not** have a function that really exclude the pseudorandomness so that we can use it for planning.\r\n\r\nAny ideas? Is there any way we can have a function that clone/restore state of Atari without pseudorandomness so that we can use it for planning?"}}
{"actor":{"display_login":"YuhangSong"}}