during evaluation [^4]. We used $\beta=0.5,\ell=0.6$ in all the experiments. The results are averaged over 5 random seeds, shown alongside error bars (std/3).

\[fig:egg\]

**The Egg Quest:** In this quest, the agent’s goal is to find and open the jewel-encrusted egg, hidden on a tree in the forest. The agent is awarded 100 points upon successful completion of this task. We experimented with the AE-DQN (blue) agent and a vanilla DQN agent (green) in this quest (Figure \[fig:egg\]). The action set in this quest is composed of two subsets. A fixed subset of $9$ actions that allow it to complete the Egg Quest like *navigate* (south, east etc.) *open* an item and *fight*; And a second subset consists of $N_{\text{Take}}$ *“take”* actions for possible objects in the game. The “take” actions correspond to taking a single object and include objects that need to be collected to complete quests, as well as other irrelevant objects from the game dictionary. We used two versions of this action set, $A_1$ with $N_{\text{Take}}=200$ and $A_2$ with $N_{\text{Take}}=300$. **Robustness to hyperparameter tuning:** We can see that for $A_1$, with T=100, (Figure \[fig:egg\]$a$), and for $A_2$, with T=200, (Figure \[fig:egg\]$c$) *Both* agents solve the task well. 