Given how strong the first four sections (five pages) of the paper were, I was relatively disappointed in the experiments, which were somewhat light. Specifically: 1) While the authors' methods allow for learning a state-action-dependent weighting of the shaping rewards, it seemed to me possible that in all of the experiments presented, learning a *uniform* state-action-independent weighting would have sufficed. Moreover, since learning a state-action-independent weighting is much simpler (i.e. it is a single scalar), it may even outperform the authors' methods for the current experiments. Based on this, I would like to suggest the following: 1a) Could the authors provide visualizations of the state-action variation of their learnt weightings? They plot the average weight in some cases (Fig 1 and 3), but given Cartpole has such a small state-action space, it should be possible to visualize the variation. The specific question here is: do the weights vary much at all in these cases? 1b) Could the authors include a baseline of learnt state-action-*independent* weights? In other words, this model has a single parameter, replacing z_phi(s,a) with a single scalar z. This should be pretty easy to implement. The authors could take any (or all) of their existing gradient approximators and simply average them across all (s,a) in a batch to get the gradient w.r.t. z. 1c) Could the authors include an additional experiment that specifically benefits from learning state-action-*dependent* (so non-uniform) weights? Here is a simple example for Cartpole: the shaping reward f(s,a) is helpful for half the state space and unhelpful for the other half. The "halves" could be whether the pole orientation is in the left or right half. The helpful reward could be that from Section 5.1 while the unhelpful reward could be that from the first adaptability test in Section 5.3. 2) To me, the true power of the author's approach is not in learning to ignore bad rewards (just turn them off!) but to intelligently incorporate sort-of-useful-but-not-perfect rewards. This way a researcher can quickly hand design an ok shaping reward but then let the authors' method transform it into a good one. Thus, I was surprised the experiments focussed primarily on ignoring obviously bad rewards and upweighting obviously good rewards. In particular, the MuJoCo experiments would be more compelling if they included more than just a single unhelpful shaping reward. I think the authors could really demonstrate the usefulness of their method there by doing the following: hand design a roughly ok shaping reward for each task. For example, the torso velocity or head height off the ground for Humanoid-v2. Then apply the authors' method and show that it outperforms naive use of this shaping reward. 3) Although the authors discussed learning a shaping reward *from scratch* in the related work section, I was surprised that they did not included this as a baseline. One would like to see that their method, when provided with a decent shaping reward to start, can learn faster by leveraging this hand-crafted knowledge. Fortunately, it seems to me again very easy to implement a baseline like this within the author's framework: simply set f(s,a)=1 and use the authors' methods (perhaps also initializing z_phi(s,a)=0).
1) While the authors' methods allow for learning a state-action-dependent weighting of the shaping rewards, it seemed to me possible that in all of the experiments presented, learning a *uniform* state-action-independent weighting would have sufficed. Moreover, since learning a state-action-independent weighting is much simpler (i.e. it is a single scalar), it may even outperform the authors' methods for the current experiments. Based on this, I would like to suggest the following: 1a) Could the authors provide visualizations of the state-action variation of their learnt weightings? They plot the average weight in some cases (Fig 1 and