The problem formulation is incomplete. The paper does not define the safety properties expected from the RL agent.
- Lack of theoretical results. This paper provides only empirical results to support its claims.
- The results are presented in a convoluted way. In particular, the results disregard the safety violations of the agent in the first 1000 episodes. The reason for presenting the results in this way is unclear.
- The presentation of the DDPG-Lag as a constrained RL algorithm is imprecise, as it uses a fixed weight for the costs, which works as simple reward engineering. In general, with a Lagrangian relaxation, this weight should be adjusted online to ensure the accumulated cost stays below a predefined threshold [1].
- The evaluation in CMDPs is inconsistent. These approaches solve different problems where a predefined accumulated cost is allowed.
- Weak baseline. From the results in Figure 10, it is clear that Tabular Shield does not recognize any unsafe state-action pairs, making it an unsuitable baseline. This is not surprising considering how the state-action space is discretized. Perhaps it is necessary to finetune the discretization of this baseline. Alternatively, it would be more suitable to consider stronger baselines, such as the accumulating safety rules [2] **references**
- [1] Ray, A., Achiam, J., and Amodei, D. (2019). *Benchmarking safe exploration in deep reinforcement learning*. <https://github.com/openai/safety-gym>
- [2] Shperberg, S. S., Liu, B., Allievi, A., and Stone, P. (2022). A rule-based shield: Accumulating safety rules from catastrophic action effects. *CoLLAs*, 231–242. <https://proceedings.mlr.press/v199/shperberg22a.html>
- The results are presented in a convoluted way. In particular, the results disregard the safety violations of the agent in the first 1000 episodes. The reason for presenting the results in this way is unclear.