Weakness main comments: • what is the advantage of using a differentiable LP layer (GNN and a LP solver) as a high-level policy, shown in Eq. 10?
– compare it to [1] that considers the LP optimization layer as a meta-environment?
– compare it to an explicit task assignment protocol (e.g. not implicit).
E.g. a high-level policy that directly outputs task weightings instead of the intermediary C matrix?
• How does this method address sparse reward problems in a better way? From the experiments, this does not support well. in practice, the proposed method requires sub-task-specific rewards to be specified, which would be similar to providing a dense reward signal that includes rewards for reaching sub-goals. If given the sum of low-level reward as the global reward, will the other methods (Qmix) solve the sparse-reward tasks as well?
minor comments: • It is hard to determine whether the solution to the matching problem (learned agent-task score matrix C) optimized by LP is achieving global perspective over the learning process.
• When the lower-level policies are also trained online, the learning could be unstable. Details on how to solve the instability in hierarchical learning are missing.
• What is the effect of the use of hand-defined tasks on performance? what is the effect of the algorithm itself? maybe do an ablation study.
• Section 5.2 ”training low-level actor-critic” should be put in the main text.
[1] Carion N, Usunier N, Synnaeve G, et al. A structured prediction approach for generalization in cooperative multi-agent reinforcement learning[J]. Advances in neural information processing systems, 2019, 32: 8130-8140.
• How does this method address sparse reward problems in a better way? From the experiments, this does not support well. in practice, the proposed method requires sub-task-specific rewards to be specified, which would be similar to providing a dense reward signal that includes rewards for reaching sub-goals. If given the sum of low-level reward as the global reward, will the other methods (Qmix) solve the sparse-reward tasks as well? minor comments: