/ Major concerns: - It is difficult to evaluate whether the MovieQA result should be considered significant given that +10% gap exists between MemN2N on dataset with explicit answers (Task 1) and RBI + FP on dataset with other forms of supervision, especially Task 3. If I understood correctly, the different tasks are coming from the same data, but authors provide different forms of supervision. Also, Task 3 gives full supervision of the answers. Then I wonder why RBI + FP on task 3 (69%) is doing much worse than MemN2N on task 1 (80%). Is it because the supervision is presented in a more implicit way ("No, the answer is kitchen" instead of "kitchen")? - For RBI, they only train on rewarded actions. Then this means rewardless actions that get useful supervision (such as "No, the answer is Timothy Dalton." in Task 3) is ignored as well. I think this could be one significant factor that makes FP + RBI better than RBI alone. If not, I think the authors should provide stronger baseline than RBI (that is supervised by such feedback) to prove the usefulness of FP. Questions / Minor concerns: - For bAbI, it seems the model was only tested on single supporting fact dataset (Task 1 of bAbI). How about other tasks? - How is dialog dataset obtained from QA datasets? Are you using a few simple rules? - Lack of lexical / syntactic diversity of teacher feedback: assuming the teacher feedback was auto-generated, do you intend to turk the teacher feedback and / or generate a few different kinds of feedback (which is more real-life situation)? - How does other models than MemN2N do on MovieQA?
- For RBI, they only train on rewarded actions. Then this means rewardless actions that get useful supervision (such as "No, the answer is Timothy Dalton." in Task 3) is ignored as well. I think this could be one significant factor that makes FP + RBI better than RBI alone. If not, I think the authors should provide stronger baseline than RBI (that is supervised by such feedback) to prove the usefulness of FP. Questions / Minor concerns: