1.	Related Work: As the available space allows it, the paper would benefit from a more detailed discussion of related work, by not only describing the related works, but also discussing the differences to the presented work.
2.	Qualitative results: To underline the success of the work, the paper should include some qualitative examples, comparing its generated sentences to the ones by related work.
3.	Experimental setup: For the Coco image cations, the paper does not rely on the official training/validation/test split used in the COCO captioning challenge.
3.1.	Why do the authors not use the entire training set?
3.2.	It would be important for the automatic evaluation to report results using the evaluation sever and report numbers on the blind test set (for the human eval it is fine to use the validation set). Conclusion:
I hope the authors will include the coco caption evaluation server results in the rebuttal and final version as well as several qualitative results.
Given the novelty of the approach and strong experiments without major flaws I recommend accepting the paper.
It would be interesting if the authors would comment on which problems and how their approach can be applied to non-sequence problems.
1. Related Work: As the available space allows it, the paper would benefit from a more detailed discussion of related work, by not only describing the related works, but also discussing the differences to the presented work.