- I wasn't fully clear about the repeat/remember example in Section 4. I understand that the unrolled reverse computation of a TBPTT of an exactly reversible model for the repeat task is equivalent to the forward pass of a regular model for the remember task, but aren't they still quite different in major ways? First, are they really equivalent in terms of their gradient updates? In the end, they draw two different computation graphs? Second, at *test time*, the former is not auto-regressive (i.e., it uses the given input sequence) whereas the latter is. Maybe I'm missing something simple, but a more careful explanation of the example would be helpful. Also a minor issue: why are an NF-RevGRU and an LSTM compared in Appendix A? Shouldn't an NF-RevLSTM be used for a fairer comparison? - I'm not familiar with the algorithm of Maclaurin et al., so it's difficult to get much out of the description of Algorithm 1 other than its mechanics. A review/justification of the algorithm may make the paper more self-contained. - As the paper acknowledges, the reversible version has a much higher computational cost during training (2-3 times slower). Given how cheap memory is, it remains to be seen how actually practical this approach is. OTHER COMMENTS - It'd still be useful to include the perplexity/BLEU scores of a NF-Rev{GRU, LSTM} just to verify that the gating mechanism is indeed necessary. - More details on using attention would be useful, perhaps as an extra appendix.
- More details on using attention would be useful, perhaps as an extra appendix.