a. The paper lacks structure and clarity
b. The paper lacks a more qualitative study of the model:
it would be interesting to see what layers the layer-wise attention mechanism attends to.
it would be great to understand how this model uses the latent variables, for instance by measuring the KL divergence at each layer, as done in the previous work (LVAE, BIVA) (connection to "posterior collapse").
c. Experiments are limited to CIFAR-10, larger scaler experiments (i.e. ImageNet) would be beneficial to the paper. It is not guaranteed that such an architecture would translate in the same gains for larger datasets (i.e. ImageNet).
3. Clarification needed:
a. Table 5: I interpreted the column "non-local layers" as using "attention across layers", I hope I was right. The nomenclature needs to be improved.
b. Is the layer-wise attention mechanism specific to deep VAEs, or can it be more generally applied to ResNet architectures?
c. Section 2.3 (paragraph cited bellow): I get the idea, but unless demonstrated, this remains a hypothesis.
"... in practice the network may no longer respect the factorization of the prior p ( z ) = ∏ l p ( z l ∣ z < l )
leading to diminished performance gains as shown in Table 1":
4. Minor comments / suggestions
a. The main contributions are introducing two types of attention for deep VAEs, it might help to describe them in a separate section, and only then describe the generative and inference models. Right now the description of the layer-wise attention mechanism is scattered across sections 2.3 and 2.4.
b. tricks like normalisation or feature scaling could be referenced in a separate section.
c. eq8: you might want to cite ReZero [1] here
d. Fig 1. a: the lack of arrows going from the activations ( h l , k l q )
to the attention block ( A ( . . . ) )
was confusing on the first read
e. It would be better practice to report likelihoods for multiple random seeds
f. Typo in section 2.1: "both q ( z x ) and p ( x )
are fully factorized gaussian..." -> "both q ( z x ) and p ( z )
are fully factorized gaussian..."
[1] Bachlechner, T., Prasad Majumder, B., Mao, H. H., Cottrell, G. W., and McAuley, J., “ReZero is All You Need: Fast Convergence at Large Depth”, <i>arXiv e-prints</i>, 2020.
4. Minor comments / suggestions a. The main contributions are introducing two types of attention for deep VAEs, it might help to describe them in a separate section, and only then describe the generative and inference models. Right now the description of the layer-wise attention mechanism is scattered across sections 2.3 and 2.4. b. tricks like normalisation or feature scaling could be referenced in a separate section. c.