and strengths in their analysis with sufficient experimental detail, it is admirable, but they could provide more intuition why other methods do better than theirs.
The claims could be better supported. Some examples and questions(if I did not miss out anything)
Why using normalization is a problem for a network or a task (it can be thought as a part of cosine distance)? How would Barlow Twins perform if their invariance term is replaced with a euclidean distance?
Your method still uses 2048 as the batch size, I would not consider it as small. For example, Simclr uses examples in the same batch and its batch size changes between 256-8192. Most of the methods you mentioned need even much lower batch size.
You mentioned not sharing weights as an advantage, but you have shared weights in your results, except Table 4 in which the results degraded as you mentioned. What stops the other methods from using different weights? It should be possible even though they have covariance term between the embeddings, how much their performance would be affected compared with yours?
My intuition is that a proper design might be sufficient rather than separating variance terms.
- Do you have a demonstration or result related to your model collapsing less than other methods? In line 159, you mentioned gradients become 0 and collapse; it was a good point, is it commonly encountered, did you observe it in your experiments?
- I am also not convinced to the idea that the images and their augmentations need to be treated separately, they can be interchangeable.
- Variances of the results could be included to show the stability of the algorithms since it was another claim in the paper(although "collapsing" shows it partly, it is a biased criteria since the other methods are not designed for var/cov terms).
- How hard is it to balance these 3 terms?
- When someone thinks about gathering two batches from two networks and calculate the global batch covariance in this way; it includes both your terms and Barlow Twins terms. Can anything be said based on this observation, about which one is better and why? Significance:
Currently, the paper needs more solid intuition or analysis or better results to make an impact in my opinion. The changes compared with the prior work are minimal. Most of the ideas and problems in the paper are important, but they are already known.
The comparisons with the previous work is valuable to the field, they could maybe extend their experiments to the more of the mentioned methods or other variants.
The authors did a great job in presenting their work's limitations, their results in general not being better than the previous works and their extensive analysis(tables). If they did a better job in explaining the reasons/intuitions in a more solid way, or include some theory if there is any, I would be inclined to give an accept.
- Do you have a demonstration or result related to your model collapsing less than other methods? In line 159, you mentioned gradients become 0 and collapse; it was a good point, is it commonly encountered, did you observe it in your experiments?