1. The approach of decomposing video representation into spatial and temporal representation for efficient and effective spatio-temporal modelling is a general idea in video understanding. I'm not going to blame using this in large video language models, however, I think proper credit and literature reviews should be included. For example, TimeSFormer[1], Uniformer[2], Dual-AI[3] and others using transformer for video recognition.
2. Training specialized and efficient video QFormer has been explored and utilized by UMT [4] and VideoChat2 [5]. Please clarify the difference and include them in literature reviews.
2. Comfusing attentive pooling module architecture. It seems the temporal representation $v_{t}$ is derived from spatial queries $Q_{s}$ attending to a set of frame features (with T as batch dimension). It means the spatial queries can only attend to in-frame content. This is doubtable why the representation is called temporal representation.
3. Training data: what specific data are used from training? Please provide details of how many videos from what dataset and how you make sampling. This is critical for reproduction and measure the method effectiveness.
4. Experiments - Comparison fairness: More latest methods should be included in comparison, especially those with similar motivations, e.g., VideoChat2.
5. Experiments - Image benchmark: As image dataset is used, it would be great to show the performance variance after such ST QFormer tuning. Also compared to normal QFormer.
6. Experiments - Video summarization: there are some new good benchmarks, like Shot2Story ranging from different topics and using only text and frames modalities. This is not mandatory, but it should be good to include.
7. Experiments - Ablation - missing components: There should be experiments and explanation regarding the different queries used in spatio-temporal representation, i.e., spatial, temporal and summary. That is the key difference to VideoChatGPT and other works. What if only have spatial one, or temporal and summary one?
8. Experiments - Ablation - metric: for abaltions, I suggest to use QA benchmarks for experiments rather than captioning benchmark. When things come to LLM, the current captioning metrics such as B@4 and CIDEr might not be ideal to reflect model ability.
[1] Is Space-Time Attention All You Need for Video Understanding?
[2] UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning
[3] Dual-AI: Dual-Path Actor Interaction Learning for Group Activity Recognition
[4] MVBench
7. Experiments - Ablation - missing components: There should be experiments and explanation regarding the different queries used in spatio-temporal representation, i.e., spatial, temporal and summary. That is the key difference to VideoChatGPT and other works. What if only have spatial one, or temporal and summary one?