W1) The paper states that "In order to introduce epipolar constraints into attention-based feature matching while maintaining robustness to camera pose and calibration inaccuracies, we develop a Window-based Epipolar Transformer (WET), which matches reference pixels and source windows near the epipolar lines." It claims that it introduces "a window-based epipolar Transformer (WET) for enhancing patch-to-patch matching between the reference feature and corresponding windows near epipolar lines in source features". To me, taking a window around the epipolar line into account seems like an approximation to estimating the uncertainty region around the epipolar lines caused by inaccuracies in calibration and camera pose and then searching within this region (see [Förstner & Wrobel, Photogrammetric Computer Vision, Springer 2016] for a detailed derivation of how to estimate uncertainties). Is it really valid to claim this part of the proposed approach as novel?
W2) I am not sure how significant the results on the DTU dataset are: a) The difference with respect to the best performing methods is less than 0.1 mm (see Tab. 1). Is the ground truth sufficiently accurate enough that such a small difference is actually noticeable / measurable or is the difference due to noise or randomness in the training process? b) Similarly, there is little difference between the results reported for the ablation study in Tab. 4. Does the claim "It can be seen from the table that our proposed modules improve in both accuracy and completeness" really hold? Why not use another dataset for the ablation study, e.g., the training set of Tanks & Temples or ETH3D?
W3) I am not sure what is novel about the "novel geometric consistency loss (Geo Loss)". Looking at Eq. 10, it seems to simply combine a standard reprojection error in an image with a loss on the depth difference. I don't see how Eq. 10 provides a combination of both losses.
W4) While the paper discusses prior work in Sec. 2, there is mostly no mentioning on how the paper under review is related to these existing works. In my opinion, a related work section should explain the relation of prior work to the proposed approach. This is missing.
W5) There are multiple parts in the paper that are unclear to me: a) What is C in line 106? The term does not seem to be introduced. b) How are the hyperparameters in Sec. 4.1 chosen? Is their choice critical? c) Why not include UniMVSNet in Fig. 5, given that UniMVSNet also claims to generate denser point clouds (as does the paper under review)? d) Why use only N=5 images for DTU and not all available ones? e) Why is Eq. 9 a reprojection error? Eq. 9 measures the depth difference as a scalar and no projection into the image is involved. I don't see how any projection is involved in this loss.
Overall, I think this is a solid paper that presents a well-engineered pipeline that represents the current state-of-the-art on a challenging benchmark. While I raised multiple concerns, most of them should be easy to address. E.g., I don't think that removing the novelty claim from W1 would make the paper weaker. The main exception is the ablation study, where I believe that the DTU dataset is too easy to provide meaningful comparisons (the relatively small differences might be explained by randomness in the training process.
The following minor comments did not affect my recommendation:
References are missing for Pytorch and the Adam optimizer.
Post-rebuttal comments
Thank you for the detailed answers. Here are my comments to the last reply:
Q: Relationship to prior work.
Thank you very much, this addresses my concern.
A: Fig. 5 is not used to claim our method achieves the best performance among all the methods in terms of completeness, it actually indicates that our proposed method could help reconstruct complete results while keeping high accuracy (Tab. 1) compared with our baseline network [7] and the most relevant method [3]. In that context, we not only consider the quality of completeness but also the relevance to our method to perform comparison in Fig. 5.
As I understand lines 228-236 in the paper, in particular "The quantitative results of DTU evaluation set are summarized in Tab. 1, where Accuracy and Completeness are a pair of official evaluation metrics. Accuracy is the percentage of generated point clouds matched in the ground truth point clouds, while Completeness measures the opposite. Overall is the mean of Accuracy and Completeness. Compared with the other methods, our proposed method shows its capability for generating denser and more complete point clouds on textureless regions, which is visualized in Fig. 5.", the paper seems to claim that the proposed method generates denser point clouds. Maybe this could be clarified?
A: As a) nearly all the learning-based MVS methods (including ours) take the DTU as an important dataset for evaluation, b) the GT of DTU is approximately the most accurate GT we can obtain (compared with other datasets), c) the final results are the average across 22 test scans, we think that fewer errors could indicate better performance. However, your point about the accuracy of DTU GT is enlightening, and we think it's valuable future work.
This still does not address my concern. My question is whether the ground truth is accurate enough that we can be sure that the small differences between the different components really comes from improvements provided by adding components. In this context, stating that "the GT of DTU is approximately the most accurate GT we can obtain (compared with other datasets)" does not answer this question as, even though DTU has the most accurate GT, it might not be accurate enough to measure differences at this level of accuracy (0.05 mm difference). If the GT is not accurate enough to differentiate in the 0.05 mm range, then averaging over different test scans will not really help. That "nearly all the learning-based MVS methods (including ours) take the DTU as an important dataset for evaluation" does also not address this question. Since the paper claims improvements when using the different components and uses the results to validate the components, I do not think that answering the question whether the ground truth is accurate enough to make these claims in future work is really an option. I think it would be better to run the ablation study on a dataset where improvements can be measured more clearly.
Final rating
I am inclined to keep my original rating ("6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations."). I still like the good results on the Tanks & Temples dataset and believe that the proposed approach is technically sound. However, I do not find the authors' rebuttals particularly convincing and thus do not want to increase my rating. In particular, I still have concerns about the ablation study as I am not sure whether the ground truth of the DTU dataset is accurate enough that it makes sense to claim improvements if the difference is 0.05 mm or smaller. Since this only impacts the ablation study, it is also not a reason to decrease my rating.
4. Does the claim "It can be seen from the table that our proposed modules improve in both accuracy and completeness" really hold? Why not use another dataset for the ablation study, e.g., the training set of Tanks & Temples or ETH3D?