- The proposed method is computationally very costly. The outer optimization needs to differentiate through several (n = 10+) chained calls of a large score model in each iteration, with an overall N (N=100+) outer loop iterations, per image. Thus, it requires at least 1000 NFEs plus the heavy cost of N backpropagations through a large, chained model.
- There are lots of hyperparameters that need to be tuned (step size, N, $\delta t$) and as the optimization needs to be solved on a sample-by-sample basis it is not clear how much variation in optimal hyperparameters can occur.
- I have serious doubts about the experiments:
1) There is no comparison with DPS which is a well-known diffusion-based solver that has better performance than the included competing techniques and source code is made public. For instance, for 4x SR *with noise* DPS reports far better LPIPS than any techniques in this paper.
2) Distortion metrics such as PSNR, NMSE and SSIM are completely missing, which are crucial in evaluating inverse problem solvers.
3) A very small (100) number of samples is used for evaluation. In other competing methods it is standard to use a 1000-sample validation split. Thus the results are not necessarily reliable and it is very difficult to compare to existing published results (DDRM, DDNM, DPS). Furthermore, since FID heavily depends on the number of samples used in the generated distribution, the reposrted FIDs are not compatible with the ones reported in competing method's original papers.
4) I have doubts about the reported timing results in Table 3. SHRED is reported as approx. 5x slower than DDRM. According to the DDRM paper, they use 20 NFEs. How is the reported timing possible, when SHRED uses 100 outer loop iterations with 10 NFEs in each outer loop (total 1000 NFEs) plus the additional cost of 100 backpropagation?
5) The robustness experiments could be more rigorous. Instead of showing some good looking samples, it would be more meaningful to quantify the variation of image quality metrics for the validation dataset over 5-10 samples.
6) The framework is developed for noisy inverse problems, however there are no experiments for the noisy case. Reconstruction performance under measurement noise is crucial in evaluating the utility of the algorithm.
- There are lots of hyperparameters that need to be tuned (step size, N, $\delta t$) and as the optimization needs to be solved on a sample-by-sample basis it is not clear how much variation in optimal hyperparameters can occur.