(W1) Eproxy alone does not actually work that well, it is only Eproxy+DPS that delivers convincing results - however, the necessity of having extra trained architectures (even if only 20) puts this work in a significantly different complexity regime than most of the baselines to which the paper compares - in short, the choice of baselines seems inadequate to the cost of performing EProxy+DPS.
(W2) Related to the above, searching cost on ImageNet for Eproxy+DPS seems to ignore the necessity of training 20 architectures for DPS - including this cost would increase the searching time for Eproxy+DPS SIGNIFICANTLY, make it fall short to many of the baselines presented in Table 8 (e.g., TE-NAS, P-DARTS, likely SNAS and PC-DARTS).
(W3) Comparison to synflow and NASWOT in table 5 seems biased -- why only assume 0 queries for these method where both were originally proposed to be used with either random sampling or evolutionary search? Specifically, results reported for synflow in the original paper are between 51 and 34 queries to reach test accuracy of 94.22, which is significantly better than what is reported for Eproxy+DPS (150 queries). Also, it is unclear what the authors mean by "we utilize the Eproxy as the fitness function for Regularized Evolution" and then showing results for varying number of queries, especially 0 queries (this might be also related to the point below).
(W4) Showing that "Eproxy+DPS can find optimal global architectures within the RE search history" in Table 4 is incorrect in its context - it is possible that RE significantly helps Eproxy+DPS by prefiltering all architectures based on their accuracy (thus being very costly), showing results of Eproxy+DPS conditioned by running RE is not the same as showing results of just Eproxy+DPS (what the Table's caption suggests, by saying: "Eproxy+DPS uses substantially lower queries to find the global optimal architectures.")
Minor shortcomings and suggestions:
Figure 1 is not exactly informative as it does not convey any information about quality of each method
Figure 2 also does not seem to add much, I actually had pretty hard time understanding what it tries to say - what is "W" at x axis? why does "arch 1" finish earlier than "arch 2"? why do they follow the same curve? Just to be clear, I am quite sure I know what the authors tried to say here, the concept of cheap proxies for NAS is not hard to grasp, but the Figure in its current form is not helpful, in my opinion
while the choice of NAS benchmarks seems adequate to me, it would strengthen the paper if the authors could include experiments on some additional, large search spaces other than benchmarks/DARTS, for example a MobileNet-like search space
zero-cost proxies considered in the paper are not very new, some of the more recent ones that the authors could include to make comparison more convincing are: ZenNAS, KNAS
I would suggest changing the title - in its current form it is unclear really what the authors tried to say, some reasoning: 1) it is unclear if the authors mean a particular efficient proxy or efficient proxies in general, "is" suggests that it is a particular proxy, but then there is not proxy called "Efficient Proxy", which suggests that it is rather referring to a family of efficient proxies; 2) however, saying that a generic class of efficient proxies is now extensible suggests that the authors propose a generic method to make any efficient proxy extensible, which is not the case. I would recommend changing the title to something like "Extensible and Efficient Proxy for NAS", or something similar, it would make it much clear that the authors propose a new proxy that is supposed to be both efficient and extensible.
Actually, another minor comment is that I am not sure if "extensible" is the best term here -- in my opinion a better term would be: "generalizable", "transferable" or "adaptive", all of them suggest to me some kind of changes in the training data (i.e., downstream task), which seems to be the main point of Eproxy. At the same time, "extensible" suggests more that something new could be easily added to the proposed method in order to make it better. Details:
W1: Specifically, on NDS Eproxy is only comparable to NASWOT, despite performing few-shot training of architectures (i.e., even if the absolute cost is not very high, it's still 10x more costly than NASWOT, without any obvious benefits), furthermore on TransNAS-Bench-Micro Eproxy fails quite significantly on 6 out of 7 tasks. On NAS-Bench-MR Eproxy achieves convincing improvement (i.e., better in at least one metric - spearman-r or top% models - while not compromising another) over the selected baselines in only 2 out of 9 cases, consequently it falls short to synflow on average. On the other hand, as mentioned above, Eproxy+DPS does improve upon both synflow and naswot, most of the time, but it also requires training a number of architecture, which makes it no longer low-cost method - at best, it should be treated as efficient, predictor-based approach. However, the comparison is done almost exclusively with methods that do not assume any training (the only notable exceptions are Table 6, 7, and 8, but they have their own problems, as mentioned in other parts of the review, and the choice of baselines in Table 6 and 7 is not also not very convincing).
W2: reported searching cost is approximately 1.5 GPU hours (0.06 GPU days) but training a single model from the DARTS CNN space on CIFAR-10, using the official code, takes roughly a day on a V100. Even though the authors use models from a different search space (NDS-DARTS benchmark), that in practice come for free, and might also be cheaper to train than standard DARTS, I am not convinced that the presented results are fair in comparing the proposed methods vs. baselines. Specifically, considering that the main focus of the submitted work is to enable low-cost NAS for diverse tasks, it is not unreasonable in my opinion to expect the proposed method to work well in situations when we do not have a priori knowledge of performance of certain architectures, or even if there exists a highly-correlated proxy task/search space (like DARTS vs. NDS-DARTS). Therefore not including the cost of obtaining accuracy for the 20 architectures needed to conduct DPS is likely to be misleading, were the method to be applied in real-world situations, which it tries to address.
W4: to be clear, I am inclined to believe that Eproxy+DPS is likely to produce better results than just using RE, especially if the authors used it to augment RE, like, e.g., Abdelfattah et al. However, formally speaking, this has not been shown in the current paper. Specifically, if we know that RE finds optimal architecture in N steps, then "finding optimal architecture within RE history" is actually a problem of finding the correct model within the N models, rathar than within the whole search space -- the a priori knowledge that the optimal models exists in a relatively small subset of the search space provides a very strong prior which can naturally boost a method's performance. In the context of Table 7, RE is shown to find the optimal architecture in ~560 queries, so Eproxy+DPS needing 58 actually explores 10% of all models - if we were to apply the same level of performance to the entire search space (15k models) then it would be 1500 queries. While I would not claim that this is the expected number for Eproxy+DPS, I hope that this clearly shows why the current results are misleading. Even a random search would do significantly better in this setting.
1) it is unclear if the authors mean a particular efficient proxy or efficient proxies in general, "is" suggests that it is a particular proxy, but then there is not proxy called "Efficient Proxy", which suggests that it is rather referring to a family of efficient proxies;