1. Determining hyperparameters and reporting complexity 1.1. The paper requires setting âaccuracy goalsâ when encountering a new task. However, it might be unclear which accuracy can be reached and the paper is opaque how these accuracy goals are determined e.g. when comparing to prior work. To reach optimal performance algorithm 1 might need significant manual intervention. 1.1.1. How are the âaccuracy goalsâ determined (especially for Table 6,7)? 1.1.2. What happens if growing the network does not lead to achieving the accuracy goal? E.g. increasing the network capacity might lead to stronger overfitting and a reduced accuracy? 1.2. The approach may need many iterations to retrain the model to meet the âaccuracy goalâ (both w.r.t. growing and compressing) 1.3. How much is the model grown, how much is picked, how much is compressed? It would be interesting to see this for the different models in Table 6, as well as the accuracy targets. 1.4. It would be good to report the memory overhead from the binary masks and relate this to memory-based approached such as GEM, A-GEM, and generative replay. 2. Experimental Evaluation 2.1. Ablations 2.1.1. The paper claims that âAnother distinction of our approach is the âpickingâ step â. However, this aspect is not ablated. 2.2. Experiments on CIFAR. The comparison on CIFAR is not convincing 2.2.1. The continual learning literature has extensive experiments on this dataset and the paper only compares to one approach (DEN). 2.2.2. It is unclear if DEN is correctly used/evaluated. It would have been more convincing if the authors used the same setup as in the DEN paper to make sure the comparison is fair/correct. 3. Motivation 3.1. The paper claims forgetting is fully avoided due to the usage of a mask. While it is true that *after* model compression no further forgetting happens, but there is an accuracy drop during pruning, in contrast to e.g. regularization-based methods. Specifically, the original value (before pruning) is not recoverable and hence should be reported as forgetting. 4. The checklist is not fully accurate. The paper does not provide error bars and std-deviation for experiments. 5. Minor: 5.1. Grammar issue in word âdeterminingâ in the 4th paragraph on page 3. 5.2. On page 3, in âMethod overviewâ it says âAn overview of our method is depicted belowâ whereas it should directly refer to Figure 1 because Figure 1 is on page 2 5.3. On page 6, right below Figure 2, it says âin all experiments, but realize DENâ. Word ârealizeâ does not fit into the context. 5.4. In future, please use the submission template (not the camera-ready version) so that line numbers on the margins can be used to easily refer to the text. I lean more towards accept: The overall convincing results (especially Table 6) and overall novel model outweigh the limitations discussed above.
2. Experimental Evaluation 2.1. Ablations 2.1.1. The paper claims that âAnother distinction of our approach is the âpickingâ step â. However, this aspect is not ablated. 2.2. Experiments on CIFAR. The comparison on CIFAR is not convincing 2.2.1. The continual learning literature has extensive experiments on this dataset and the paper only compares to one approach (DEN). 2.2.2. It is unclear if DEN is correctly used/evaluated. It would have been more convincing if the authors used the same setup as in the DEN paper to make sure the comparison is fair/correct.