compared to [1], which are advantages of the IBP. 1) Unlike a factorized model with an IBP prior, the proposed method lacks a sparsity constraint in the number of factors used by subsequent tasks. As such, the model will not be incentivized to use less factors, leading to increasing number of factors and increased computation with more tasks. 2) The IBP prior allows the data to dictate the number of factors to add for each task. The proposed method has no such mechanism, requiring setting the growth rate by hand using heuristics or a pre-determined schedule. Either is liable to over- or under-utilization of model capacity. Table 4 in the Experiments show that this does indeed have a significant impact on performance.
Overall, I think this is an example of convergent ideas rather than plagiarism, but a discussion of the connections is warranted.
Task incremental learning: This method requires knowing the task ID at test time to pick which factor selector weights to use. Without it, the proposed method doesn’t know which subnetwork to use, and would likely have to resort to trying all of them, which isn’t guaranteed to produce the right results. Recent continual learning methods are often evaluated in the more challenging class incremental setting, where task ID is not known.
Experiments 1. (+) Experiments are conducted on a good set of datasets 2. (+) Error bars are shown 3. (+) The proposed method mostly outperforms the baselines, especially on the more complex datasets. 4. (-) More baselines should be compared against, particularly dynamic architecture approaches, as that’s the category that this method falls under. Many of the compared methods don’t operate on the same set of continual learning assumptions as this paper; in particular, the replay-based methods are often using replay because they consider class incremental learning. 5. (-) Why are the results of Multitask learning so bad for S-CIFAR-100 and S-miniImageNet? My understanding is that it trains on all the data jointly, which should actually be the upper bound for a single model. 6. It would have been nice to visualize the factor selection matrices S for each task in order to visualize knowledge transfer.
Miscellaneous: 1. \citep should be used for parenthetical citations. 2. Initial double quote “ is backwards (Related Works). 3. “the first task,rk,1” 4. Figure 3 caption: “abd”
Questions: 1. How would you apply the weight factorization to 4D convolutional kernels?
[1] “Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors”, AISTATS 2021
1) Unlike a factorized model with an IBP prior, the proposed method lacks a sparsity constraint in the number of factors used by subsequent tasks. As such, the model will not be incentivized to use less factors, leading to increasing number of factors and increased computation with more tasks.