Weakness:
I have some concerns on identification mechanism based on identity bank. 1) Scalability. As shown in Table 3 (a), the performance is getting worse with growth of the maximum number of identities. It means that the capacity should be preset to some small number (e.g., 10). In real-world scenario, we can have more than 10 objects and most of the time we don't know how many objects we will need to handle in the future. Have the authors thought about how to scale up without compromising performance? 2) Randomness. Identities are randomly assigned one embedding from the identity bank. How the results are robust against this randomness? It would be undesirable for the result to change with each inference. It would be great to have some analysis on this aspect.
Overall Evaluation:
The paper present a novel approach for multi-object video object segmentation and the proposed method outperfrom previous state-of-the-arts on several benchmarks.
Now, I would recommend to accept this paper. I will finalize the score after seeing how authors address my concerns in Weakness.
While future works are discussed in Supplementary Materials, I encourage the authors to include more discussions on limitations and societal impacts.
1) Scalability. As shown in Table 3 (a), the performance is getting worse with growth of the maximum number of identities. It means that the capacity should be preset to some small number (e.g., 10). In real-world scenario, we can have more than 10 objects and most of the time we don't know how many objects we will need to handle in the future. Have the authors thought about how to scale up without compromising performance?