Weakness:
The paper doesn't present comprehensive ablation study of the effectiveness of the cost model.
The paper seems to overlook some latest advances in distributed training and model parallelism, for example, Alpa [1] that considers homogeneous setting but still formulates the problem similarly, while having heterogeneity seems to be a plain extension. Would love to hear from the reviewer about this particular comparison.
[1] Zheng L, Li Z, Zhang H, Zhuang Y, Chen Z, Huang Y, Wang Y, Xu Y, Zhuo D, Gonzalez JE, Stoica I. Alpa: Automating Inter-and Intra-Operator Parallelism for Distributed Deep Learning. arXiv preprint arXiv:2201.12023. 2022 Jan 28.
As the author stated, memory footprint is right now not under consideration. However, the reviewer does not consider this as a significant drawback.
2022 Jan 28. As the author stated, memory footprint is right now not under consideration. However, the reviewer does not consider this as a significant drawback.