Having trained a few large NLP models myself, the bottleneck is pretty much always communication among GPUs and loading data fast enough to avoid starving GPUs. $NVDA has now bought Mellanox for communication and ARM for CPUs, integrating all. Are RAM and SSD next? 