Why does it matter?
Distributed training would make total company compute more important than any single visible supercomputer.
Direct quote
Power constraints will likely become the primary bottleneck for AI supercomputer growth, driving a shift toward distributed training across multiple sites. This evolution could change how we measure AI training capabilities—from focusing on individual AI supercomputers to assessing companies' aggregate compute capacity.
Trends in AI Supercomputers