To rank the world's biggest supercomputers, we're gonna need a bigger bench
For decades, the Top500 list has tirelessly ranked the world’s biggest, most powerful, and often most unique publicly-known supercomputers.
- They'll just need a new benchmark better suited to judging the capability of these arrays while at the same time preventing operators from gaming the system.
- They’re not designed to run one piece of code but have instead been architected to run many, many smaller or medium-sized code instantiations, which may include training, fine-tuning, reinforcement learning, and of course inference.
- Allowing for lower precisions — HPL-MxP already does this to some extent — is one avenue for addressing the fact that these accelerators are increasingly trading FP64 performance for lower-accuracy floating-point data types like FP8 or FP4.
China’s Lineshine supercomputer at 2.2 exaFLOPS of double precision compute in the High-Performance Linpack (HPL) benchmark may be the top system on the bi-annual ranking, but it’s nowhere near the biggest compute cluster in the world.
- “Our prime rule for benchmarking is, you have to benchmark your whole system with one code and you cannot just add two parts up to make up a number,” he explained.
- “What’s really needed for tracking that more appropriately is not just a different benchmark but also different methodology,” Top500 author Erich Strohmaier said.
- Superclusters like SpaceXAI’s Colossus, Meta’s Promethius, and OpenAI’s Stargate pack hundreds of thousands of GPUs, dwarfing anything on the Top500 today.
- Because of this, Strohmaier says, it wouldn’t be reasonable to expect SpaceXAI to run something High-Performance Linpack (HPL) on a cluster like Colossus.
- “We were and we still are very focused on capability.” By capability, Strohmaier is referring to the ability to run one code across the entire cluster.
How this summary was made: extractive summary — built from the sources' own reporting. It is drawn only from the reporting of the sources listed below — read the originals for full detail. How we summarize →