What is a cluster?
A cluster is a group of separate machines wired together with fast networking so they act like one bigger computer.
Real cluster vs. a pile of desktop cards
A datacenter cluster — like the rack below — uses GPUs connected by NVLink and high-speed networking specifically built so the GPUs can share memory and work on the same problem in lock-step, with almost no delay talking to each other. A pile of desktop graphics cards plugged into a few gaming PCs looks similar on paper (lots of GPUs, lots of memory added up) but those cards can't pool their memory or coordinate fast enough — you can run several separate small jobs on them, but you can't run one giant model that needs all the memory to act as a single pool. That's the real difference: a cluster is engineered to behave like one machine; a pile of desktop cards is just several small machines standing near each other.
Example: a rack-class cluster
1× NVIDIA DGX H100 SuperPOD (4-node cluster)
- Total GPU memory: 2560 GB
- Total power: 44000 W (~37 homes, 1056 kWh/day, 11.7 EV batteries/day)
- Total price: $1,400,000
How a request flows through a cluster
Every server in the cluster holds part of the model or handles part of the traffic; the cluster software splits the work out and stitches the results back together before anything reaches the user.