Compute for AI: choosing the right hardware (and connecting it properly)
Third post in the series where I translate AI into the language of people who live and breathe infrastructure. In the previous post, we talked about the storage bottleneck nobody notices until it hurts. This one is about compute. Spoiler: it is not enough to buy the most expensive GPU. You need the right GPU, connected the right way. tl;dr Pick hardware based on training vs inference, not on sticker price. GPU family, memory size, and interconnect matter more than vCPU count. For distributed jobs, quota, availability, and network fabric decide whether the cluster performs or stalls. The story you don’t want to live The ML team asks for “a GPU cluster for training.” You do what any infra engineer would do under time pressure: provision eight Standard_D64s_v5 VMs. Sixty-four vCPUs each, 256 GiB of RAM, Premium SSD. On paper, it looks respectable. ...