Cost engineering for AI: when idle GPUs cost more than your car
Ninth post in the series. In the previous one, we hardened the platform against prompt injection and data leakage. Now: how not to go bankrupt in the process. The $127,000 Monday Monday morning. Coffee in hand, email from Finance in the subject line: “URGENT: Azure invoice $127,000, please explain.” Forecast was $42,000. Two ND96isr_H100_v5 VMs, provisioned three weeks ago for a “quick experiment,” never shut down. At ~$98/hour each, running 24/7 for three weeks: $33,000 in idle GPU compute. Nobody using them. Nobody remembered they existed. ...