Cost engineering for AI: when idle GPUs cost more than your car
Ninth post in the series. In the previous one, we hardened the platform against prompt injection and data leakage. Now for the part Finance notices first: how not to go bankrupt in the process. tl;dr AI cost control starts with lifecycle policy, model choice, and quota discipline. Shut down idle GPUs, use cheaper models where quality allows, and treat every exact cost number as time-sensitive. The $127,000 Monday Monday morning. Coffee in hand, email from Finance with the subject line: “URGENT: Azure invoice $127,000, please explain.” Forecast was $42,000. Two ND96isr_H100_v5 VMs, provisioned three weeks ago for a “quick experiment,” never shut down. At about $98/hour each, running 24/7 for three weeks: roughly $99,000 in idle GPU compute. Nobody was using them. Nobody remembered they existed. ...