AI

Twelfth post in the series. In the previous one, we ran Azure OpenAI with HA and sane retry patterns. This one is for when the nice diagram meets real life. This post is organized as real-world failure scenarios. Each follows: Symptoms → Diagnosis → Root Cause → Resolution → Prevention. Read it once for pattern recognition. Then bookmark it. You will need it again. tl;dr Most late-night AI infra incidents come down to driver drift, memory pressure, scheduler mismatch, throttling, or cold starts. Start with the first check that rules out the biggest class of failure. Scenario 1: NVIDIA driver crash after kernel update Symptoms Monday morning. The ML team reports that all GPU workloads failed over the weekend. Nobody deployed anything. You SSH in: ...

Eleventh post in the series. In the previous one, we built the self-service AI platform with multi-tenancy and scheduling. This time it’s the service everybody wants to consume: Azure OpenAI, and how to run it without getting slapped by 429s. tl;dr Azure OpenAI capacity is a token problem before it is a scaling problem. Design around TPM and RPM, back off on 429s, and route across deployments instead of betting everything on one endpoint. The 429 that changed everything Your team launched an internal GPT-4o chatbot on Monday. Day 1 was demos for leadership and Slack praise. Day 3 brought “the bot is slow.” Day 5 brought HTTP 429 on 30% of requests. You open Azure Monitor and find the 80K TPM ceiling waiting for you. ...

Tenth post in the series. In the previous one, we controlled costs with Spot VMs, right-sizing, and FinOps. Now for the next problem: how to stop being a human help desk for GPU access. tl;dr Self-service AI platforms need isolation, quotas, and scheduling together. The goal is fewer tickets, not faster manual provisioning. The Slack channel that ate your calendar Six months ago, you provisioned a single GPU VM for the ML team. Configured drivers, mounted storage, closed the ticket. Felt like any other infrastructure request. ...

Ninth post in the series. In the previous one, we hardened the platform against prompt injection and data leakage. Now for the part Finance notices first: how not to go bankrupt in the process. tl;dr AI cost control starts with lifecycle policy, model choice, and quota discipline. Shut down idle GPUs, use cheaper models where quality allows, and treat every exact cost number as time-sensitive. The $127,000 Monday Monday morning. Coffee in hand, email from Finance with the subject line: “URGENT: Azure invoice $127,000, please explain.” Forecast was $42,000. Two ND96isr_H100_v5 VMs, provisioned three weeks ago for a “quick experiment,” never shut down. At about $98/hour each, running 24/7 for three weeks: roughly $99,000 in idle GPU compute. Nobody was using them. Nobody remembered they existed. ...

Eighth post in the series. In the previous one, we learned that a green dashboard doesn’t guarantee a healthy model. Now: the threats your WAF won’t catch. tl;dr Most AI security failures happen at identity, data access, and prompt boundaries, not at the firewall. Use managed identity, RBAC, Key Vault, private connectivity, and content filtering together. The chatbot that knew too much Your organization deploys an internal chatbot with Azure OpenAI, connected to a knowledge base of policies, documentation, and FAQs. The rollout goes smoothly, adoption takes off, and leadership is already planning a customer-facing version. ...

Seventh post in the series. In the previous one, we put models into production with CI/CD pipelines. Now: how do you know they’re actually healthy? tl;dr Infra health is not model health. Track GPU, token, application, and answer-quality signals together or you will miss regressions while every dashboard stays green. The silent failure Your Azure OpenAI endpoint returns 200 OK on every request. Latency is normal, P95 under 800ms. CPU and memory within thresholds. Kubernetes shows healthy pods, no restarts. By every infra metric you trust, the system is perfect. ...

Sixth post in the series. In the previous one, we automated GPU cluster provisioning. Next comes what happens after the hardware is ready: how a model goes from “works on my notebook” to “running in production with an SLA.” tl;dr Models need the same artifact, promotion, and rollback discipline as application builds. Use a real registry with metadata and controlled deployments. Prefer MLflow aliases over deprecated stages when describing promotions. The model with no birth certificate A data scientist drops a message in the team channel with a link to a shared drive: “Here’s the model. It’s a 15 GB PyTorch checkpoint. We need it in production by Friday.” ...

Fifth post in the series. In the previous one, we went inside the GPU. This time we automate everything around it. Understanding GPUs is useful. Provisioning them consistently and at scale is where infrastructure engineering actually meets AI. tl;dr IaC is the only sane way to provision expensive GPU infrastructure repeatedly. Validate SKU choices, remote state, and deployment guardrails before apply. If the pipeline uses OIDC, Terraform and GitHub Actions both need explicit OIDC settings. The $4,000 typo I started the week with a win. I manually provisioned a GPU cluster in East US 2 for an ML experiment: AKS with a Standard_NC6s_v3 node pool, accelerated networking, GPU drivers, correct taints. It took most of a day, but it worked. ...

Fourth post in the series. In the previous one, you learned which GPU VMs to provision and how to connect them. This time we look inside the GPU so you can troubleshoot better and talk to the ML team without guessing. tl;dr GPU memory is consumed by more than model weights. Gradients, optimizer states, and activations usually dominate training memory. Understanding memory hierarchy and topology makes troubleshooting much faster. The 2 AM ticket Slack fires at 2 AM. The ML team’s training job crashed again. The error is a single line: ...

Third post in the series where I translate AI into the language of people who live and breathe infrastructure. In the previous post, we talked about the storage bottleneck nobody notices until it hurts. This one is about compute. Spoiler: it is not enough to buy the most expensive GPU. You need the right GPU, connected the right way. tl;dr Pick hardware based on training vs inference, not on sticker price. GPU family, memory size, and interconnect matter more than vCPU count. For distributed jobs, quota, availability, and network fabric decide whether the cluster performs or stalls. The story you don’t want to live The ML team asks for “a GPU cluster for training.” You do what any infra engineer would do under time pressure: provision eight Standard_D64s_v5 VMs. Sixty-four vCPUs each, 256 GiB of RAM, Premium SSD. On paper, it looks respectable. ...