Fourteenth post in the series. In the previous one, we used AI for our own infrastructure work. This time the scope is bigger: how to take an entire organization from “let’s use AI” to a governed platform that can survive contact with finance, security, and production support.
Best intentions, worst outcomes
Your CTO walks into the all-hands and says: “We’re going all-in on AI.” The room buzzes. Teams start brainstorming use cases before the meeting ends. Within two weeks, Slack is full of threads about GPU availability.
Three months later, five teams have provisioned GPU VMs independently across four subscriptions. Nobody can tell which models are in production versus a weekend experiment. Two teams are paying for reserved capacity on clusters that sit idle 80% of the time. Security has not reviewed a single deployment. The CFO wants to know why the Azure bill went up 40%.
The enthusiasm was there. The framework wasn’t.
The 6-phase model
This borrows the discipline of Microsoft’s Cloud Adoption Framework, but it is rebuilt for infra teams. Each phase has deliverables and an exit test. If you cannot tell whether you are ready to move on, you are not ready.
Assessment → Enablement → Infra Preparation → Experimentation → Scale & Governance → Continuous Adoption
Think of it as the infrastructure lifecycle applied to AI: assess, build, validate, scale, operate, iterate.
Phase 1: assessment (where are we today?)
Before building anything, do the honest assessment. If a team needed to deploy a model to production tomorrow, would your infrastructure support it securely?
Readiness scorecard
| Area | Key questions | Rating (1-5) |
|---|---|---|
| Team skills | Can the team provision and manage GPU compute? | ___ |
| GPU readiness | Quotas approved? Regions selected? | ___ |
| Networking | Private endpoints, bandwidth, DNS? | ___ |
| Security | Managed identity, Key Vault, network isolation? | ___ |
| Automation | IaC coverage, CI/CD maturity, GitOps? | ___ |
| Shadow AI | Unauthorized deployments identified? | ___ |
A score below 3 in any area = focused work in Phase 2 before proceeding. Don’t hide low scores; they’re the most valuable output of this phase.
Shadow AI detection
The audit everyone skips and everyone needs. Look for: teams running models in personal subscriptions, API keys in code repos, GPU VMs provisioned outside IaC pipelines, SaaS AI tools processing company data without security review.
Shadow AI isn’t just a governance problem. It’s a security exposure. Every unreviewed model endpoint is a potential data leak. Treat it with the same urgency as unpatched servers.
Phase 2: enablement (building the foundation)
Close the gaps from Assessment. This is where you spend money on people, process, and tooling before you spend much more cleaning up the alternative.
Team upskilling
Infrastructure engineers don’t need to understand backpropagation. They need: GPU memory management, inference scaling patterns, token-based pricing. Three tiers:
- Foundational: AI concepts in infra language (the visual glossary coming in the next post)
- Operational: Deploying and monitoring AI workloads (posts 3-8 in this series)
- Advanced: Performance tuning, cost optimization (posts 9-12)
Security baseline (non-negotiable)
- All services authenticate via managed identity (no exceptions)
- All secrets live in Key Vault with automated rotation
- All model endpoints behind private endpoints
- All data access follows least-privilege RBAC
Document these as policies, not suggestions.
Phase 3: infrastructure preparation (building the platform)
This is where IaC stops being background plumbing and becomes the platform. Turn the baseline into a repeatable self-service system. Codify everything. If it cannot be deployed from a git commit, it should not exist.
Templates for common patterns
- GPU VM clusters for training (Bicep/Terraform)
- AKS clusters with GPU node pools for inference
- Azure ML workspaces with networking
- Azure OpenAI deployments with diagnostic settings
Each template includes security controls baked in: private endpoints, managed identity, diagnostic settings, resource tagging.
Monitoring stack (deploy before workloads)
- GPU utilization and memory (DCGM exporter)
- Inference endpoint latency (P50/P95/P99)
- Token consumption tracking
- Cost attribution by team and project
- Model health indicators
Cost governance (implement before it gets expensive)
Budgets per team, alerts at 50%/75%/90%, mandatory tagging, GPU quota governance. If it doesn’t exist before the workloads, it won’t exist after.
Phase 4: experimentation (controlled exploration)
Once the platform is ready, teams can experiment with guardrails.
Sandbox environments
- Dedicated resource groups with cost caps via Azure Policy
- GPU quotas sized for experimentation
- Automatic cleanup: sandboxes inactive 14 days = flagged, 30 days = decommissioned
- Unique tag per experiment from day 1
Mandatory success criteria
Before starting an experiment, the team defines:
- What success looks like (accuracy threshold, latency target, cost ceiling)
- What infra signals indicate viability at scale
- Next step if it works
Experiments without success criteria aren’t experiments; they’re hobbies.
Phase 5: scale & governance (going to production)
This is the jump from “works in the sandbox” to “runs with SLAs and pages people when it does not.”
Multi-tenancy and isolation
- Namespace or resource group isolation per team
- GPU quota enforcement per tenant
- Network segmentation between workloads
- Per-team monitoring dashboards
SLA/SLO design for AI
Define SLOs for: availability, latency (P99), throughput, error budget. AI endpoints have unique failure modes (model loading delays, GPU memory exhaustion, token rate limiting) that your SLO design needs to account for.
Infra ↔ AI translation: “Inference endpoint SLA” is exactly like a web API SLA. The difference: cold start can be 30 seconds (loading GBs of model weights into GPU memory), and resource exhaustion is usually GPU memory, not CPU. Same discipline, different resources.
Fleet management and runbooks
Document procedures for: scaling during traffic spikes, zero-downtime model version rotation, GPU hardware failure response, token rate limiting (429s), cost overrun management.
Phase 6: continuous adoption (never ends)
AI infra is not a project with an end date. It is an operating capability. Treat it like one.
Quarterly cadence
- Utilization trend review
- Cost optimization actions
- Security updates
- Technology radar changes
- Self-service adoption metrics
- Next quarter roadmap
Technology radar
Categorize tools and services as:
- Adopt: Proven, standardize
- Trial: Promising, time-boxed evaluation
- Assess: Interesting, monitor
- Hold: Not ready
The 5 anti-patterns that kill adoption
| Anti-pattern | What happens | How to avoid |
|---|---|---|
| Big Bang | 6 months building a perfect platform, nobody uses it | Start with MVP, iterate |
| Shadow AI | Teams deploy without infra involvement | Make the governed path the easiest one |
| GPU Hoarding | Teams reserve quota “just in case” | Use-it-or-lose-it: <20% utilization for 30 days = reclaimed |
| Security Afterthought | “We’ll add security later” | Templates with managed identity and private endpoints by default |
| Build Everything | Custom framework when a managed service exists | Default to managed services |
These anti-patterns stack on top of each other. Big Bang causes Shadow AI because teams will not wait. Shadow AI creates Security Afterthought because deployments skip review. By the time finance notices, you are already arguing about cleanup instead of building anything useful.
In the next post
That is the framework. The final post is the visual glossary: an infra ↔ AI translation sheet for the terms people keep throwing around as if everybody was born knowing them.