Chapter 3: From Script to Agent

In the previous post, the Azure OpenAI quota watchdog was a script with if pct_of_tpm > 0.8: alert. That works, but it has the same flaw every blunt monitoring rule has: context does not exist. A batch job that predictably eats 90% of TPM for 10 minutes at month-end looks identical to an agent gone feral and burning tokens all afternoon. Both cross the threshold. Only one should wake somebody up.

This post is about closing that gap: giving the watchdog a reasoning layer, without giving up any of the guardrails we’ve already set.

What changes (and what doesn’t)

What does not change is the part that matters most: the server still only reads telemetry and only writes to notification channels. No new tool gets to act on the resource. Giving autonomy over how to alert is one thing. Giving autonomy to change production is a different class of problem, and two blog posts do not buy you that much operational maturity.

What changes is what happens between detecting the threshold and deciding what to do. Two new tools join the server:

@mcp.tool()
def get_token_usage_history(deployment_name: str, days_back: int = 30) -> dict:
    """Returns the deployment's consumption pattern over the last N days,
    aggregated by day of week and hour of day, for comparison against
    the current spike."""
    response = _get_metrics_client().query_resource(
        resource_uri=OPENAI_RESOURCE_ID,
        metric_names=["TokenTransaction"],
        timespan=timedelta(days=days_back),
        granularity=timedelta(hours=1),
        filter=f"ModelDeploymentName eq '{deployment_name}'",
    )
    # aggregates by weekday-hour and returns avg/max per bucket
    ...

import httpx
from typing import Literal

@mcp.tool()
def send_priority_alert(message: str, priority: Literal["info", "warning", "urgent"]) -> str:
    """Posts an alert with a priority level. The caller decides the priority;
    this function only formats and sends the message."""
    prefix = {"info": "ℹ️", "warning": "⚠️", "urgent": "🚨 @oncall-ai"}[priority]
    httpx.post(SLACK_WEBHOOK_URL, json={"text": f"{prefix} {message}"}, timeout=10).raise_for_status()
    return "sent"

(mcp here is the same FastMCP instance created in post 2; these two tools join the same watchdog429 server, not a separate one.)

The first one gives the agent a baseline for comparison: “has this happened before at this same time?” The second separates the act of notifying from the urgency level. The reasoning layer decides the level, not the tool.

Where the model comes in (and where it doesn’t)

One thing to get right here: the model does not run every minute. The deterministic script from post 2 stays the poller: it runs on the cron, once a minute, at zero LLM cost, and only triggers a model call once the threshold is crossed. Putting an LLM in the loop on every iteration of a monitoring cycle is wasting money on a path that, the overwhelming majority of the time, needs no reasoning at all. It’s only worth paying the cost of a model call to decide the response level in the minutes the threshold actually gets crossed.

 cron (1x/min, zero LLM cost)
       │
       ▼
 get_token_usage_trend > 0.8 ?  ──no──▶  loop continues
       │ yes
       ▼
 invoke the agent (1 model call)
       │
       ▼
 get_token_usage_history + reasoning
       │
       ▼
 send_priority_alert(priority = info | warning | urgent)

The agent’s system prompt stays small and direct: it doesn’t need much more than this:

You are an Azure OpenAI quota watchdog. You were triggered because TPM
consumption passed 80%. Your only decision is the alert's priority
level: info, warning, or urgent.

Use get_token_usage_history to compare the current spike against the
pattern from the last 30 days at the same day of week and time.

- If the current pattern is consistent with past spikes at this same time: info.
- If it's an unprecedented spike but the curve is leveling off: warning.
- If it's an unprecedented spike AND the curve keeps climbing fast: urgent.

You have no tool that can act on the deployment. Your only possible
output is calling send_priority_alert exactly once.

Notice the last line: it exists to reinforce, in the prompt itself, the limit that’s already baked into the architecture. Intentional redundancy. The real guardrail is the absence of the tool; the prompt is just the second layer.

Where this actually runs: the schedule from post 2 and the conditional model invocation fit comfortably in an Azure Container Apps Job with a */1 * * * * cron schedule. It starts a Python container, runs get_token_usage_trend, decides whether to invoke the model, and exits. No always-on server. No VM to babysit. The job’s managed identity (azurerm_user_assigned_identity plus an azurerm_role_assignment for Monitoring Reader on the Cognitive Services account) replaces any fixed API key; the full Terraform is in the series companion repo.

Two scenarios side by side

Scenario A: last business day of the month, 11 p.m., TPM at 87%. The agent calls get_token_usage_history, sees this same deployment has hit 80-90% every month-end at this hour for six months straight, always returning to normal in under 15 minutes. Decision: info. A message in the channel, no mention, no page.

Scenario B: a random Tuesday, 2 p.m., TPM at 84% and climbing fast over the last three minutes. The agent checks history, finds no similar pattern for this day of week or time, and the curve shows no sign of leveling off. Decision: urgent. Same tool, different priority, and this time someone actually gets paged.

The difference between the two scenarios wasn’t in any fixed threshold; it was in comparing the present against history, which is exactly the kind of judgment a simple if doesn’t handle well and an agent handles reasonably well, as long as the right tools are available.

The extra guardrails this step requires

Giving decision autonomy, even just over an alert’s level, opens a new risk category the pure script didn’t have: alert fatigue in reverse. A poorly calibrated agent can either over-alert (everything becomes “urgent” and the team learns to ignore it) or under-alert (a real incident gets classified as “info” because history had a coincidental false match). Three things handle most of this.

First, a rate limit on the alert itself: at most N calls to send_priority_alert per hour, regardless of what the agent decides, to keep a once-a-minute reassessment from turning into a flood. Second, logging every decision along with the reasoning the model gave, not just the outcome. That’s what lets you, in a retro after an incident, answer “why was this classified as info” without guessing. Third, periodic human review of decisions classified as info: not to approve each one in real time (that would defeat the point of automating it), but to audit in batches, weekly, whether the classification pattern still makes sense.

One difference from the risk in post 1 matters here. There, the agent reasoned over outside text like logs, which can be tampered with. Here, the input is numeric telemetry from Azure Monitor. The prompt-injection surface is tiny because there is no arbitrary third-party text entering the context. Not every agent carries the same risk profile, so the checklist should change with the workload.

What this step buys you

This is the point where the watchdog stops being a threshold alarm and starts being an operator aid. It still cannot fix anything, and that is the right call. In the next post, I wire it to the AKS diagnoser so the alert comes with a candidate cause instead of a shrug.

Companion repository: agentic-infra-handbook

Leia este post em Português.