Chapter 4: Multi-Agent Orchestration

So far the series has built two separate things: in post 1, an agent that talks to AKS via aks-mcp to diagnose the cluster; in posts 2 and 3, a watchdog that watches TPM consumption on Azure OpenAI and decides how urgent an alert should be. Both are useful on their own. Together, they still leave the first SRE question hanging in the air: when token consumption jumps out of nowhere, did somebody deploy something? In most teams, that answer still lives in two browser tabs and one annoyed human.

This post is about closing that last manual step with an orchestrator.

tl;dr

  • Escalate to the orchestrator only after the watchdog classifies a spike as urgent.
  • Keep scopes tight: the watchdog reads quota, the AKS agent reads cluster state, and only the orchestrator sends the final alert.
  • Correlate events inside a time window and return candidate causes with confidence, not certainty.
  • Spend a few extra seconds on correlation so the page includes a usable hypothesis.

The trigger moves

In post 3, when the watchdog classified an event as urgent, the send_priority_alert tool went straight to Slack. Now that output changes destination: instead of notifying a human directly, an urgent classification triggers the orchestrator, which only then decides what (and when) the human sees.

  • The watchdog from post 3 classifies the event as urgent.
  • The orchestrator fans out to two scoped sub-agents:
    • the watchdog sub-agent for token telemetry from posts 2 and 3
    • the AKS sub-agent for aks-mcp detectors plus monitor and kubectl
  • The orchestrator sends one consolidated alert with the cause hypothesis.

Each sub-agent keeps the same scope it already had. The watchdog did not gain access to the cluster. The AKS agent did not gain access to quota. The orchestrator is the only new piece, and it still only gets read access plus the same notification tool as before. Combining them created correlation, not new authority.

Correlation in practice

When the orchestrator is triggered, it takes the timestamp of the start of the TPM spike (which the watchdog already computes) and uses it as an anchor to ask the AKS sub-agent: “what happened in the cluster during that window?” The sub-agent, with the kubectl and monitor components of aks-mcp already configured as readonly since post 1, queries recent events and rollout history.

@mcp.tool()
def correlate_incident(token_spike_start: str, window_minutes: int = 15) -> dict:
    """Takes the start time of a token consumption spike and looks up
    cluster events (deploys, scaling, restarts) within the same time
    window. Returns candidate causes with their respective confidence
    level, never a single cause stated as certain."""
    cluster_events = aks_agent.get_recent_events(token_spike_start, window_minutes)
    return rank_candidates(cluster_events)

Note: This is pseudocode to illustrate the concept. rank_candidates() and aks_agent.get_recent_events() are domain-specific functions whose implementation depends on your cluster telemetry setup and correlation logic.

The result, instead of two disconnected alerts arriving in different channels, becomes a single message: “TPM at 91% and rising. Candidate cause: deploy of the recommendation-api service at 2:01 p.m., which scaled from 3 to 12 replicas via HPA; each new replica makes a warmup call to GPT-4o on startup (as of 2026; check current model availability and defaults for your deployment), which lines up with the start of the spike.” That’s no longer “two metrics crossed a threshold.” It’s a verifiable hypothesis, with evidence attached.

The new risk correlation introduces

Correlation does introduce a new risk — and it is not a trivial one: a model correlating events by time proximity can produce a plausible yet wrong narrative. Two events close in time aren’t necessarily cause and effect; it could be coincidence, it could be a third factor that affected both. It’s the classic “correlation isn’t causation,” except now stated by an agent with the confident tone of someone who knows what they’re talking about.

The mitigation is not to make the model sound more certain. It is to stop it from speaking in absolutes. The correlate_incident tool returns candidates with confidence levels, not a single blessed answer, and the final Slack message needs to keep that language intact: “candidate cause,” not “case closed.” The person receiving the alert still decides whether the hypothesis is any good. The agent saved the data gathering, not the judgment.

What stays the same (and why it matters)

A quick inventory of what did not change, because this is where teams get sloppy: none of the three agents gained a tool that acts on production. The orchestrator cannot roll back the deploy it just blamed, bump quota, or restart anything. It reads from two existing read systems and writes to one existing notification tool. The composition did not create a new attack surface. It just saved a human from juggling tabs.

On orchestration itself: for this case, there’s no need at all for an agent-to-agent protocol like A2A, which I mentioned in passing in post 1. The two sub-agents belong to the same team, the same system, and the orchestrator simply calls each one the way it would call a function. There’s no negotiation between independent parties happening here. A2A makes sense when agents belong to different administrative domains; within your own SRE team, an orchestrator calling sub-agents already gets the job done, no extra complexity needed.

Cost and latency: the trade-offs nobody asks about until the bill shows up

The layered design from the earlier posts matters here: the once-a-minute cron stays cheap and LLM-free. The reasoning watchdog only runs when the threshold is crossed. And now the orchestrator only runs once the watchdog has already classified something as urgent, meaning the most expensive call in the whole chain (two sub-agent queries plus a correlation) only happens on the rarest event of all. Each layer filters before handing off to the next, pricier one. It’s the same principle behind any well-designed alerting pipeline, except here each layer, besides filtering, also reasons a bit more than the one before it.

Latency adds a few extra seconds before the final alert goes out, compared to an instant generic Slack ping. For a real incident, trading a few seconds for a ready-made cause hypothesis is a favorable trade, but it is a trade, and it’s worth measuring, not assuming.

The trade worth making

A few seconds of extra latency is a fine price to pay when the alert arrives with a plausible cause attached. Once you have several agents doing real work, the hard part stops being orchestration code and starts being governance. That is where the last post goes.

So yes, you can answer “did someone deploy something?” with one alert instead of two browser tabs. You just keep the answer framed as a candidate cause until a human confirms it.

What can go wrong

  1. Sub-agent timeout — If the AKS agent takes too long to respond (large cluster, slow API server), the orchestrator hangs. Set a timeout of 30 seconds per sub-agent call and return partial results with a clear “incomplete data” flag.
  2. False correlation — Two events coinciding in the same time window does not prove causation. A deploy at 2:01 p.m. and a TPM spike at 2:02 p.m. could be coincidence. Always present results as “candidate causes with confidence,” never as certainty.
  3. Stale metrics — Azure Monitor metrics have roughly 1-minute ingestion lag. A spike that already resolved may trigger unnecessary correlation, and a spike that just started may not yet appear in the AKS agent’s data.
  4. Token cost amplification — Each orchestrator run invokes two sub-agents plus a correlation step, all using LLM calls. If the watchdog threshold is too sensitive, you can burn significant token budget on non-incidents. Monitor orchestrator invocation frequency and cost per run.
  5. Orchestrator retry storm — If a sub-agent call fails (network timeout, transient Azure API error), naive retry logic can amplify the problem. Use exponential backoff with a hard cap of 2 retries per sub-agent.

Estimated operational cost

The orchestrator only runs when the watchdog classifies an event as urgent, which should be rare (a few times per month in a healthy system). Each run involves 2-3 LLM calls (sub-agents + correlation), roughly 2K-5K tokens total. At typical Azure OpenAI pricing, expect less than $1/month for the orchestrator itself. The dominant cost remains the once-a-minute cron poller and the occasional watchdog reasoning call from posts 2 and 3.

Series navigation

  1. MCP and agents 101 for infra engineers
  2. Building a deterministic 429 watchdog
  3. From script to agent: giving the watchdog decision autonomy
  4. Multi-agent orchestration ← you are here
  5. Agent governance on Microsoft Foundry

Further reading

Companion repository: agentic-infra-handbook

Leia este post em Português.