Postmortems on Azure: automation with Azure DevOps and learning metrics (Part 2)

Second post in the Azure postmortem series. In Part 1, we built the foundation: blameless culture, a reusable template, KQL-based evidence collection, and Logic Apps automation. Now we move from documentation to operations.

A mature postmortem process should leave traces in the engineering system: linked work items, measurable trends, dashboards, and visible feedback into reliability practices such as SLOs, alert tuning, and chaos experiments. If a postmortem ends as a document nobody operationalizes, the process failed.

Integrating postmortems with Azure DevOps

Azure DevOps is the natural place to track remediation work, especially when incident improvements compete with feature work. The point is not to “store the document in DevOps.” The point is to make the learning executable.

Creating a work item for postmortem follow-up

You can use the built-in Bug type or introduce a custom work item type for incident reviews. Even if you stay with Bug, standardize fields and tags so the portfolio becomes queryable.

# Configure Azure DevOps CLI first
az extension add --name azure-devops

# Set the organization and project defaults
az devops configure --defaults organization=https://dev.azure.com/myorg project=MyProject

# Create a work item for a new postmortem
az boards work-item create \
  --type "Bug" \
  --title "Postmortem: [Incident Title]" \
  --description "$(cat postmortem-template.md)" \
  --area "\\MyProject\\SRE" \
  --iteration "\\MyProject\\Current-Sprint" \
  --fields "System.Tags=postmortem;sev-1;action-required" \
           "Microsoft.VSTS.Common.Priority=1" \
           "Microsoft.VSTS.Common.Severity=1 - Critical"

Recommended Azure DevOps structure

WIQL query for open postmortem action items

SELECT [System.Id], [System.Title], [System.State],
       [System.AssignedTo], [Microsoft.VSTS.Scheduling.TargetDate],
       [System.Tags]
FROM WorkItems
WHERE [System.TeamProject] = @project
  AND [System.Tags] CONTAINS "postmortem"
  AND [System.State] <> "Closed"
  AND [System.State] <> "Done"
ORDER BY [Microsoft.VSTS.Common.Priority] ASC,
         [Microsoft.VSTS.Scheduling.TargetDate] ASC

Useful Azure DevOps dashboard widgets

Postmortem metrics: are you learning or just documenting?

If you want executive sponsorship for postmortems, show outcomes. The best metrics are not vanity counts. They measure whether the organization learns faster and repeats failures less often.

Core postmortem metrics

The important word in that table is trend. You do not need one magic number. You need evidence that the organization gets better over time.

KQL queries for incident metrics

Calculating MTTD and MTTR

// Calculating incident MTTD and MTTR (using custom logs or Sentinel)
let incidents = datatable(
    IncidentId: string,
    ImpactStart: datetime,
    DetectionTime: datetime,
    ResolutionTime: datetime,
    Severity: string
) [
    "INC-001", datetime("2026-04-01 14:00"), datetime("2026-04-01 14:03"), datetime("2026-04-01 15:30"), "SEV-1",
    "INC-002", datetime("2026-04-05 09:15"), datetime("2026-04-05 09:20"), datetime("2026-04-05 10:00"), "SEV-2",
    "INC-003", datetime("2026-04-10 22:00"), datetime("2026-04-10 22:15"), datetime("2026-04-10 23:45"), "SEV-1"
];
incidents
| extend
    MTTD_minutes = datetime_diff('minute', DetectionTime, ImpactStart),
    MTTR_minutes = datetime_diff('minute', ResolutionTime, DetectionTime),
    TotalDuration_minutes = datetime_diff('minute', ResolutionTime, ImpactStart)
| summarize
    AvgMTTD = round(avg(MTTD_minutes), 1),
    AvgMTTR = round(avg(MTTR_minutes), 1),
    AvgTotal = round(avg(TotalDuration_minutes), 1),
    P95_MTTR = round(percentile(MTTR_minutes, 95), 1),
    IncidentCount = count()
  by Severity

Detecting recurring incidents

Recurring failures are where postmortems either prove their value or expose that the organization is mostly writing documents.

// Identify recurring incident patterns
AppExceptions
| where TimeGenerated > ago(90d)
| summarize
    OccurrenceCount = count(),
    DistinctDays = dcount(bin(TimeGenerated, 1d)),
    FirstOccurrence = min(TimeGenerated),
    LastOccurrence = max(TimeGenerated),
    AffectedOperations = dcount(OperationName)
  by ProblemId, ExceptionType = tostring(split(Type, ".")[-1])
| where DistinctDays > 3
| extend RecurrencePattern = iff(DistinctDays > 7, "Chronic", "Intermittent")
| order by OccurrenceCount desc

Use this output during quarterly reliability reviews. High recurrence almost always means one of three things: the fix was too local, the action item was never done, or the team misdiagnosed the real contributor.

Creating an Azure Workbook for postmortem metrics

Azure Workbooks are ideal for turning postmortem data into something leaders and engineers can both consume.

// Pseudocode - adapt to actual Workbook ARM schema
{
  "version": "Notebook/1.0",
  "items": [
    {
      "type": "markdown",
      "content": "## Incident learning and postmortem metrics"
    },
    {
      "type": "query",
      "title": "MTTD and MTTR - monthly trend",
      "query": "// MTTD/MTTR query grouped by month",
      "visualization": "linechart"
    },
    {
      "type": "query",
      "title": "Recurring incident rate",
      "query": "// Recurring incident query",
      "visualization": "piechart"
    },
    {
      "type": "query",
      "title": "Top 10 exceptions - last 30 days",
      "query": "AppExceptions | where TimeGenerated > ago(30d) | summarize count() by ProblemId | top 10 by count_",
      "visualization": "barchart"
    }
  ]
}

A practical dashboard usually includes four sections:

1. Incident volume and severity trend
2. MTTD and MTTR trend over time
3. Recurring exceptions and recurring services
4. Action item health: open, overdue, completed

Connecting postmortems to the full SRE cycle

A postmortem should close the loop across the rest of your reliability program.

From postmortems to SLOs

Every meaningful review should ask:

• Which SLO was violated?
• How much Error Budget did we burn?
• Was detection slow because the wrong indicator was measured?
• Do we need to change the SLO, or improve the system so it can meet the existing one?

A common decision path looks like this:

Postmortem -> Error Budget consumed > 50% -> Freeze feature releases
                                           -> Focus sprint on reliability
                                           -> Prioritize P1 action items

From postmortems to chaos engineering

Incidents reveal the failure scenarios you should have tested proactively.

Postmortem -> Root cause: timeout in dependency X
           -> Action item: create a Chaos Studio experiment
           -> Scenario: inject latency into dependency X
           -> Validation: does the circuit breaker trip in < 5 seconds?

From postmortems to alerts and runbooks

Advanced scenarios

Automated postmortem draft with an SRE agent

If your organization uses an internal SRE assistant or Azure-focused troubleshooting agent, the most useful automation is getting the first draft on the page:

1. Rebuild the timeline from Azure Monitor, Activity Logs, and Resource Health.
2. Correlate recent changes with the first signs of degradation.
3. Suggest candidate action items based on recurring gaps.
4. Produce a draft that the facilitator reviews before the meeting.

The boundary is simple: use automation for collection and synthesis, not for assigning blame or declaring root cause with false certainty.

Cross-cloud postmortems

Many modern incidents span Azure plus another provider or SaaS boundary. In those cases, extend the postmortem with a few extra sections.

Error Budget-triggered review workflows

# Example alert based on Error Budget burn that forces a review
az monitor metrics alert create \
  --resource-group rg-monitoring \
  --name "error-budget-critical" \
  --scopes "/subscriptions/{sub}/resourceGroups/rg-app/providers/Microsoft.Insights/components/app-production" \
  --condition "total requests/failed > 50" \
  --window-size 1h \
  --evaluation-frequency 5m \
  --action ag-postmortem-trigger \
  --description "Failed request count above threshold - investigate Error Budget impact" \
  --severity 1

Running the review meeting well

The meeting is where an incident stops being a Slack story and becomes shared engineering learning.

Before the meeting

During the meeting

1. Restate the blameless principle in the first minute.
2. Walk the timeline chronologically and correct missing context.
3. Discuss root cause and contributing factors using structured questioning.
4. Capture what worked, what did not, and where luck helped.
5. Define action items with owner, deadline, and completion criteria.
6. End on time with explicit next steps.

The 5 Whys technique in practice

Incident: Payment API returned 500s for 45 minutes

Why? -> The database connection pool was exhausted.

Why? -> One query took 30 seconds instead of 200 ms.

Why? -> The transactions table index was removed.

Why? -> A CI/CD migration removed the index.

Why? -> The migration was not reviewed by someone with
         production schema expertise.

Action item: Require peer review for database migrations,
             including an index impact checklist.

Golden rules for the facilitator

• Never allow “Person X should have…” to stay unchallenged.
• If someone becomes defensive, acknowledge complexity and redirect to the system.
• Park side topics instead of letting the meeting drift.
• Pull in quieter participants explicitly; they often have the observations everyone else missed.
• Keep the cadence efficient. If the process feels wasteful, teams will stop taking it seriously.

Common mistakes and how to avoid them

Conclusion and implementation checklist

A blameless Azure postmortem process is not done when the document is approved. It is done when the system changes.

Implementation checklist

1. Adopt the template from Part 1.
2. Save the KQL queries needed for fast evidence collection.
3. Trigger draft creation automatically when critical alerts resolve.
4. Track every postmortem and action item in Azure DevOps.
5. Build a Workbook for MTTD, MTTR, recurrence, and follow-up health.
6. Feed gaps back into alerts, runbooks, SLOs, and chaos scenarios.
7. Run quarterly trend reviews across all postmortems.

Part 1 covers the mechanics. Part 2 covers the operating model. Put together, they turn Azure incidents into a repeatable learning system instead of a cycle of institutional amnesia.

References

• Azure Well-Architected Framework - Incident response
• Azure Monitor overview
• Log Analytics overview
• Investigate failures with Application Insights
• Azure DevOps Work Items
• Azure Logic Apps overview
• Azure Workbooks overview
• Google SRE Book - Postmortem Culture
• KQL reference