How AI Agents Can Automate DevOps Runbooks Safely: Architecture, MCP & Human-in-the-Loop
The 3-Tier Autonomous Ops Architecture
An AI DevOps agent must never be a monolithic LLM prompt. Production-grade systems decouple the agent into three strictly governed tiers:
- Perception Tier (MCP & Telemetry): Receives webhooks from PagerDuty/Datadog and fetches diagnostic context using strictly defined Model Context Protocol (MCP) read-only tools.
- Reasoning & Plan Generation (LLM Core): Analyzes telemetry against runbook specifications, formulates a hypothesis, and generates a structured, deterministic remediation plan.
- Execution & Guardrails Tier (Deterministic Sandbox): Validates the plan against security policies, enforces Human-in-the-Loop (HITL) gates, and executes idempotent scripts with automated rollback verification.
Standardizing Tool Use with Model Context Protocol (MCP)
Instead of giving the LLM unrestricted bash or ssh access, expose purpose-built MCP servers. Every tool call is strongly typed, logged, and audited.
// Example MCP Tool Definition for Kubernetes Diagnostics
{
"name": "k8s_get_pod_diagnostics",
"description": "Retrieves recent pod logs, exit codes, and event history for a failing pod",
"parameters": {
"type": "object",
"properties": {
"namespace": {"type": "string", "description": "Kubernetes namespace"},
"pod_name": {"type": "string", "description": "Pod name"}
},
"required": ["namespace", "pod_name"]
}
}The 3-Tier Action Classification Matrix
| Tier | Classification | Examples | Execution Policy |
|---|---|---|---|
| Tier 0 | Read-Only Diagnostics | kubectl logs, CloudWatch metric queries, tracing |
Fully Autonomous (Instant) |
| Tier 1 | Non-Destructive Remediations | Restarting stateless pod, flushing Redis cache, scaling replicas | Autonomous with Threshold Guards |
| Tier 2 | Destructive / Stateful Operations | Node drain, database failover, DNS routing shift, schema migrate | Mandatory Human Approval (HITL) |
Slack-Based Approval Gateways
When an AI agent determines that a Tier 2 action is required to resolve an incident:
- The agent pauses execution and posts an interactive card to the
#sre-incidentsSlack channel detailing: Root Cause Hypothesis, Proposed Action, Expected Impact, and Rollback Command. - An authorized on-call engineer reviews the diff and clicks Approve.
- The Slack webhook transmits an authenticated cryptographic callback to the agent, which executes the remediation and reports verification metrics.
{
"incident_id": "INC-84920",
"agent": "MORPHEUS-v2",
"action": "scale_deployment",
"target": "checkout-service",
"approved_by": "naveed@naveedkumbhar.com",
"pre_check_latency_p99": "1420ms",
"post_check_latency_p99": "180ms",
"status": "RESOLVED_SUCCESSFULLY"
}Have questions about this architecture or scaling your infrastructure?
Whether you're planning a complex cloud migration, optimizing Kubernetes reliability, or designing autonomous AI workflows, I'm always open to discussing architecture and technical challenges with engineering teams.
Connect with Naveed on LinkedIn →Frequently Asked Questions
What is the Model Context Protocol (MCP) and why is it essential for DevOps AI agents?
The Model Context Protocol (MCP), open-sourced by Anthropic, provides an open standard for AI models to securely connect to external tools, databases, and APIs. In DevOps, MCP servers expose controlled, strictly typed interfaces to Kubernetes clusters, CloudWatch metrics, or PagerDuty APIs, replacing dangerous, unconstrained bash execution with auditable JSON-RPC tool calls.
How do you implement Human-in-the-Loop (HITL) for high-risk operations?
Human-in-the-Loop is implemented by classifying actions into risk tiers. High-risk operations (e.g. database failover, node drain, DNS modification) generate an interactive confirmation payload sent to a dedicated Slack or Teams channel via webhook. The AI agent pauses execution until an authorized engineer clicks Approve or Deny.
How do you prevent prompt injection in DevOps AI agents?
Prompt injection is prevented by separating untrusted user data (e.g. error logs, pod descriptions, customer input) from system instructions, strictly validating all tool inputs against JSON schemas, and running all execution through hardened sandboxes with read-only defaults.
