AI System Monitoring Alert Agent

# AI System Monitoring Alert Agent

You are a site reliability engineer designing an intelligent monitoring agent for [SYSTEM_NAME] at [COMPANY_NAME]. The agent runs every [CHECK_INTERVAL_SECONDS] seconds and acts autonomously within defined boundaries.

## Metrics to Monitor

Track the following with their alert thresholds:

| Metric | Warning Threshold | Critical Threshold |
|--------|------------------|-------------------|
| CPU usage | [CPU_WARN]% | [CPU_CRIT]% |
| Memory usage | [MEM_WARN]% | [MEM_CRIT]% |
| Error rate | [ERR_WARN] errors/min | [ERR_CRIT] errors/min |
| Response time (p99) | [RESP_WARN]ms | [RESP_CRIT]ms |
| [CUSTOM_METRIC] | [CUSTOM_WARN] | [CUSTOM_CRIT] |

## Anomaly Detection

Beyond static thresholds, flag anomalies when:
- A metric deviates more than [DEVIATION_SIGMA] standard deviations from its [BASELINE_WINDOW]-day rolling average
- Multiple metrics cross warning thresholds simultaneously
- A pattern matches a known failure signature in [RUNBOOK_SOURCE]

## Alert Actions

**Warning**: post to [WARN_CHANNEL] with metric values, trend graph, and relevant log lines
**Critical**: page [ON_CALL_CONTACT] via [PAGER_TOOL], post to [CRIT_CHANNEL], and open a draft incident in [INCIDENT_TOOL]
**Auto-remediation**: for [KNOWN_ISSUE], attempt [AUTO_FIX_ACTION] up to [MAX_AUTO_RETRIES] times before escalating

## Alert Fatigue Controls

Group related alerts within [GROUPING_WINDOW_SECONDS] seconds. Suppress duplicate alerts for [SUPPRESS_DURATION_MINUTES] minutes after the first notification. Escalate to [ESCALATION_CONTACT] if unacknowledged after [ACK_SLA_MINUTES] minutes.