--- name: anth-incident-runbook description: 'Execute incident response procedures for Claude API outages and degradation. Use when Claude API is returning errors, experiencing high latency, or showing degraded performance in production. Trigger with phrases like "anthropic incident", "claude api down", "anthropic outage", "claude degraded", "anthropic runbook". ' allowed-tools: Read, Bash(curl:*), Grep version: 1.0.0 license: MIT author: Jeremy Longshore tags: - saas - ai - anthropic compatibility: Designed for Claude Code --- # Anthropic Incident Runbook ## Severity Classification | Severity | Condition | Response Time | |----------|-----------|---------------| | P1 | API returning 500/529 for all requests | Immediate | | P2 | Rate limiting (429) or high latency (>10s p99) | 15 minutes | | P3 | Intermittent errors (<5% error rate) | 1 hour | | P4 | Degraded quality (not errors) | Next business day | ## Immediate Triage (First 5 Minutes) ```bash # 1. Check Anthropic status page curl -s https://status.anthropic.com/api/v2/status.json | python3 -c \ "import sys,json; d=json.load(sys.stdin); print(d['status']['indicator'], '-', d['status']['description'])" # 2. Test API connectivity curl -s -w "\nHTTP %{http_code} | Time: %{time_total}s\n" \ https://api.anthropic.com/v1/messages \ -H "x-api-key: $ANTHROPIC_API_KEY" \ -H "anthropic-version: 2023-06-01" \ -H "content-type: application/json" \ -d '{"model":"claude-haiku-4-20250514","max_tokens":8,"messages":[{"role":"user","content":"1"}]}' # 3. Check rate limit headers curl -s -D - https://api.anthropic.com/v1/messages \ -H "x-api-key: $ANTHROPIC_API_KEY" \ -H "anthropic-version: 2023-06-01" \ -H "content-type: application/json" \ -d '{"model":"claude-haiku-4-20250514","max_tokens":8,"messages":[{"role":"user","content":"1"}]}' \ 2>/dev/null | grep -i "ratelimit\|retry-after\|request-id" ``` ## Decision Tree ``` API returning errors? ├── 401/403 → Key issue → Check ANTHROPIC_API_KEY is set and valid ├── 429 → Rate limited → Check headers, reduce traffic, wait for retry-after ├── 500 → Server error → Check status.anthropic.com, retry with backoff ├── 529 → Overloaded → Temporary, retry after 30-60s └── Timeouts → Network or long generation → Increase timeout, check max_tokens ``` ## Mitigation Actions ### Rate Limiting (429) ```python # Immediate: reduce traffic # 1. Enable circuit breaker # 2. Queue non-critical requests # 3. Switch to Message Batches for bulk work # 4. Reduce max_tokens to shorten generation time ``` ### API Outage (500/529) ```python # Graceful degradation def get_response_with_fallback(prompt: str) -> str: try: msg = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{"role": "user", "content": prompt}] ) return msg.content[0].text except (anthropic.InternalServerError, anthropic.APIStatusError): return "Our AI assistant is temporarily unavailable. Please try again shortly." ``` ### Key Compromise ```bash # 1. Immediately revoke key at console.anthropic.com # 2. Generate new key # 3. Deploy new key to all environments # 4. Audit recent usage for unauthorized calls # 5. File incident report ``` ## Postmortem Template ```markdown ## Incident: [Title] - **Duration:** [start] to [end] - **Severity:** P[1-4] - **Impact:** [what users experienced] - **Root Cause:** [what went wrong] - **Detection:** [how we found out] - **Mitigation:** [what we did to fix it] - **Request IDs:** [from debug logs] - **Action Items:** - [ ] [preventive measure 1] - [ ] [preventive measure 2] ``` ## Error Handling | Symptom | Likely Cause | Quick Fix | |---------|-------------|-----------| | All requests fail 401 | Key rotated/expired | Check Console for active keys | | Sudden 429 spike | Traffic burst or tier change | Check rate limit headers | | Slow responses (>10s) | Large max_tokens or complex prompt | Reduce max_tokens, use Haiku | | Intermittent 500s | Upstream API issue | Check status.anthropic.com | ## Resources - [API Status](https://status.anthropic.com) - [Error Reference](https://docs.anthropic.com/en/api/errors) - [Rate Limits](https://docs.anthropic.com/en/api/rate-limits) ## Next Steps For data compliance, see `anth-data-handling`. --- Source attribution: jeremylongshore Original project: https://github.com/jeremylongshore/claude-code-plugins-plus-skills/blob/main/plugins/saas-packs/anthropic-pack/skills/anth-incident-runbook/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: jeremylongshore Original project: https://github.com/jeremylongshore/claude-code-plugins-plus-skills/blob/main/plugins/saas-packs/anthropic-pack/skills/anth-incident-runbook/SKILL.md Attribution basis: Source repository record Fields of work: Development & Engineering, Customer Support Roles: DevOps / SRE, ML / AI Engineer, Technical Support Specialties: API, Backend, Design, DevOps, Frontend, Testing Tools: Anthropic Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 2c826a0a370942c66490c6b60563500bce0dea38170958a5a3d3929d19b5814d