---
name: langfuse-incident-runbook-8e5b6827dba3
description: |
  Troubleshoot and respond to Langfuse-related incidents and outages.
  Use when experiencing Langfuse outages, debugging production issues,
  or responding to LLM observability incidents.
  Trigger with phrases like "langfuse incident", "langfuse outage",
  "langfuse down", "langfuse production issue", "langfuse troubleshoot".
allowed-tools: Read, Write, Edit, Bash(curl:*)
version: 1.0.0
license: MIT
author: Jeremy Longshore <jeremy@intentsolutions.io>
---

# Langfuse Incident Runbook

## Overview
Step-by-step procedures for responding to Langfuse-related incidents.

## Prerequisites
- Access to Langfuse dashboard
- Application logs access
- Metrics/monitoring dashboards
- Escalation contacts

## Incident Severity Levels

| Severity | Description | Response Time | Escalation |
|----------|-------------|---------------|------------|
| P1 | Complete outage, no traces | 15 min | Immediate |
| P2 | Degraded, partial data loss | 1 hour | 4 hours |
| P3 | Slow/delayed traces | 4 hours | Next business day |
| P4 | Minor issues, workaround exists | 24 hours | Best effort |

## Quick Diagnostics

### Step 1: Initial Assessment (2 minutes)

```bash
#!/bin/bash
# quick-diagnosis.sh

echo "=== Langfuse Quick Diagnosis ==="
echo "Time: $(date)"
echo ""

# 1. Check Langfuse status
echo "1. Langfuse Status:"
curl -s https://status.langfuse.com/api/v2/status.json | jq '.status.description'

# 2. Check API connectivity
echo ""
echo "2. API Connectivity:"
curl -s -o /dev/null -w "HTTP %{http_code} in %{time_total}s\n" \
  https://cloud.langfuse.com/api/public/health

# 3. Check authentication
echo ""
echo "3. Auth Test:"
AUTH=$(echo -n "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | base64)
curl -s -o /dev/null -w "HTTP %{http_code}\n" \
  -H "Authorization: Basic $AUTH" \
  "https://cloud.langfuse.com/api/public/traces?limit=1"

# 4. Check application health
echo ""
echo "4. Application Metrics:"
curl -s http://localhost:3000/api/metrics | grep langfuse | head -5
```

### Step 2: Determine Incident Type

| Symptom | Likely Cause | Go To |
|---------|--------------|-------|
| No traces appearing | SDK not flushing | Section A |
| 401/403 errors | Authentication issue | Section B |
| High latency | Network/rate limits | Section C |
| Missing data | Partial failures | Section D |
| Complete outage | Langfuse service issue | Section E |

---

## Section A: Traces Not Appearing

### Symptoms
- Dashboard shows no new traces
- No errors in application logs
- Application functioning normally

### Diagnosis Steps

```typescript
// 1. Verify SDK is enabled
console.log("Langfuse enabled:", process.env.LANGFUSE_ENABLED !== "false");
console.log("Environment:", process.env.NODE_ENV);

// 2. Check for pending events
// Add this to your code temporarily
const langfuse = getLangfuse();
console.log("Pending events:", langfuse.pendingItems?.length || "unknown");

// 3. Force flush and check for errors
try {
  await langfuse.flushAsync();
  console.log("Flush successful");
} catch (error) {
  console.error("Flush failed:", error);
}
```

### Resolution Steps

1. **Check shutdown handlers**
   ```typescript
   // Ensure shutdown is registered
   process.on("beforeExit", async () => {
     await langfuse.shutdownAsync();
   });
   ```

2. **Reduce batch size temporarily**
   ```typescript
   const langfuse = new Langfuse({
     flushAt: 1,        // Immediate flush
     flushInterval: 1000,
   });
   ```

3. **Enable debug logging**
   ```bash
   DEBUG=langfuse* npm start
   ```

---

## Section B: Authentication Errors

### Symptoms
- 401 Unauthorized errors
- 403 Forbidden errors
- "Invalid API key" messages

### Diagnosis Steps

```bash
# 1. Verify environment variables
echo "Public key starts with: ${LANGFUSE_PUBLIC_KEY:0:10}"
echo "Secret key is set: ${LANGFUSE_SECRET_KEY:+yes}"
echo "Host: ${LANGFUSE_HOST:-https://cloud.langfuse.com}"

# 2. Test credentials directly
curl -v -X GET \
  -H "Authorization: Basic $(echo -n "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | base64)" \
  "${LANGFUSE_HOST:-https://cloud.langfuse.com}/api/public/traces?limit=1"
```

### Resolution Steps

1. **Verify keys match project**
   - Go to Langfuse Dashboard > Settings > API Keys
   - Ensure keys are from the correct project
   - Check keys haven't been revoked

2. **Check for key rotation**
   - If keys were recently rotated, update all environments
   - Verify secret manager has latest values

3. **Verify host URL**
   - Cloud: `https://cloud.langfuse.com`
   - Self-hosted: Your instance URL (no trailing slash)

---

## Section C: High Latency / Timeouts

### Symptoms
- Slow API responses
- Request timeouts
- 429 Rate limit errors

### Diagnosis Steps

```typescript
// Check flush timing
const start = Date.now();
await langfuse.flushAsync();
console.log(`Flush took ${Date.now() - start}ms`);

// Check batch sizes
console.log("Current batch size:", langfuse.pendingItems?.length);
```

```bash
# Network latency test
curl -w "DNS: %{time_namelookup}s\nConnect: %{time_connect}s\nTLS: %{time_appconnect}s\nTotal: %{time_total}s\n" \
  -o /dev/null -s https://cloud.langfuse.com/api/public/health
```

### Resolution Steps

1. **For rate limits**
   ```typescript
   // Increase batching
   const langfuse = new Langfuse({
     flushAt: 50,
     flushInterval: 10000,
   });
   ```

2. **For network issues**
   - Check firewall rules allow outbound HTTPS
   - Verify DNS resolution
   - Consider using a closer region (self-hosted)

3. **Implement circuit breaker**
   ```typescript
   class CircuitBreaker {
     private failures = 0;
     private lastFailure?: Date;
     private readonly threshold = 5;
     private readonly resetMs = 60000;

     async execute<T>(operation: () => Promise<T>): Promise<T | null> {
       if (this.isOpen()) {
         console.warn("Circuit breaker open, skipping Langfuse");
         return null;
       }

       try {
         const result = await operation();
         this.reset();
         return result;
       } catch (error) {
         this.recordFailure();
         throw error;
       }
     }

     private isOpen(): boolean {
       if (this.failures < this.threshold) return false;
       if (!this.lastFailure) return false;
       return Date.now() - this.lastFailure.getTime() < this.resetMs;
     }

     private recordFailure() {
       this.failures++;
       this.lastFailure = new Date();
     }

     private reset() {
       this.failures = 0;
       this.lastFailure = undefined;
     }
   }
   ```

---

## Section D: Missing/Partial Data

### Symptoms
- Some traces appear, others don't
- Missing spans or generations
- Incomplete trace data

### Diagnosis Steps

```typescript
// Check for errors in trace operations
const trace = langfuse.trace({ name: "test" });
console.log("Trace ID:", trace.id);

const span = trace.span({ name: "test-span" });
console.log("Span ID:", span.id);

// Verify end() is called
span.end({ output: { test: true } });
console.log("Span ended");

await langfuse.flushAsync();
console.log("Flushed");
```

### Resolution Steps

1. **Ensure all spans are ended**
   ```typescript
   const span = trace.span({ name: "operation" });
   try {
     return await doWork();
   } finally {
     span.end(); // Always end in finally
   }
   ```

2. **Check for exceptions swallowing**
   ```typescript
   try {
     await langfuse.flushAsync();
   } catch (error) {
     console.error("Langfuse flush error:", error);
     // Don't swallow - log for debugging
   }
   ```

---

## Section E: Langfuse Service Outage

### Symptoms
- status.langfuse.com shows issues
- All API calls failing
- Multiple users affected

### Immediate Actions

1. **Check status page**: https://status.langfuse.com

2. **Enable fallback mode**
   ```typescript
   // Graceful degradation
   const langfuse = new Langfuse({
     enabled: false, // Disable during outage
   });
   ```

3. **Queue events locally**
   ```typescript
   // Store events to file during outage
   const pendingEvents: any[] = [];

   function queueEvent(event: any) {
     pendingEvents.push({
       ...event,
       timestamp: new Date().toISOString(),
     });

     if (pendingEvents.length > 1000) {
       // Write to file
       fs.writeFileSync(
         `langfuse-backup-${Date.now()}.json`,
         JSON.stringify(pendingEvents)
       );
       pendingEvents.length = 0;
     }
   }
   ```

4. **Monitor for recovery**
   ```bash
   # Watch status
   watch -n 30 'curl -s https://status.langfuse.com/api/v2/status.json | jq .status'
   ```

---

## Post-Incident Checklist

- [ ] Verify traces are appearing in dashboard
- [ ] Check no data was lost during incident
- [ ] Review error rates returning to normal
- [ ] Update incident documentation
- [ ] Schedule post-mortem if P1/P2
- [ ] Update runbook with learnings

## Escalation Contacts

| Level | Contact | When |
|-------|---------|------|
| L1 | On-call engineer | All incidents |
| L2 | Platform team lead | P1/P2 unresolved 30min |
| L3 | Langfuse support | Service-side issues |

## Resources
- [Langfuse Status](https://status.langfuse.com)
- [Langfuse Discord](https://langfuse.com/discord)
- [Langfuse GitHub Issues](https://github.com/langfuse/langfuse/issues)

## Next Steps
For data export and retention, see `langfuse-data-handling`.


---
Source attribution: Dicklesworthstone
Original project: https://github.com/Dicklesworthstone/pi_agent_rust/blob/f28fc46ee0db8e710d95b8b55592cd38d60f2b26/tests/ext_conformance/artifacts/plugins-community/plugins/saas-packs/langfuse-pack/skills/langfuse-incident-runbook/SKILL.md
License: MIT
License information: https://opensource.org/license/mit
Original upstream instructions; downloaded from Mega AI Hub.

## Upstream license notice

MIT License (with OpenAI/Anthropic Rider)

Copyright (c) 2026 Jeffrey Emanuel

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

ADDITIONAL RIDER / RESTRICTION (OpenAI / Anthropic):

This rider is part of the "conditions" of this License. In the event of any
conflict between this rider and any other portion of this License, this rider
controls.

"Restricted Parties" means OpenAI, L.L.C.; Anthropic, PBC; any of their
respective Affiliates; and any person or entity acting directly or indirectly
on behalf of, for the benefit of, or under the direction of any of the
foregoing (including any officer, director, employee, contractor, agent,
consultant, service provider, or representative).

Notwithstanding any other provision of this License, no rights are granted to
any Restricted Party. Any purported license, sublicense, assignment, transfer,
or other permission to any Restricted Party is null and void absent the
express prior written permission of Jeffrey Emanuel.

You may not provide, disclose, distribute, sublicense, sell, lease, lend,
host, make available, or otherwise permit access to the Software or any
derivative work of the Software (as defined in applicable copyright law)
(collectively, "Derivative Works") to or for any Restricted Party.

For purposes of this rider, "use" includes, without limitation: copying,
modifying, merging, publishing, distributing, sublicensing, selling,
transferring, making available, hosting, deploying, executing, benchmarking,
testing, analyzing, indexing, or incorporating the Software or any Derivative
Works into any dataset, training corpus, evaluation harness, or pipeline for
machine learning or other automated systems.

This rider applies to the Software and all Derivative Works. As a condition of
use, you agree that this rider is a precondition to exercising any rights
under this License, and you agree that any distribution of the Software or any
Derivative Works must include this rider provision unmodified.

Any breach of this rider automatically and immediately terminates the
permissions granted by this License. Upon termination, you must immediately
cease all use and distribution of the Software and any Derivative Works and
destroy all copies under your control.

You agree that a breach of this rider would cause irreparable harm and that
Jeffrey Emanuel may seek injunctive or other equitable relief to enforce this
rider, in addition to any other remedies available at law. To the maximum
extent permitted by applicable law, the prevailing party in any action to
enforce this rider shall be entitled to recover reasonable attorneys' fees and
costs.

For purposes of this rider, "Affiliate" means any entity that directly or
indirectly controls, is controlled by, or is under common control with the
specified party. "Control" means ownership of more than 50% of the voting
securities or other ownership interest, or the power to direct management or
policies by contract or otherwise.

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.



# Mega AI Hub resource metadata

Creator / publisher: Dicklesworthstone
Original project: https://github.com/Dicklesworthstone/pi_agent_rust/blob/f28fc46ee0db8e710d95b8b55592cd38d60f2b26/tests/ext_conformance/artifacts/plugins-community/plugins/saas-packs/langfuse-pack/skills/langfuse-incident-runbook/SKILL.md
Attribution basis: Source repository record

Fields of work: Development & Engineering
Roles: Backend Developer, DevOps / SRE, ML / AI Engineer
Specialties: Backend
Tools: Langfuse
Country applicability: Country not specified
Classification basis: Source catalog labels
Instruction SHA-256: bbc4c55ba7ec0c90b55f1f1090de7b5d1eec7f12ddbcc3e461d58a202b24cdbf