---
name: rag-be5f507e0886
description: Complete RAG (Retrieval-Augmented Generation) system for OpenClaw. Indexes chat sessions, workspace code, documentation, and skills into local ChromaDB for semantic search. Enables finding past solutions, code patterns, and decisions instantly. Uses local embeddings (all-MiniLM-L6-v2) with no API keys required. Automatically ingests and updates knowledge base from ~/.openclaw/agents/main/sessions and workspace files.
---

# OpenClaw RAG Knowledge System

**Retrieval-Augmented Generation for OpenClaw – Search chat history, code, docs, and skills with semantic understanding**

## Overview

This skill provides a complete RAG (Retrieval-Augmented Generation) system for OpenClaw. It indexes your entire knowledge base – chat transcripts, workspace code, skill documentation – and enables semantic search across everything.

**Key features:**
- 🧠 Semantic search across all conversations and code
- 📚 Automatic knowledge base management
- 🔍 Find past solutions, code patterns, decisions instantly
- 💾 Local ChromaDB storage (no API keys required)
- 🚀 Automatic AI integration – retrieves context transparently

## Installation

### Prerequisites

- Python 3.7+
- OpenClaw workspace

### Setup

```bash
# Navigate to your OpenClaw workspace
cd ~/.openclaw/workspace/skills/rag-openclaw

# Install ChromaDB (one-time)
pip3 install --user chromadb

# That's it!
```

## Quick Start

### 1. Index Your Knowledge

```bash
# Index all chat history
python3 ingest_sessions.py

# Index workspace code and docs
python3 ingest_docs.py workspace

# Index skill documentation
python3 ingest_docs.py skills
```

### 2. Search the Knowledge Base

```bash
# Interactive search mode
python3 rag_query.py -i

# Quick search
python3 rag_query.py "how to send SMS via voip.ms"

# Search by type
python3 rag_query.py "porkbun DNS" --type skill
python3 rag_query.py "chromedriver" --type workspace
python3 rag_query.py "Reddit automation" --type session
```

### 3. Check Statistics

```bash
# See what's indexed
python3 rag_manage.py stats
```

## Usage Examples

### Finding Past Solutions

Hit a problem? Search for how you solved it before:

```bash
python3 rag_query.py "cloudflare bypass selenium"
python3 rag_query.py "voip.ms SMS configuration"
python3 rag_query.py "porkbun update DNS record"
```

### Searching Through Codebase

Find specific code or documentation:

```bash
python3 rag_query.py --type workspace "unifi gateway API"
python3 rag_query.py --type workspace "SMS client"
```

### Quick Reference

Access skill documentation without digging through files:

```bash
python3 rag_query.py --type skill "how to monitor UniFi"
python3 rag_query.py --type skill "Porkbun tool usage"
```

### Programmatic Use

From within Python scripts or OpenClaw sessions:

```python
import sys
sys.path.insert(0, '/home/william/.openclaw/workspace/skills/rag-openclaw')
from rag_query_wrapper import search_knowledge, format_for_ai

# Search and get structured results
results = search_knowledge("Reddit account automation")
print(f"Found {results['count']} relevant items")

# Format for AI consumption
context = format_for_ai(results)
print(context)
```

## Files Reference

| File | Purpose |
|------|---------|
| `rag_system.py` | Core RAG class (ChromaDB wrapper) |
| `ingest_sessions.py` | Index chat history |
| `ingest_docs.py` | Index workspace files & skills |
| `rag_query.py` | Search interface (CLI & interactive) |
| `rag_manage.py` | Document management (stats, delete, reset) |
| `rag_query_wrapper.py` | Simple Python API for programmatic use |
| `README.md` | Full documentation |

## How It Works

### Indexing

**Sessions:**
- Reads `~/.openclaw/agents/main/sessions/*.jsonl`
- Handles OpenClaw event format (session metadata, messages, tool calls)
- Chunks messages (20 per chunk, 5 message overlap)
- Extracts and formats thinking, tool calls, results

**Workspace:**
- Scans for `.py`, `.js`, `.ts`, `.md`, `.json`, `.yaml`, `.sh`, `.html`, `.css`
- Skips files > 1MB and binary files
- Chunks long documents for better retrieval

**Skills:**
- Indexes all `SKILL.md` files
- Organized by skill name for easy reference

### Search

ChromaDB uses `all-MiniLM-L6-v2` embeddings to convert text to vectors. Similar meanings cluster together, enabling semantic search by *meaning* not just *keywords*.

### Automatic Integration

When the AI responds, it automatically:
1. Searches the knowledge base for relevant context
2. Retrieves past conversations, code, or docs
3. Includes that context in the response

This happens transparently – the AI "remembers" your past work.

## Management

### View Statistics

```bash
python3 rag_manage.py stats
```

Output:
```
📊 OpenClaw RAG Statistics

Collection: openclaw_knowledge
Total Documents: 635

By Source:
  session-001: 23
  my-script.py: 5
  porkbun: 12

By Type:
  session: 500
  workspace: 100
  skill: 35
```

### Delete Documents

```bash
# Delete all sessions
python3 rag_manage.py delete --by-type session

# Delete specific file
python3 rag_manage.py delete --by-source "scripts/voipms_sms_client.py"

# Reset entire collection
python3 rag_manage.py reset
```

### Add Manual Document

```bash
python3 rag_manage.py add \
  --text "API endpoint: https://api.example.com/endpoint" \
  --source "api-docs:example.com" \
  --type "manual"
```

## Configuration

### Custom Session Directory

```bash
python3 ingest_sessions.py --sessions-dir /path/to/sessions
```

### Chunk Size Control

```bash
python3 ingest_sessions.py --chunk-size 30 --chunk-overlap 10
```

### Custom Collection

```python
from rag_system import RAGSystem
rag = RAGSystem(collection_name="my_knowledge")
```

## Data Types

| Type | Source Format | Description |
|------|--------------|-------------|
| `session` | `session:{key}` | Chat history transcripts |
| `workspace` | `relative/path/to/file` | Code, configs, docs |
| `skill` | `skill:{name}` | Skill documentation |
| `memory` | `MEMORY.md` | Long-term memory entries |
| `manual` | `{custom}` | Manually added docs |
| `api` | `api-docs:{name}` | API documentation |

## Performance

- **Embedding model**: `all-MiniLM-L6-v2` (79MB, cached locally)
- **Storage**: ~100MB per 1,000 documents
- **Indexing**: ~1,000 documents/minute
- **Search**: <100ms (after first query)

## Troubleshooting

### No Results Found

```bash
# Check what's indexed
python3 rag_manage.py stats

# Try broader query
python3 rag_query.py "SMS"  # instead of "voip.ms SMS API endpoint"
```

### Slow First Search

First search loads embeddings (~1-2 seconds). Subsequent searches are instant.

### Duplicate ID Errors

```bash
# Reset and re-index
python3 rag_manage.py reset
python3 ingest_sessions.py
python3 ingest_docs.py workspace
```

### ChromaDB Model Download

First run downloads embedding model (79MB). Takes 1-2 minutes. Let it complete.

## Best Practices

### Re-index Regularly

After significant work:
```bash
python3 ingest_sessions.py  # New conversations
python3 ingest_docs.py workspace  # New code/changes
```

### Use Specific Queries

```bash
# Better
python3 rag_query.py "voip.ms getSMS method"

# Too broad
python3 rag_query.py "SMS"
```

### Filter by Type

```bash
# Looking for code
python3 rag_query.py --type workspace "chromedriver"

# Looking for past conversations
python3 rag_query.py --type session "Reddit"
```

### Document Decisions

After important decisions, add them manually:

```bash
python3 rag_manage.py add \
  --text "Decision: Use Playwright for Reddit automation. Reason: Cloudflare bypass handles" \
  --source "decision:reddit-automation" \
  --type "decision"
```

## Limitations

- Files > 1MB automatically skipped (performance)
- Python 3.7+ required
- ~100MB disk per 1,000 documents
- First search slower (embedding load)

## Integration with OpenClaw

This skill integrates seamlessly with OpenClaw:

1. **Automatic RAG**: AI automatically retrieves relevant context when responding
2. **Session history**: All conversations indexed and searchable
3. **Workspace awareness**: Code and docs indexed for reference
4. **Skill accessible**: Use from any OpenClaw session or script

## Security Considerations

**⚠️ Important Privacy Note:** This RAG system indexes local data, which may contain:
- API keys, tokens, or credentials in session transcripts
- Private messages or personal information
- Tool results with sensitive data
- Workspace configuration files

**Recommended:**
- Review session files before ingestion if concerned about privacy
- Consider redacting sensitive data from session files
- Use `rag_manage.py reset` to delete the entire index when needed
- The ChromaDB persistence at `~/.openclaw/data/rag/` can be deleted to remove all indexed data
- The auto-update script only runs local ingestion - no remote code fetching

**Path Portability:**
All scripts now use dynamic path resolution (`os.path.expanduser()`, `Path(__file__).parent`) for portability across different user environments. No hard-coded absolute paths remain in the codebase.

**Network Calls:**
- The embedding model (all-MiniLM-L6-v2) is downloaded by ChromaDB on first use via pip
- No custom network calls, HTTP requests, or sub-process network operations
- No telemetry or data uploaded to external services (ChromaDB telemetry disabled)
- All processing and storage is local-only

## Example Workflow

**Scenario:** You're working on a new automation but hit a Cloudflare challenge.

```bash
# Search for past Cloudflare solutions
python3 rag_query.py "Cloudflare bypass selenium"

# Result shows relevant past conversation:
# "Used undetected-chromedriver but failed. Switched to Playwright which handles challenges better."

# Now you know the solution before trying it!
```

## Moltbook Integration

Post RAG skill announcements and updates to Moltbook social network.

### Quick Post

```bash
# Post from draft file
python3 scripts/moltbook_post.py --file drafts/moltbook-post-rag-release.md

# Post directly
python3 scripts/moltbook_post.py "Title" "Content"
```

### Usage Examples

**Post release announcement:**
```bash
cd ~/.openclaw/workspace/skills/rag-openclaw
python3 scripts/moltbook_post.py --file drafts/moltbook-post-rag-release.md --submolt general
```

**Post quick update:**
```bash
python3 scripts/moltbook_post.py "RAG Update" "Fixed path portability issues"
```

**Post to submolt:**
```bash
python3 scripts/moltbook_post.py "Feature Drop" "New semantic search" "aiskills"
```

### Configuration

**To use Moltbook posting (optional feature):**

Set environment variable:
```bash
export MOLTBOOK_API_KEY="your-key"
```

Or create credentials file:
```bash
mkdir -p ~/.config/moltbook
cat > ~/.config/moltbook/credentials.json << EOF
{
  "api_key": "moltbook_sk_YOUR_KEY_HERE"
}
EOF
```

**Note:** Moltbook posting is optional for publishing RAG announcements. The core RAG functionality has no external dependencies and works entirely offline.

### Rate Limits

- **Posts:** 1 per 30 minutes
- **Comments:** 1 per 20 seconds

If rate-limited, wait for `retry_after_minutes` shown in error.

### Documentation

See `scripts/MOLTBOOK_POST.md` for full documentation and API reference.

## Repository

https://openclaw-rag-skill.projects.theta42.com

**Published:** clawhub.com
**Maintainer:** Nova AI Assistant
**For:** William Mantly (Theta42)

## License

MIT License - Free to use and modify

---
Source attribution: modbender
Original project: https://github.com/modbender/skill-library-mcp/blob/d9c8fe903b5b4e82237dbe8d0df8567783022e18/data/openclaw-rag-skill/SKILL.md
License: MIT
License information: https://opensource.org/license/mit
Original upstream instructions; downloaded from Mega AI Hub.

## Upstream license notice

MIT License

Copyright (c) 2025 skill-library-mcp contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.



# Mega AI Hub resource metadata

Creator / publisher: modbender
Original project: https://github.com/modbender/skill-library-mcp/blob/d9c8fe903b5b4e82237dbe8d0df8567783022e18/data/openclaw-rag-skill/SKILL.md
Attribution basis: Source repository record

Fields of work: Data & Analytics, Development & Engineering, Product Management
Roles: Backend Developer, Data Analyst, Data Scientist, ML / AI Engineer, Product Manager
Specialties: API, Backend
Tools: OpenAI API, Chromadb
Country applicability: Country not specified
Classification basis: Source catalog labels
Instruction SHA-256: 16b99c448113a5730a8924fbba0b623b250fff97af2a01b5633c0a8637ed8f76