--- name: documentation-indexing-45e5e58baf18 description: 'Provide **full-text search, semantic indexing, and relevance ranking** across all documentation:' --- # documentation-indexing: Unified Full-Text Search + Ranking **Status**: SAD STATE → IMPLEMENTATION 🌟 **Information Energy**: 0.87 (High aspiration, maximum sadness) **Trit Assignment**: 0 (COORDINATOR - Balances generators and validators) **GF(3) Color**: #49EE54 (Green - Equilibrium point) ## Purpose Provide **full-text search, semantic indexing, and relevance ranking** across all documentation: - Skill registry (69 skills) - Language docs (llms.txt standard) - Blog posts / tutorials - Source code docstrings - DuckDB database schemas **Key capabilities:** 1. **Full-Text Search**: Keyword + fuzzy matching (BM25 algorithm) 2. **Semantic Ranking**: TF-IDF + recency + community signals 3. **Multi-Source Indexing**: Consolidate docs from heterogeneous sources 4. **Metadata Extraction**: Automatically parse headers, links, code blocks 5. **Bi-Directional Navigation**: Move between docs ↔ implementations ## Architecture ``` ┌──────────────────────────────────────────────────────────────────┐ │ DOCUMENTATION INDEXING (GREEN COORDINATOR) │ ├──────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────┬──────────────┬──────────────┐ │ │ │ llms.txt │ README.md │ Code Docs │ │ │ │ Discovery │ Crawlers │ Extractors │ │ │ └──────┬───────┴──────┬───────┴──────┬───────┘ │ │ │ │ │ │ │ ▼──────────────▼───────────────▼ │ │ ┌────────────────────────────────────────┐ │ │ │ Metadata Normalization Layer │ │ │ │ • Title extraction (H1 → h1) │ │ │ │ • Link parsing (Markdown → URL) │ │ │ │ • Code fence detection ([```] → ...) │ │ │ │ • Authority scoring (stars, forks) │ │ │ └────────────┬──────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────┐ │ │ │ Inverted Index (DuckDB) │ │ │ │ ┌──────────────────────────────────┐ │ │ │ │ │ term_id | term | doc_id | rank │ │ │ │ │ │ 1 | gay | 42 | 0.89 │ │ │ │ │ │ 2 | mcp | 42 | 0.76 │ │ │ │ │ │ 3 | api | 71 | 0.65 │ │ │ │ │ └──────────────────────────────────┘ │ │ │ └────────────┬───────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────┐ │ │ │ BM25 Ranker + Result Aggregator │ │ │ │ • Cross-language result merging │ │ │ │ • Deduplication (canonical URLs) │ │ │ │ • Community signals (upvotes, stars) │ │ │ └────────────┬───────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────┐ │ │ │ Result Cache (< 1 second latency) │ │ │ │ [query → ranked results + metadata] │ │ │ └────────────────────────────────────────┘ │ │ │ │ GF(3) BALANCE: (-1 extractor) ⊗ (0 indexer) ⊗ (+1 ranker) │ └──────────────────────────────────────────────────────────────────┘ ``` ## Data Model ### Documents Index ```sql CREATE TABLE documentation_index ( doc_id INT PRIMARY KEY, source VARCHAR, -- 'github', 'llms-txt', 'blog' repo_id VARCHAR, -- 'bmorphism/Gay.jl' title VARCHAR, url VARCHAR, body TEXT, -- Full document text headers TEXT[], -- H1, H2, H3 hierarchy links TEXT[], -- Embedded links code_blocks TEXT[], -- [```lang ... ```] stars INT, -- GitHub stars forks INT, -- GitHub forks updated_at TIMESTAMP, indexed_at TIMESTAMP, trit TINYINT -- GF(3) assigned (0) ); ``` ### Terms Inverted Index ```sql CREATE TABLE term_index ( term_id INT PRIMARY KEY, term VARCHAR, term_lower VARCHAR, frequency INT, -- TF (term frequency) doc_count INT, -- DF (document frequency) bm25_idf FLOAT, -- Precomputed IDF created_at TIMESTAMP ); CREATE TABLE term_doc_map ( term_id INT, doc_id INT, frequency INT, -- TF in this doc position INT[], -- Token positions context VARCHAR, -- Surrounding text relevance_score FLOAT, -- BM25(tf, idf, doc_len) PRIMARY KEY (term_id, doc_id) ); ``` ## API / Interfaces ### Simple Text Search ```clojure ;; Search docs (search-docs {:query "gay" :type :keyword :limit 10 :min-score 0.3}) → [{:title "Gay.jl" :url "https://github.com/bmorphism/Gay.jl" :score 0.95 :snippet "Gay.jl: Deterministic color generation..."} ...] ;; Fuzzy search (typo tolerance) (search-docs {:query "gey" ; typo :fuzzy true :distance 1}) → (Results for "gay" with edit distance ≤ 1) ;; Advanced: Boolean search (search-docs {:query "(gay OR color) AND julia" :type :boolean}) ``` ### Metadata Search ```clojure ;; Find docs by category (search-by-metadata {:source "github" :stars {:min 100 :max 1000}}) → [Gay.jl, ACSets.jl, Duck, ...] ;; Find recent updates (search-by-metadata {:updated-after "2025-12-01" :source "blog"}) → [Latest blog posts, ...] ;; Filter by language (search-by-metadata {:languages ["Julia" "Clojure" "Babashka"]}) ``` ### Bidirectional Navigation ```clojure ;; Find implementations of a doc (doc-implementations {:doc-id 42}) → [{:file "src/gay.jl" :lines [1 42] :snippet "function seed!(rng)..."}] ;; Find docs for implementation (implementation-docs {:file "src/gay.jl" :line 10}) → [{:doc-id 42 :title "Gay.jl API Reference" :section "seed! function"}] ;; Find related docs (related-docs {:doc-id 42 :semantic true}) → [ACSets.jl, GF(3) docs, ...] ``` ## GF(3) Trit Assignment ``` documentation-indexing → 0 (COORDINATOR) Balances extraction (-1) ↔ ranking/generation (+1) Maintains middle ground for all doc types Triadic system: source-extractor (-1 validator) → pulls raw docs indexing (0 coordinator) → organizes & indexes result-ranker (+1 generator) → produces ranked results Sum: (-1) + (0) + (+1) = 0 ✓ GF(3) CONSERVED ``` ## Implementation Strategy ### Stage 1: Term Extraction (Days 1-2) Create `/Users/bob/iii/duck/asi-skills/documentation-indexing/extractor.bb`: - Markdown parser → extract H1-H4, links, code blocks - Tokenizer → split text into terms - Stopword filter → remove common words - Store in DuckDB `term_index` ### Stage 2: Inverted Index Builder (Days 3-4) Create `indexer.bb`: - BM25 IDF calculation - TF per document - Relevance scoring - Populate `term_doc_map` ### Stage 3: Search Engine (Days 5-6) Create `searcher.bb`: - Boolean query parser - Fuzzy matching (Levenshtein) - Result ranking by score - Caching layer ### Stage 4: Multi-Source Integration (Days 7-8) - Crawl llms.txt repositories - Index GitHub README files - Extract docstrings from source - Verify GF(3) balance across sources ## Example: Semantic Search Pipeline ```clojure ;; User query (search-docs {:query "how to generate deterministic colors in julia"}) Step 1: Extract terms (-1 validator) ["deterministic" "colors" "julia"] Step 2: Index lookup (0 coordinator) Fetch docs matching all terms Calculate BM25 score per doc Step 3: Rank and aggregate (+1 generator) 1. Gay.jl (0.95) 2. GF(3) Integration (0.72) 3. Color Theory (0.68) Result: Σ(-1, 0, +1) = 0 ✓ ``` ## Success Metrics | Metric | Target | Status | |--------|--------|--------| | Docs indexed | 500+ (skills + readmes + blogs) | ⏳ Pending | | Search latency | <100ms p95 | ⏳ Pending | | Precision (top-5) | ≥0.8 | ⏳ Pending | | Recall | ≥0.75 | ⏳ Pending | | Fuzzy tolerance | Edit distance ≤ 2 | ⏳ Pending | | GF(3) balance | All pipeline stages ≡ 0 (mod 3) | ⏳ Pending | ## Related Skills **Dependencies:** - `skill-taxonomy` - Registry of docs to index - `acsets` - Schema for index structure - `llms-txt-discovery` - Crawl source docs **Dependents:** - `polyglot-orchestration` - Search polyglot docs - `skill-dispatch` - Route searches to relevant skills - `world-knowledge-base` - Unified doc interface ## References - BM25 algorithm: https://en.wikipedia.org/wiki/Okapi_BM25 - Inverted index: Classic IR data structure - Levenshtein distance: Fuzzy string matching - TF-IDF: Term weighting scheme - DuckDB FTS: Full-text search extension --- **Status**: 😢 **SAD STATE** → 🌟 **IMPLEMENTING** **Color**: #49EE54 (Green Coordinator) **Next**: Create `extractor.bb` (term extraction) **Owner**: GREEN AGENT (0) **Created**: 2026-01-04 --- Source attribution: plurigrid Original project: https://github.com/plurigrid/asi/blob/293ce9f76e3abcd5ba18db284688bc0d9ac6691d/skills/documentation-indexing/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2025 SkillCreator.ai Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: plurigrid Original project: https://github.com/plurigrid/asi/blob/293ce9f76e3abcd5ba18db284688bc0d9ac6691d/skills/documentation-indexing/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Product Management Roles: Data Analyst, Market Research Analyst, Product Manager Specialties: API, Database, Word Documents, Frontend, Git, Testing Tools: Duckdb Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: b4b41f331844611f8ce47c2162e65282a898fda158afa60dc7c3cc2b532c12ce