--- name: voice-ai-engine-development-93d266bd849f description: Architecting real-time Voice AI agents. category: development version: 4.1.0-fractal layer: master-skill --- # Voice AI Engine Development > **Goal**: Build low-latency, conversational Voice AI agents capable of full-duplex communication. ## 1. The Voice Pipeline (Latency is King) The total loop Latency (Voice-to-Ear) should be < 1000ms (Ideal < 500ms). 1. **Transport**: WebRTC (preferred for browser) or WebSocket (server-server). 2. **VAD (Voice Activity Detection)**: Detect when user starts/stops speaking. - *Tools*: Silero VAD, WebRTC VAD. 3. **STT (Speech-to-Text)**: Transcribe audio to text. - *Tools*: Deepgram (fastest), Whisper (high accuracy but slower), AssemblyAI. 4. **LLM (Brain)**: Process text and generate response. - *Tools*: Groq (Llama 3), GPT-4o, Claude 3.5 Sonnet. 5. **TTS (Text-to-Speech)**: Convert response to audio. - *Tools*: ElevenLabs (Quality), Cartesia (Speed), OpenAI TTS. ## 2. Architecture Patterns - **Streaming Pipeline**: DO NOT wait for full transcription or full generation. Stream everything. - User Audio Stream -> VAD -> STT Stream -> LLM Stream -> TTS Stream -> Audio Output. - **Interruption Handling (Barge-in)**: - If VAD detects user speech while AI is talking -> Immediately CUT text generation and audio playback. Clear buffers. ## 3. Implementation Stack - **Backend**: Python (FastAPI) or Node.js. Python ecosystem is stronger for audio processing (numpy/scipy). - **Frameworks**: - **Pipecat**: Open source framework for building voice agents. - **LiveKit**: WebRTC infrastructure for real-time audio/video. - **Twilio**: For telephony integration. ## 4. Optimization Techniques - **Optimistic VAD**: Tune VAD to be sensitive to start, but careful with "silence" timeout (usually 500ms-800ms) to detect end of turn. - **Prompt Engineering**: Instruct LLM to be concise and conversational. - *System Prompt*: "You are a helpful voice assistant. Keep responses short (1-2 sentences). Do not use markdown or emojis." - **Audio Formats**: Use OPUS or PCM (16khz/24khz/48khz) for transmission. Avoid MP3 transcoding latency. ## 5. Debugging & Metrics - **WER (Word Error Rate)**: For STT accuracy. - **TTFT (Time to First Token)**: LLM speed. - **TTA (Time to Audio)**: The critical metric. Time from user silence to first AI sound. --- **Common Pitfalls**: - Echo cancellation issues (User hears themselves). Use WebRTC's built-in AEC. - Hallucination in STT (Whisper transcribing silence). - Race conditions during interruptions. --- Source attribution: Dokhacgiakhoa Original project: https://github.com/Dokhacgiakhoa/antigravity-ide/blob/c4a059bbf7a8d6b2162071f0fbfe74053faa6d20/.agent/skills/voice-ai-engine-development/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2026 Dokhacgiakhoa Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================================================ UNIVERSAL ACKNOWLEDGEMENTS & REFERENCES ================================================================================ This project is authored and maintained by Dokhacgiakhoa. Antigravity IDE (Enhanced Edition) draws significant inspiration and inherits open-source knowledge from the following masters and repositories: 1. Antigravity Kit (Core Reference) - Original Author: vudovn - Link: https://github.com/vudovn/antigravity-kit 2. Antigravity Awesome Skills (Skill Metadata Reference) - Original Author: sickn33 - Link: https://github.com/sickn33/antigravity-awesome-skills 3. UI/UX Pro Max (Design Philosophy Reference) - Original Author: sickn33 - Link: https://github.com/sickn33/ui-ux-pro-max 4. Model Context Protocol (MCP) standards - Authors: Anthropic & Ecosystem Contributors - Link: https://modelcontextprotocol.io/ 5. High-Performance Web Standards - Inspiration: Vercel Engineering & Anthony Fu (antfu) 6. Strategic Knowledge Extraction - Inspiration: Fabric Patterns (danielmiessler) *Note: All trademarks and copyrights belong to their respective owners. The references above reflect research-level inspiration and antecedent contributions. These parties are not involved in the management of this repository.* # Mega AI Hub resource metadata Creator / publisher: Dokhacgiakhoa Original project: https://github.com/Dokhacgiakhoa/antigravity-ide/blob/c4a059bbf7a8d6b2162071f0fbfe74053faa6d20/.agent/skills/voice-ai-engine-development/SKILL.md Attribution basis: Source repository record Fields of work: Development & Engineering, Product Management Roles: Backend Developer, DevOps / SRE, ML / AI Engineer, Product Manager, Technical Product Manager Specialties: API, Backend Tools: ElevenLabs, Groq, OpenAI API, Twilio, Whisper Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: bcf5da5288338a297dbc0114e050ff6f7d84572c43cebdb19cf77f0d232910b1