--- name: senior-data-engineer-59006c81d5a2 description: World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, or implementing data governance. --- # Senior Data Engineer World-class senior data engineer skill for production-grade AI/ML/Data systems. ## Quick Start ### Main Capabilities ```bash # Core Tool 1 python scripts/pipeline_orchestrator.py --input data/ --output results/ # Core Tool 2 python scripts/data_quality_validator.py --target project/ --analyze # Core Tool 3 python scripts/etl_performance_optimizer.py --config config.yaml --deploy ``` ## Core Expertise This skill covers world-class capabilities in: - Advanced production patterns and architectures - Scalable system design and implementation - Performance optimization at scale - MLOps and DataOps best practices - Real-time processing and inference - Distributed computing frameworks - Model deployment and monitoring - Security and compliance - Cost optimization - Team leadership and mentoring ## Tech Stack **Languages:** Python, SQL, R, Scala, Go **ML Frameworks:** PyTorch, TensorFlow, Scikit-learn, XGBoost **Data Tools:** Spark, Airflow, dbt, Kafka, Databricks **LLM Frameworks:** LangChain, LlamaIndex, DSPy **Deployment:** Docker, Kubernetes, AWS/GCP/Azure **Monitoring:** MLflow, Weights & Biases, Prometheus **Databases:** PostgreSQL, BigQuery, Snowflake, Pinecone ## Reference Documentation ### 1. Data Pipeline Architecture Comprehensive guide available in `references/data_pipeline_architecture.md` covering: - Advanced patterns and best practices - Production implementation strategies - Performance optimization techniques - Scalability considerations - Security and compliance - Real-world case studies ### 2. Data Modeling Patterns Complete workflow documentation in `references/data_modeling_patterns.md` including: - Step-by-step processes - Architecture design patterns - Tool integration guides - Performance tuning strategies - Troubleshooting procedures ### 3. Dataops Best Practices Technical reference guide in `references/dataops_best_practices.md` with: - System design principles - Implementation examples - Configuration best practices - Deployment strategies - Monitoring and observability ## Production Patterns ### Pattern 1: Scalable Data Processing Enterprise-scale data processing with distributed computing: - Horizontal scaling architecture - Fault-tolerant design - Real-time and batch processing - Data quality validation - Performance monitoring ### Pattern 2: ML Model Deployment Production ML system with high availability: - Model serving with low latency - A/B testing infrastructure - Feature store integration - Model monitoring and drift detection - Automated retraining pipelines ### Pattern 3: Real-Time Inference High-throughput inference system: - Batching and caching strategies - Load balancing - Auto-scaling - Latency optimization - Cost optimization ## Best Practices ### Development - Test-driven development - Code reviews and pair programming - Documentation as code - Version control everything - Continuous integration ### Production - Monitor everything critical - Automate deployments - Feature flags for releases - Canary deployments - Comprehensive logging ### Team Leadership - Mentor junior engineers - Drive technical decisions - Establish coding standards - Foster learning culture - Cross-functional collaboration ## Performance Targets **Latency:** - P50: < 50ms - P95: < 100ms - P99: < 200ms **Throughput:** - Requests/second: > 1000 - Concurrent users: > 10,000 **Availability:** - Uptime: 99.9% - Error rate: < 0.1% ## Security & Compliance - Authentication & authorization - Data encryption (at rest & in transit) - PII handling and anonymization - GDPR/CCPA compliance - Regular security audits - Vulnerability management ## Common Commands ```bash # Development python -m pytest tests/ -v --cov python -m black src/ python -m pylint src/ # Training python scripts/train.py --config prod.yaml python scripts/evaluate.py --model best.pth # Deployment docker build -t service:v1 . kubectl apply -f k8s/ helm upgrade service ./charts/ # Monitoring kubectl logs -f deployment/service python scripts/health_check.py ``` ## Resources - Advanced Patterns: `references/data_pipeline_architecture.md` - Implementation Guide: `references/data_modeling_patterns.md` - Technical Reference: `references/dataops_best_practices.md` - Automation Scripts: `scripts/` directory ## Senior-Level Responsibilities As a world-class senior professional: 1. **Technical Leadership** - Drive architectural decisions - Mentor team members - Establish best practices - Ensure code quality 2. **Strategic Thinking** - Align with business goals - Evaluate trade-offs - Plan for scale - Manage technical debt 3. **Collaboration** - Work across teams - Communicate effectively - Build consensus - Share knowledge 4. **Innovation** - Stay current with research - Experiment with new approaches - Contribute to community - Drive continuous improvement 5. **Production Excellence** - Ensure high availability - Monitor proactively - Optimize performance - Respond to incidents --- Source attribution: Dicklesworthstone Original project: https://github.com/Dicklesworthstone/pi_agent_rust/blob/f28fc46ee0db8e710d95b8b55592cd38d60f2b26/tests/ext_conformance/artifacts/templates-davila7/cli-tool/components/skills/development/senior-data-engineer/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License (with OpenAI/Anthropic Rider) Copyright (c) 2026 Jeffrey Emanuel Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: ADDITIONAL RIDER / RESTRICTION (OpenAI / Anthropic): This rider is part of the "conditions" of this License. In the event of any conflict between this rider and any other portion of this License, this rider controls. "Restricted Parties" means OpenAI, L.L.C.; Anthropic, PBC; any of their respective Affiliates; and any person or entity acting directly or indirectly on behalf of, for the benefit of, or under the direction of any of the foregoing (including any officer, director, employee, contractor, agent, consultant, service provider, or representative). Notwithstanding any other provision of this License, no rights are granted to any Restricted Party. Any purported license, sublicense, assignment, transfer, or other permission to any Restricted Party is null and void absent the express prior written permission of Jeffrey Emanuel. You may not provide, disclose, distribute, sublicense, sell, lease, lend, host, make available, or otherwise permit access to the Software or any derivative work of the Software (as defined in applicable copyright law) (collectively, "Derivative Works") to or for any Restricted Party. For purposes of this rider, "use" includes, without limitation: copying, modifying, merging, publishing, distributing, sublicensing, selling, transferring, making available, hosting, deploying, executing, benchmarking, testing, analyzing, indexing, or incorporating the Software or any Derivative Works into any dataset, training corpus, evaluation harness, or pipeline for machine learning or other automated systems. This rider applies to the Software and all Derivative Works. As a condition of use, you agree that this rider is a precondition to exercising any rights under this License, and you agree that any distribution of the Software or any Derivative Works must include this rider provision unmodified. Any breach of this rider automatically and immediately terminates the permissions granted by this License. Upon termination, you must immediately cease all use and distribution of the Software and any Derivative Works and destroy all copies under your control. You agree that a breach of this rider would cause irreparable harm and that Jeffrey Emanuel may seek injunctive or other equitable relief to enforce this rider, in addition to any other remedies available at law. To the maximum extent permitted by applicable law, the prevailing party in any action to enforce this rider shall be entitled to recover reasonable attorneys' fees and costs. For purposes of this rider, "Affiliate" means any entity that directly or indirectly controls, is controlled by, or is under common control with the specified party. "Control" means ownership of more than 50% of the voting securities or other ownership interest, or the power to direct management or policies by contract or otherwise. The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: Dicklesworthstone Original project: https://github.com/Dicklesworthstone/pi_agent_rust/blob/f28fc46ee0db8e710d95b8b55592cd38d60f2b26/tests/ext_conformance/artifacts/templates-davila7/cli-tool/components/skills/development/senior-data-engineer/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Development & Engineering, Operations, Product Management Roles: Business Process Analyst, Data Analyst, Data Engineer, ML / AI Engineer, Product Manager Specialties: API, Backend, Database Tools: AWS, Azure, Docker, GCP, Kubernetes, Pinecone, PostgreSQL Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 768bc45f2d8b9a777cb92f1a044cffdc27036a2bc29a443dcadea07da2e0ab24