--- name: 'radiomics-pathomics-fusion-agent' description: 'AI-powered multimodal fusion of radiology (CT/MRI/PET) and pathology (H&E/IHC) imaging with clinical and genomic data for comprehensive cancer diagnostics and treatment prediction.' measurable_outcome: Execute skill workflow successfully with valid output within 15 minutes. allowed-tools: - read_file - run_shell_command --- # Radiomics Pathomics Fusion Agent The **Radiomics Pathomics Fusion Agent** integrates multimodal medical imaging data from radiology (CT, MRI, PET) and digital pathology (H&E, IHC whole slide images) with clinical and genomic data using deep learning fusion architectures. It enables comprehensive cancer phenotyping, treatment response prediction, and prognostic modeling. ## When to Use This Skill * When predicting treatment response using multimodal imaging. * For comprehensive tumor phenotyping combining macro and micro views. * To identify imaging biomarkers correlated with genomic features. * When building prognostic models from combined radiology-pathology. * For AI-powered second opinion integrating all imaging modalities. ## Core Capabilities 1. **Cross-Modal Fusion**: Integrate radiology and pathology features using attention. 2. **Radiomics Extraction**: Compute 3D texture, shape, intensity features from CT/MRI. 3. **Pathomics Extraction**: Extract histopathological features from WSI. 4. **Clinical Integration**: Combine imaging with clinical variables and genomics. 5. **Treatment Response Prediction**: Predict chemotherapy, immunotherapy response. 6. **Survival Prediction**: Multi-modal prognostic modeling. ## Supported Imaging Modalities | Modality | Features Extracted | Resolution | |----------|-------------------|------------| | CT | Texture, shape, density | Volumetric 3D | | MRI | Multi-sequence, perfusion | Volumetric 3D | | PET | SUV, metabolic features | Volumetric 3D | | H&E WSI | Nuclear, tissue architecture | 40x magnification | | IHC WSI | Marker quantification | 20-40x | | Multiplexed IF | Spatial protein patterns | Subcellular | ## Fusion Architectures | Architecture | Method | Strengths | |--------------|--------|-----------| | Early Fusion | Concatenate features | Simple, baseline | | Late Fusion | Combine predictions | Modular | | Attention Fusion | Cross-modal attention | Interpretable | | Multimodal Transformer | Self-attention across modalities | State-of-art | | Graph Fusion | GNN for relationships | Spatial awareness | ## Workflow 1. **Input**: CT/MRI DICOM, pathology WSI, clinical data, optional genomics. 2. **Segmentation**: Tumor ROI extraction from radiology. 3. **Radiomics**: Extract 3D radiomic features. 4. **Pathomics**: Extract histopathology features via foundation models. 5. **Fusion**: Multimodal feature integration. 6. **Prediction**: Treatment response, survival, biomarker prediction. 7. **Output**: Integrated predictions, attention maps, explanations. ## Example Usage **User**: "Predict immunotherapy response for this lung cancer patient using their CT scan and biopsy pathology." **Agent Action**: ```bash python3 Skills/Oncology/Radiomics_Pathomics_Fusion_Agent/fusion_predict.py \ --ct_dicom ct_scan/ \ --wsi_path biopsy.svs \ --clinical_data patient_clinical.json \ --genomic_data tumor_wes.vcf \ --task immunotherapy_response \ --cancer_type nsclc \ --fusion_method attention \ --output fusion_prediction/ ``` ## Radiomic Feature Categories | Category | Features | Count | |----------|----------|-------| | Shape | Volume, surface area, sphericity | 14 | | First-Order | Mean, variance, skewness, entropy | 18 | | GLCM | Contrast, correlation, homogeneity | 24 | | GLRLM | Run length, gray level emphasis | 16 | | GLSZM | Zone size, gray level variance | 16 | | GLDM | Dependence features | 14 | | NGTDM | Texture features | 5 | | **Total** | | **~107** | ## Pathomics Feature Categories | Category | Source | Features | |----------|--------|----------| | Nuclear | Segmentation | Size, shape, texture | | Cellular | Detection | Density, clustering | | Tissue | Architecture | Glandular, stromal ratios | | Foundation Model | CONCH, TITAN, UNI | Deep embeddings | | Spatial | Graph analysis | Neighborhood patterns | ## Output Components | Output | Description | Format | |--------|-------------|--------| | Prediction | Response/outcome probability | .json | | Confidence | Prediction uncertainty | .json | | Attention Maps | Cross-modal importance | .npy, .png | | Feature Importance | Shapley values | .csv | | ROI Highlights | Predictive regions | DICOM-SEG, GeoJSON | | Report | Clinical summary | .pdf | ## Clinical Applications | Application | Modalities Used | Performance | |-------------|-----------------|-------------| | NSCLC Immunotherapy | CT + H&E | AUC 0.82-0.88 | | HCC Survival | MRI + H&E | C-index 0.78 | | Breast Neoadjuvant | MRI + H&E | AUC 0.85 | | HNSCC HPV/Response | CT + H&E | AUC 0.89 | | CRC MSI Prediction | CT + H&E | AUC 0.86 | ## AI/ML Components **Radiomics Pipeline**: - PyRadiomics for feature extraction - 3D-CNN for learned features - Transformer for volumetric analysis **Pathomics Pipeline**: - Foundation models (CONCH, UNI, TITAN) - MIL (Multiple Instance Learning) for WSI - Graph networks for spatial patterns **Fusion Models**: - Cross-attention transformers - Multimodal variational autoencoders - Contrastive learning for alignment ## Prerequisites * Python 3.10+ * PyRadiomics, SimpleITK * OpenSlide, HistoEncoder * PyTorch, transformers * CONCH/TITAN model weights * GPU with 16GB+ VRAM ## Related Skills * Pathology_AI/CONCH_Agent - Pathology foundation model * Radiology_AI agents - Modality-specific analysis * Pan_Cancer_MultiOmics_Agent - Genomic integration * TMB_Estimation_Agent - Tumor mutational burden ## Multimodal Integration Strategies | Strategy | Description | Use Case | |----------|-------------|----------| | Feature-Level | Combine extracted features | Limited data | | Embedding-Level | Fuse latent representations | Moderate data | | Decision-Level | Ensemble predictions | Interpretability | | End-to-End | Joint training | Large data | ## Special Considerations 1. **Data Alignment**: Ensure imaging from same timepoint 2. **Missing Modalities**: Handle incomplete multimodal data 3. **Class Imbalance**: Balance training across outcomes 4. **Interpretability**: Attention maps for clinical trust 5. **Validation**: External multi-site validation essential ## Quality Control | QC Check | Threshold | Action | |----------|-----------|--------| | CT coverage | >90% tumor | Rescan if needed | | WSI quality | Blur score 0.85 | Manual review | | Feature stability | ICC >0.8 | Robust features only | ## Regulatory Considerations | Aspect | Status | |--------|--------| | FDA Clearance | Individual modality tools cleared | | Multimodal Fusion | Research use only (RUO) | | Clinical Integration | PACS/LIS integration pathways | | Explainability | Required for clinical adoption | ## Author AI Group - Biomedical AI Platform --- Source attribution: FreedomIntelligence Original project: https://github.com/FreedomIntelligence/OpenClaw-Medical-Skills License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2026 FridrichMethod Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. --- NOTE ON BUNDLED CONTENT This repository aggregates skills from multiple upstream projects (see the README for the source list). Each `skills//` directory may carry its own license inherited from its upstream source. The MIT license above applies to this repository's curation and the scripts/workflows in `/`, `/scripts/`, and `/.github/` — not to the upstream skill content itself. Consult the upstream repository linked in the README and any LICENSE file inside a skill directory before reusing skill content commercially. # Mega AI Hub resource metadata Creator / publisher: FreedomIntelligence Original project: https://github.com/FreedomIntelligence/OpenClaw-Medical-Skills Attribution basis: Exact package in biomedical source index and publisher source table Fields of work: Data & Analytics, Design, Healthcare & Life Sciences Roles: Clinical Research, Data Scientist, Health Informatics, Medical Writer, UX Designer Specialties: API, Testing Tools: Read_File, Run_Shell_Command Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: ee5307cc79a2024392acd8b773e73309a43ef935d4c2381776057d0c0537efa1 Metadata reviewed: 2026-10-27 This metadata supersedes earlier Mega AI Hub catalog attribution. Original instruction and supporting files are preserved.