--- name: finviz-crawler-6168d8f45712 description: Continuous financial news crawler for finviz.com with SQLite storage, article extraction, and query tool. Use when monitoring financial markets, building news digests, or needing a local financial news database. Runs as a background daemon or systemd service. metadata: {"openclaw":{"requires":{"bins":["python3"]}}} --- # finviz-crawler ## Why This Skill? 📰 **Your own financial news database** — most finance skills just wrap an API for one-shot queries. This skill runs continuously, building a local archive of every headline and article from Finviz. Query your history anytime — no API limits, no missing data. 🆓 **No API key, no subscription** — scrapes finviz.com directly using Crawl4AI + RSS. Bloomberg, Reuters, Yahoo Finance, CNBC articles extracted automatically. Zero cost. 🤖 **Built for AI summarization** — the query tool outputs clean text/JSON optimized for LLM digests. Pair with an OpenClaw cron job for automated morning briefings, evening wrap-ups, or weekly investment summaries. 💾 **Auto-cleanup** — configurable expiry automatically deletes old articles from both the database and disk. Set `--expiry-days 30` to keep a month of history, or `0` to keep everything forever. 🔄 **Daemon architecture** — runs as a background service that starts/stops with OpenClaw. No manual intervention needed after setup. Works with systemd (Linux) and launchd (macOS). ## Install ```bash python3 scripts/install.py ``` Works on **macOS, Linux, and Windows**. Installs Python packages (`crawl4ai`, `feedparser`), sets up Playwright browsers, creates data directories, and verifies everything. ### Manual install ```bash pip install crawl4ai feedparser crawl4ai-setup # or: python -m playwright install chromium ``` ## Usage ### Run the crawler ```bash # Default: ~/workspace/finviz/, 7-day expiry python3 scripts/finviz_crawler.py # Custom paths and settings python3 scripts/finviz_crawler.py --db /path/to/finviz.db --articles-dir /path/to/articles/ # Keep 30 days of articles python3 scripts/finviz_crawler.py --expiry-days 30 # Never auto-delete (keep everything) python3 scripts/finviz_crawler.py --expiry-days 0 # Custom crawl interval (default: 300s) python3 scripts/finviz_crawler.py --sleep 600 ``` ### Query articles ```bash # Last 24 hours of headlines python3 scripts/finviz_query.py --hours 24 # Titles only (compact, good for LLM summarization) python3 scripts/finviz_query.py --hours 12 --titles-only # With full article content python3 scripts/finviz_query.py --hours 12 --with-content # List downloaded articles with content status python3 scripts/finviz_query.py --list-articles --hours 24 # Database stats python3 scripts/finviz_query.py --stats ``` ### Manage tickers ```bash # List all tracked tickers python3 scripts/finviz_query.py --list-tickers # Add single ticker (auto-generates keywords from symbol) python3 scripts/finviz_query.py --add-ticker NVDA # Add with custom keywords python3 scripts/finviz_query.py --add-ticker "NVDA:nvidia,jensen huang" # Add multiple tickers (batch) python3 scripts/finviz_query.py --add-ticker NVDA TSLA AAPL python3 scripts/finviz_query.py --add-ticker "NVDA:nvidia,jensen" "TSLA:tesla,elon musk" # Remove tickers (batch) python3 scripts/finviz_query.py --remove-ticker NVDA TSLA # Custom DB path python3 scripts/finviz_query.py --list-tickers --db /path/to/finviz.db ``` Tickers are stored in the `tickers` table inside `finviz.db` alongside articles. The crawler reads this table each cycle to know which ticker pages to scrape. ### Configuration | Setting | CLI flag | Env var | Default | |---------|----------|---------|---------| | Database path | `--db` | — | `~/workspace/finviz/finviz.db` | | Articles directory | `--articles-dir` | — | `~/workspace/finviz/articles/` | | Crawl interval | `--sleep` | — | `300` (5 min) | | Article expiry | `--expiry-days` | `FINVIZ_EXPIRY_DAYS` | `7` days | | Timezone | — | `FINVIZ_TZ` or `TZ` | System default | ## 💬 Chat Commands (OpenClaw Agent) When this skill is installed, the agent recognizes `/finviz` as a shortcut: | Command | Action | |---------|--------| | `/finviz list` | Show tracked tickers | | `/finviz add NVDA, TSLA` | Add tickers to track | | `/finviz remove NVDA` | Remove a ticker | | `/finviz stats` | Show article/ticker counts | | `/finviz help` | Show available commands | The agent runs these via the `finviz_query.py` CLI internally. ## 📱 PrivateApp Dashboard A companion mobile dashboard is available in [PrivateApp](https://github.com/camopel/PrivateApp) — a personal PWA dashboard for your home server. The **Finviz** app provides: - Headlines browser with time-range filters (12h / 24h / Week) - Ticker-specific news filtering - LLM-powered summaries on demand Install PrivateApp, and the Finviz dashboard is built-in — no extra setup needed. ## Architecture **Crawler daemon** (`finviz_crawler.py`): - Crawls finviz.com/news.ashx headlines every 5 minutes - Fetches article content via Crawl4AI (Playwright) or RSS (paywalled sites) - Bot/paywall detection rejects garbage content - Per-domain rate limiting, user-agent rotation - Deduplicates via SHA-256 title hash - Auto-expires old articles (configurable) - Clean shutdown on SIGTERM/SIGINT **Query tool** (`finviz_query.py`): - Read-only SQLite queries (no HTTP, stdlib only) - Filter by time window, export titles or full content - Designed for LLM summarization pipelines ## Run as a service (optional) ### systemd (Linux) ```ini [Unit] Description=Finviz News Crawler [Service] ExecStart=python3 /path/to/scripts/finviz_crawler.py --expiry-days 30 Restart=on-failure RestartSec=30 [Install] WantedBy=default.target ``` ### launchd (macOS) ```xml Labelcom.finviz.crawler ProgramArguments python3 /path/to/scripts/finviz_crawler.py --expiry-days 30 RunAtLoad KeepAlive ``` ## Data layout ``` ~/workspace/finviz/ ├── finviz.db # SQLite: articles + tickers (single DB) ├── articles/ # Full article content as .md files │ ├── market/ # General market headlines │ ├── nvda/ # Per-ticker articles │ └── tsla/ └── summaries/ # LLM summary cache (.json) ``` ## Cron integration Pair with an OpenClaw cron job for automated digests: ``` Schedule: 0 6 * * * (6 AM daily) Task: Query last 24h → LLM summarize → deliver to Matrix/Telegram/Discord ``` --- Source attribution: modbender Original project: https://github.com/modbender/skill-library-mcp/blob/d9c8fe903b5b4e82237dbe8d0df8567783022e18/data/finviz-crawler/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2025 skill-library-mcp contributors Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: modbender Original project: https://github.com/modbender/skill-library-mcp/blob/d9c8fe903b5b4e82237dbe8d0df8567783022e18/data/finviz-crawler/SKILL.md Attribution basis: Source repository record Fields of work: Marketing Roles: Growth Marketer, Marketing Analyst, Product Marketer Specialties: API, Backend Tools: Crawl4Ai, Feedparser, Python3 Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 96c65599a211925b8a0aa4850c6674495a99548dfbdce3e732399d850fc498fa