---
name: web-research-d931d2670b1c
description: "Use when extracting article text from news URLs or downloading video or audio with metadata"
allowed-tools: [Bash(python3*), Bash(yt-dlp*), Read]
version: 1.0.0
author: ykotik
license: MIT
---

# Web Research

## When to Use
- Extracting clean article text and metadata from a news/blog URL
- Downloading video or audio from supported sites with full metadata
- Getting video/audio metadata without downloading the media

## Tools

| Tool | Purpose | Structured output |
|------|---------|-------------------|
| **newspaper4k** | Extract article text, title, authors, date from URLs | Python object (access `.title`, `.text`, `.authors`) |
| **yt-dlp** | Download video/audio from 1000+ sites | `--dump-json` for metadata JSON |

## Patterns

### Extract article text from a URL
```bash
python3 -c "
from newspaper import Article
a = Article('https://example.com/news/article')
a.download()
a.parse()
import json
print(json.dumps({'title': a.title, 'authors': a.authors, 'date': str(a.publish_date), 'text': a.text[:2000]}, indent=2))
"
```

### Extract multiple articles
```bash
for url in "https://example.com/article1" "https://example.com/article2"; do
  python3 -c "
from newspaper import Article
import json
a = Article('$url')
a.download()
a.parse()
print(json.dumps({'url': '$url', 'title': a.title, 'text': a.text[:1000]}))
"
done
```

### Get video metadata without downloading
```bash
yt-dlp --dump-json "https://www.youtube.com/watch?v=VIDEO_ID" | jq '{title, duration, view_count, upload_date}'
```

### Download audio only (best quality)
```bash
yt-dlp -x --audio-format mp3 "https://www.youtube.com/watch?v=VIDEO_ID"
```

### Download video (best quality, specific format)
```bash
yt-dlp -f "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]" "https://www.youtube.com/watch?v=VIDEO_ID"
```

### Download with metadata and subtitles
```bash
yt-dlp --write-info-json --write-subs --sub-langs en "https://www.youtube.com/watch?v=VIDEO_ID"
```

### List available formats for a video
```bash
yt-dlp -F "https://www.youtube.com/watch?v=VIDEO_ID"
```

## Pipelines

### Get video metadata → query with DuckDB
```bash
yt-dlp --dump-json "https://www.youtube.com/@channel/videos" --flat-playlist | head -20 > videos.jsonl
duckdb -c "SELECT title, view_count, duration FROM read_json_auto('videos.jsonl') ORDER BY view_count DESC LIMIT 10"
```
Each stage: yt-dlp dumps playlist metadata as JSONL, DuckDB queries for top videos by views.

## Prefer Over
- Prefer **newspaper4k** over curl + HTML parsing for article extraction — handles boilerplate removal, metadata extraction automatically
- Prefer **yt-dlp** over browser downloads — supports 1000+ sites, can extract metadata without downloading

## Do NOT Use When
- Need to interact with a web page (click, scroll, fill forms) — use web-crawling skill (Playwright)
- Real-time search with the WebSearch tool available — prefer the built-in tool for conversational search
- Downloading copyrighted content without authorization
- Target site requires authentication — use web-crawling skill with login automation


---
Source attribution: jeremylongshore
Original project: https://github.com/jeremylongshore/claude-code-plugins-plus-skills/blob/ab85a061f7bcc039dac13ee193de7a4cd4388571/plugins/productivity/cli-power-skills/skills/web-research/SKILL.md
License: MIT
License information: https://opensource.org/license/mit
Original upstream instructions; downloaded from Mega AI Hub.

## Upstream license notice

MIT License

Copyright (c) 2026 Yuriy Kotik

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.



# Mega AI Hub resource metadata

Creator / publisher: jeremylongshore
Original project: https://github.com/jeremylongshore/claude-code-plugins-plus-skills/blob/ab85a061f7bcc039dac13ee193de7a4cd4388571/plugins/productivity/cli-power-skills/skills/web-research/SKILL.md
Attribution basis: Source repository record

Fields of work: Data & Analytics, Education & Research, Marketing
Roles: Content Marketer, Data Analyst, Researcher
Specialties: Backend, Frontend, Playwright, Remotion
Tools: Newspaper4K, Yt Dlp
Country applicability: Country not specified
Classification basis: Source catalog labels
Instruction SHA-256: 601a86748fdad5c3c2e2d593c939df3c40f02da84bb17f96b6e0373cb1b65e57