--- name: openclaw-test-heap-leaks-60ad45d5c0af description: Investigate `pnpm test` memory growth, Vitest worker OOMs, and suspicious RSS increases in OpenClaw using the `scripts/test-parallel.mjs` heap snapshot tooling. Use when Codex needs to reproduce test-lane memory growth, collect repeated `.heapsnapshot` files, compare snapshots from the same worker PID, distinguish transformed-module retention from real data leaks, and fix or reduce the impact by patching cleanup logic or isolating hotspot tests. --- # OpenClaw Test Heap Leaks Use this skill for test-memory investigations. Do not guess from RSS alone when heap snapshots are available. ## Workflow 1. Reproduce the failing shape first. - Match the real entrypoint if possible. For Linux CI-style unit failures, start with: - `pnpm canvas:a2ui:bundle && OPENCLAW_TEST_MEMORY_TRACE=1 OPENCLAW_TEST_HEAPSNAPSHOT_INTERVAL_MS=60000 OPENCLAW_TEST_HEAPSNAPSHOT_DIR=.tmp/heapsnap OPENCLAW_TEST_WORKERS=2 OPENCLAW_TEST_MAX_OLD_SPACE_SIZE_MB=6144 pnpm test` - Keep `OPENCLAW_TEST_MEMORY_TRACE=1` enabled so the wrapper prints per-file RSS summaries alongside the snapshots. - If the report is about a specific shard or worker budget, preserve that shape. 2. Wait for repeated snapshots before concluding anything. - Take at least two intervals from the same lane. - Compare snapshots from the same PID inside one lane directory such as `.tmp/heapsnap/unit-fast/`. - Use `scripts/heapsnapshot-delta.mjs` to compare either two files directly or the earliest/latest pair per PID in one lane directory. 3. Classify the growth before choosing a fix. - If growth is dominated by Vite/Vitest transformed source strings, `Module`, `system / Context`, bytecode, descriptor arrays, or property maps, treat it as retained module graph growth in long-lived workers. - If growth is dominated by app objects, caches, buffers, server handles, timers, mock state, sqlite state, or similar runtime objects, treat it as a likely cleanup or lifecycle leak. 4. Fix the right layer. - For retained transformed-module growth in shared workers: - Move hotspot files out of `unit-fast` by updating `test/fixtures/test-parallel.behavior.json`. - Prefer `singletonIsolated` for files that are safe alone but inflate shared worker heaps. - If the file should already have been peeled out by timings but is absent from `test/fixtures/test-timings.unit.json`, call that out explicitly. Missing timings are a scheduling blind spot. - For real leaks: - Patch the implicated test or runtime cleanup path. - Look for missing `afterEach`/`afterAll`, module-reset gaps, retained global state, unreleased DB handles, or listeners/timers that survive the file. 5. Verify with the most direct proof. - Re-run the targeted lane or file with heap snapshots enabled if the suite still finishes in reasonable time. - If snapshot overhead pushes tests over Vitest timeouts, fall back to the same lane without snapshots and confirm the RSS trend or OOM is reduced. - For wrapper-only changes, at minimum verify the expected lanes start and the snapshot files are written. ## Heuristics - Do not call everything a leak. In this repo, large `unit-fast` growth can be a worker-lifetime problem rather than an application object leak. - `scripts/test-parallel.mjs` and `scripts/test-parallel-memory.mjs` are the primary control points for wrapper diagnostics. - The lane names printed by `[test-parallel] start ...` and `[test-parallel][mem] summary ...` tell you where to focus. - When one or two files account for most of the delta and they are missing from timings, reducing impact by isolating them is usually the first pragmatic fix. - When the same retained object families grow across multiple intervals in the same worker PID, trust the snapshots over intuition. ## Snapshot Comparison - Direct comparison: - `node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs before.heapsnapshot after.heapsnapshot` - Auto-select earliest/latest snapshots per PID within one lane: - `node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs --lane-dir .tmp/heapsnap/unit-fast` - Useful flags: - `--top 40` - `--min-kb 32` - `--pid 16133` Read the top positive deltas first. Large positive growth in module-transform artifacts suggests lane isolation; large positive growth in runtime objects suggests a real leak. ## Output Expectations When using this skill, report: - The exact reproduce command. - Which lane and PID were compared. - The dominant retained object families from the snapshot delta. - Whether the issue is a real leak or shared-worker retained module growth. - The concrete fix or impact-reduction patch. - What you verified, and what snapshot overhead prevented you from verifying. --- Source attribution: eszxcvfd Original project: https://github.com/eszxcvfd/openclaw-team/blob/2c098f9d857dcb3eabfeed97371492fb89121d7a/openclaw/.agents/skills/openclaw-test-heap-leaks/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2025 Peter Steinberger Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: eszxcvfd Original project: https://github.com/eszxcvfd/openclaw-team/blob/2c098f9d857dcb3eabfeed97371492fb89121d7a/openclaw/.agents/skills/openclaw-test-heap-leaks/SKILL.md Attribution basis: Source repository record Fields of work: Development & Engineering Roles: Backend Developer, DevOps / SRE, QA / Test Engineer Specialties: DevOps, Testing Tools: Pnpm, Vitest Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 2b4d91dc3078b504d7b9fb1d105bc33de553625ef20cca5ce8e382df007165b8