--- name: linux-perf-profile-c375c5a98f6c description: Deep Linux perf profiling — PMU counters, topdown analysis, flamegraphs, and annotated hotspot drill-down on ARM/Graviton --- # Linux Perf Profiling Deep hardware-level performance analysis using Linux `perf` on this ARM (Neoverse V1 / Graviton3) system. Goes beyond Criterion benchmarks to explain *why* code is slow using PMU counters, topdown decomposition, cache/TLB analysis, and annotated disassembly. ## When to Use - Criterion benchmarks show a regression but the cause is unclear - You need to understand *where cycles are spent* inside a hot function - Investigating cache misses, branch mispredictions, TLB pressure, or stall cycles - Comparing two builds at the microarchitectural level - Generating flamegraphs for visual hotspot identification - After `/bench-compare` finds a regression and you need to explain it ## Prerequisites Ensure a release build with debug info exists: ```bash RUSTFLAGS="-C target-cpu=native -C debuginfo=2" cargo build --release ``` The `-C debuginfo=2` flag is critical — it enables `perf annotate` to map samples back to Rust source lines without impacting optimization. ## Profiling Modes Use the mode that matches your investigation. Modes are ordered from broadest to most focused. --- ### Mode 1: Topdown Overview Get a high-level breakdown of where cycles are going: frontend stalls, backend stalls, or retiring useful work. ```bash perf stat -e cpu_cycles,inst_retired,stall_frontend,stall_backend,stall_backend_mem,br_mis_pred_retired \ ./target/release/scanner-rs ../linux 2>&1 ``` **Derived metrics to compute:** | Metric | Formula | Healthy | Investigate | |--------|---------|---------|-------------| | IPC | `inst_retired / cpu_cycles` | > 2.0 | < 1.0 | | Frontend bound % | `stall_frontend / cpu_cycles × 100` | < 10% | > 20% | | Backend bound % | `stall_backend / cpu_cycles × 100` | < 30% | > 40% | | Backend memory % | `stall_backend_mem / cpu_cycles × 100` | < 15% | > 25% | | Branch mispredict rate | `br_mis_pred_retired / br_retired × 100` | < 1% | > 3% | **Interpretation guide:** - **High frontend stalls** → icache misses, large code footprint, bad branch targets. Check iTLB and L1i below. - **High backend stalls** → waiting on data. Drill into cache/TLB (Mode 3). - **High backend memory stalls** → specifically memory-bound. Check L2/LLC misses. - **Low IPC + low stalls** → instruction mix issue (too many µops per instruction, long-latency ops). --- ### Mode 2: Hotspot Sampling + Flamegraph Record samples and identify which functions consume the most cycles. #### 2a. Record ```bash perf record -g --call-graph dwarf,32768 -F 4999 \ ./target/release/scanner-rs ../linux ``` Flags explained: - `-g --call-graph dwarf,32768` — DWARF-based unwinding (works with Rust, unlike frame pointers). 32KB stack dump covers deep async stacks. - `-F 4999` — ~5000 samples/sec. Use a prime number to avoid aliasing with periodic code. #### 2b. Report (interactive) ```bash perf report --no-children --sort=dso,symbol --percent-limit=1.0 ``` - `--no-children` shows self time only (not cumulative call-tree cost). - `--percent-limit=1.0` hides noise below 1%. #### 2c. Report (text, for Claude analysis) ```bash perf report --no-children --sort=symbol --percent-limit=0.5 --stdio 2>&1 | head -80 ``` Pipe the output here for analysis. #### 2d. Flamegraph ```bash perf script | inferno-collapse-perf | inferno-flamegraph > flamegraph.svg ``` If `inferno` is not installed: `cargo install inferno`. Alternatively: ```bash perf script > perf.script # Then use https://www.speedscope.app/ — drag-and-drop perf.script ``` --- ### Mode 3: Cache & TLB Drill-Down When topdown shows backend/memory stalls, measure the cache hierarchy. ```bash perf stat -e l1d_cache,l1d_cache_refill,l1d_cache_lmiss_rd,l2d_cache,l2d_cache_refill,l2d_cache_lmiss_rd,dtlb_walk,itlb_walk,mem_access \ ./target/release/scanner-rs ../linux 2>&1 ``` **Derived metrics:** | Metric | Formula | Healthy | Investigate | |--------|---------|---------|-------------| | L1d miss rate | `l1d_cache_refill / l1d_cache × 100` | < 5% | > 10% | | L1d → L2 miss rate | `l1d_cache_lmiss_rd / l1d_cache_refill × 100` | < 30% | > 50% | | L2 miss rate | `l2d_cache_refill / l2d_cache × 100` | < 10% | > 20% | | L2 → LLC/DRAM rate | `l2d_cache_lmiss_rd / l2d_cache_refill × 100` | < 20% | > 40% | | dTLB miss rate | `dtlb_walk / mem_access × 100` | < 0.5% | > 2% | | iTLB miss rate | `itlb_walk / l1i_cache × 100` | < 0.1% | > 1% | **Common patterns in this codebase:** - **High L1d misses** → check struct layout, false sharing in concurrent paths, random-access patterns in `TimingWheel` or `SetAssociativeCache`. - **High L2 misses** → working set exceeds L2 (256KB/core on Graviton3). Consider data partitioning or reducing struct sizes. - **High dTLB walks** → large heap allocations scattered across pages. Consider hugepages or arena allocation. - **High iTLB walks** → code bloat from monomorphization or heavy inlining. Check generic instantiation count. --- ### Mode 4: Branch Analysis When topdown shows frontend stalls or branch misprediction issues. ```bash perf stat -e br_pred,br_mis_pred,br_retired,br_mis_pred_retired,inst_retired \ ./target/release/scanner-rs ../linux 2>&1 ``` **Derived metrics:** | Metric | Formula | Healthy | Investigate | |--------|---------|---------|-------------| | Speculative mispredict % | `br_mis_pred / br_pred × 100` | < 2% | > 5% | | Retired mispredict % | `br_mis_pred_retired / br_retired × 100` | < 1% | > 3% | | Branch density | `br_retired / inst_retired × 100` | < 20% | > 30% | To find which branches are mispredicting: ```bash perf record -e br_mis_pred_retired -c 1000 -g --call-graph dwarf \ ./target/release/scanner-rs ../linux perf report --stdio --percent-limit=1.0 2>&1 | head -60 ``` --- ### Mode 5: Annotated Source (Per-Line Costs) Once you've identified a hot function from Mode 2, drill into it at the source-line level. ```bash perf annotate --symbol= --stdio 2>&1 ``` For a specific event: ```bash perf record -e l1d_cache_refill -c 10000 --call-graph dwarf \ ./target/release/scanner-rs ../linux perf annotate --symbol= --stdio 2>&1 ``` This shows the percentage of samples on each source line / assembly instruction. Look for: - Lines consuming > 20% of function samples - Unexpected memory loads (possible bounds checks) - Spilled registers (function too complex for register allocator) --- ### Mode 6: A/B Comparison Compare two builds at the hardware counter level to explain a Criterion regression. #### Step 1: Baseline counters ```bash git stash push -m "changes" RUSTFLAGS="-C target-cpu=native -C debuginfo=2" cargo build --release perf stat -r 3 -e cpu_cycles,inst_retired,stall_frontend,stall_backend,stall_backend_mem,l1d_cache_refill,l2d_cache_refill,br_mis_pred_retired \ ./target/release/scanner-rs ../linux 2>&1 | tee /tmp/perf-baseline.txt ``` #### Step 2: Changed counters ```bash git stash pop RUSTFLAGS="-C target-cpu=native -C debuginfo=2" cargo build --release perf stat -r 3 -e cpu_cycles,inst_retired,stall_frontend,stall_backend,stall_backend_mem,l1d_cache_refill,l2d_cache_refill,br_mis_pred_retired \ ./target/release/scanner-rs ../linux 2>&1 | tee /tmp/perf-after.txt ``` #### Step 3: Diff analysis Compare the two files and compute deltas. Focus on: - Did cycle count increase? If so, did IPC drop or did instruction count increase? - If IPC dropped, which stall category increased? - If instructions increased, was it branch density or memory ops? --- ### Mode 7: Lock Contention & Scheduling For diagnosing contention in async/concurrent code paths. ```bash perf stat -e context-switches,cpu-migrations,page-faults \ -e sdt_libpthread:mutex_entry,sdt_libpthread:mutex_acquired \ ./target/release/scanner-rs ../linux 2>&1 ``` For scheduling latency: ```bash perf sched record ./target/release/scanner-rs ../linux perf sched latency --sort max 2>&1 | head -30 ``` --- ## Event Groups for Copy-Paste Pre-built event sets tuned for this system (ARM Neoverse V1, armv8_pmuv3). **Multiplexing note**: This virtualized Graviton3 environment exposes ~3 simultaneous hardware counters. Do **not** use `{}` group pinning syntax — it will fail with ``. Instead, pass events as a comma-separated list and let `perf stat` multiplex automatically. The kernel time-shares counters and scales results; the `(XX.XX%)` annotation next to each counter shows what fraction of runtime it was active. With workloads running ≥1 second, scaled values are reliable. For maximum accuracy on critical ratios, use `-r 3` (repeat 3x) and keep groups small (3-4 events). ### Set: Topdown ``` cpu_cycles,inst_retired,stall_frontend,stall_backend,stall_backend_mem,br_mis_pred_retired ``` ### Set: L1 Cache ``` l1d_cache,l1d_cache_refill,l1d_cache_lmiss_rd,l1i_cache,l1i_cache_refill,l1i_cache_lmiss ``` ### Set: L2 Cache + TLB ``` l2d_cache,l2d_cache_refill,l2d_cache_lmiss_rd,dtlb_walk,itlb_walk,mem_access ``` ### Set: Branches ``` br_pred,br_mis_pred,br_retired,br_mis_pred_retired,inst_retired,cpu_cycles ``` ### Set: Instruction Mix ``` inst_retired,inst_spec,op_retired,op_spec,cpu_cycles,stall ``` ### Set: Backend Memory ``` stall_backend,stall_backend_mem,l1d_cache_lmiss_rd,l2d_cache_lmiss_rd,mem_access,cpu_cycles ``` ### Precision set (3 events, no multiplexing) When you need exact ratios without scaling, use ≤3 events: ``` cpu_cycles,inst_retired,stall_backend ``` --- ## Output Format Report findings using this structure: ```markdown ## Perf Profile: [target / scenario] ### Environment - CPU: ARM Neoverse V1 (Graviton3), 16 cores - Build: `RUSTFLAGS="-C target-cpu=native -C debuginfo=2"` release - Target: [repo or benchmark name] ### Topdown Summary | Metric | Value | Assessment | |--------|-------|------------| | IPC | X.XX | [good/investigate] | | Frontend bound | X.X% | [ok/high] | | Backend bound | X.X% | [ok/high] | | Backend memory | X.X% | [ok/high] | | Branch mispredict | X.X% | [ok/high] | ### Hotspots (top 5 by self%) | Rank | Symbol | Self % | Module | Likely Cause | |------|--------|--------|--------|--------------| | 1 | func_name | XX.X% | scanner_rs | [explanation] | ### Cache Hierarchy (if relevant) | Level | Accesses | Misses | Miss Rate | Assessment | |-------|----------|--------|-----------|------------| | L1d | X.XXB | X.XXM | X.X% | [ok/high] | | L2 | X.XXM | X.XXM | X.X% | [ok/high] | ### Root Cause Analysis [Narrative explaining what the numbers mean for this specific code. Connect PMU data to source-level patterns. Reference specific lines/functions.] ### Recommendations 1. **[Issue]** at `file:line` — [specific fix with rationale tied to PMU data] - Expected impact: [which metric should improve and by roughly how much] - Validate: `perf stat -e ...` ### Validation Plan [Commands to re-run after applying fixes to confirm improvement] ``` ## Caveats - **Multiplexing**: This virtualized environment supports ~3 simultaneous counters. `perf stat` multiplexes automatically — the `(XX.XX%)` annotation shows sampling duty cycle. With ≥1s workloads, scaled values are reliable. Never use `{}` group pinning. - **DWARF unwinding cost**: `--call-graph dwarf` adds ~5-15% overhead to the profiled process. For tight latency measurements, use `perf stat` instead of `perf record`. - **Kernel symbols**: Some symbols may show as `[kernel.kallsyms]`. These are kernel-side costs (syscalls, page faults). If they dominate, investigate I/O patterns or memory allocation. - **Inlining**: Heavily inlined functions may not appear as separate symbols. Use `perf annotate` on the caller to see inlined code. - **Async runtimes**: Tokio worker threads share names. Use `-t` (per-thread) recording to separate them if needed: `perf record -t `. ## Related Skills - `/bench-compare` — Criterion before/after measurement (use first to detect regressions) - `/perf-regression` — Full regression workflow with acceptance criteria - `/performance-analyzer` — Static code analysis for perf anti-patterns - `/rust-hotspot-finder` — Classify hotspots by risk before profiling --- Source attribution: ahrav Original project: https://github.com/ahrav/scratch-scanner-rs/blob/3fb8740fb576a9859c84b7a5a08f3bbae06a9f05/.claude/skills/linux-perf-profile/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2026 ahrav Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: ahrav Original project: https://github.com/ahrav/scratch-scanner-rs/blob/3fb8740fb576a9859c84b7a5a08f3bbae06a9f05/.claude/skills/linux-perf-profile/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Development & Engineering, Product Management Roles: Backend Developer, Data Analyst, DevOps / SRE, ML / AI Engineer, Technical Product Manager Specialties: Backend Tools: Perf Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: d01f06ec25c102e21fd15e6373bab32c1de10b827473340d7518850a984653e0