Spectral Rank Bottleneck Dynamics and Low-Rank Adaptation Mechanics Across Sub-Billion Transformer Layers
This empirical evaluation establishes an extensive diagnostic framework and tensor-level computational audit of Low-Rank Adaptation (LoRA) mechanics across sub-billion parameter transformer models, specifically focusing on the SmolLM2 architecture family (135M to 1.7B parameter baselines). Rather than treating adapter rank scaling as a black-box hyperparameter optimization heuristic, this codebase isolates the mathematical, structural, and runtime dynamics governing parameter-efficient fine-tuning (PEFT) updates. By projecting weight updates onto decomposed low-rank matrices (ΔW = B · A, where A ∈ ℝ^(r × k) and B ∈ ℝ^(d × r) with rank r ≪ min
Resource information
- Format
- ipynb, zip
- Original repository
- 22696232
- DOI
- 10.5281/zenodo.22696232
- Credit basis
- Creators in Zenodo record metadata
- Access
- Open files available from the original Zenodo deposit
- Last checked
- Nov 5, 2026
- License
- cc-by-4.0
- Last record update
- Nov 5, 2026
About this resource
This empirical evaluation establishes an extensive diagnostic framework and tensor-level computational audit of Low-Rank Adaptation (LoRA) mechanics across sub-billion parameter transformer models, specifically focusing on the SmolLM2 architecture family (135M to 1.7B parameter baselines). Rather than treating adapter rank scaling as a black-box hyperparameter optimization heuristic, this codebase isolates the mathematical, structural, and runtime dynamics governing parameter-efficient fine-tuning (PEFT) updates. By projecting weight updates onto decomposed low-rank matrices (ΔW = B · A, where A ∈ ℝ^(r × k) and B ∈ ℝ^(d × r) with rank r ≪ min(d, k)), this repository examines how intrinsic dimensionality bounds govern representation capacity, optimization stability, and hardware execution efficiency . The studies provides various modular, reproducible analysis and interactive diagnostic notebooks designed for layer-wise tensor inspection, spectral auditing, and memory profiling. Core methodological and architectural investigations include: Rank Bottleneck Dynamics & SVD Spectral Profiling: Evaluates singular value decay rates, Frobenius norm retention, and intrinsic rank dimensionality across adapter bottleneck configurations (r ∈ {4, 8, 16, 32}). It quantifies how singular value spectrum energy decays across layers and correlates target rank choices with baseline parameter scale transitions from 135M up to 1.7B parameters ( unit5_scaleout_spectra.ipynb ). Alpha Scaling Factor (α/r) & Gradient Stability: Analyzes the numerical dynamics of the scaling hyperparameter α/r under varying rank configurations. The suite audits how invariant scaling schedules influence gradient norm trajectories, learning rate dynamics, and matrix delta magnitudes across deep linear projections, preventing optimization collapse during low-rank backpropagation. Target Sub-Layer & Projection Isolation: Diagnoses the differential impact of rank allocation across distinct functional sub-layers of the transformer block. It isolates performance and representation trade-offs when restricting low-rank updates strictly to Multi-Head Attention query/value projections (W_q, W_v) versus Feed-Forward Network expansion gates (SwiGLU W_gate and W_up projection layers). VRAM Allocation & Backpropagation Memory Footprint: Executes detailed CUDA memory profiling to measure exact operational savings during backward passes. It quantifies VRAM footprint reductions achieved by keeping base checkpoint weights W_0 ∈ ℝ^(d × k) strictly frozen while allocating optimizer states (e.g., Adam m and v vectors) exclusively to the low-rank bottleneck parameters A and B. Zero-Latency Weight Merging & Floating-Point Precision Dynamics: Evaluates floating-point accumulation behaviors (FP16, BF16, and FP32) during static adapter weight integration (W_eff = W_0 + (α/r)(B · A)). It quantifies numerical drift and precision degradation during static checkpoint merging, validating zero-latency inference serving without runtime adapter compute overhead.