This optimization artifact establishes an advanced diagnostic framework and tensor-level computational audit of gradient dynamics, subspace trajectories, and optimization stability in Low-Rank Adaptation (LoRA) mechanics across edge-constrained transformer models. Rather than treating low-rank adapter updates as decoupled black-box heuristics, this codebase isolates the mathematical, structural, and runtime challenges governing gradient-based PEFT optimization on resource-limited hardware. By analyzing how gradient update paths deviate from full-rank trajectories grad_W(L), this repository examines how subspace alignment, singular value decay
This empirical evaluation establishes an extensive diagnostic framework and tensor-level computational audit of Low-Rank Adaptation (LoRA) mechanics across sub-billion parameter transformer models, specifically focusing on the SmolLM2 architecture family (135M to 1.7B parameter baselines). Rather than treating adapter rank scaling as a black-box hyperparameter optimization heuristic, this codebase isolates the mathematical, structural, and runtime dynamics governing parameter-efficient fine-tuning (PEFT) updates. By projecting weight updates onto decomposed low-rank matrices (ΔW = B · A, where A ∈ ℝ^(r × k) and B ∈ ℝ^(d × r) with rank r ≪ min
