Full text
Text edition from the author’s arXiv HTML. Figures and equations may display differently from the original.
Recent advances in diffusion models have enabled high-quality video generation, but the additional temporal dimension significantly increases computational costs, making training and inference on long videos prohibitively expensive. In this paper, we identify a phenomenon we term Spatiotemporal Energy Decay in video diffusion models: post-softmax attention scores diminish as spatial and temporal distance between tokens increase, akin to the physical decay of signal or waves over space and time in nature. Motivated by this, we propose Radial Attention, a scalable sparse attention mechanism with 𝒪(nlogn) complexity that translates energy decay into exponentially decaying compute density, which is significantly more efficient than standard 𝒪(n2) dense attention and more expressive than linear attention. Specifically, Radial Attention employs a simple, static attention mask where each token attends to spatially nearby tokens, with the attention window size shrinking with temporal distance. Moreover, it allows pre-trained video diffusion models to extend their generation length with efficient LoRA-based fine-tuning. Extensive experiments show that Radial Attention maintains video quality across Wan2.1-14B, HunyuanVideo, and Mochi 1, achieving up to a 1.9× speedup over the original dense attention. With minimal tuning, it enables video generation up to 4× longer while reducing training costs by up to 4.4× compared to direct fine-tuning and accelerating inference by up to 3.7× compared to dense attention inference. Code is released at https://github.com/mit-han-lab/radial-attention.
“Nothing spreads without loss; every signal, every influence, every attention — decays with distance.” — Inspired by thermodynamic principles
Figure 1: We present Radial Attention, a sparse attention mechanism with 𝒪(nlogn) computational complexity. Radial Attention accelerates pre-trained HunyuanVideo Kong et al. (2024) by 1.9× at its default video length while maintaining comparable video quality. When generating 4× longer videos, it reduces tuning costs by up to 4.4× and speeds up inference by up to 3.7× versus dense attention.
1 Introduction
Figure 2: Radial Attention reduces the computational complexity of attention from 𝒪(n2) to 𝒪(nlogn). When generating a 509-frame 720p video with HunyuanVideo, it reduces the attention computation by 9×, achieves 3.7× speedup, and saves 4.4× tuning costs.
Diffusion models have achieved remarkable success in generating high-quality images Ho et al. (2020); Rombach et al. (2022). Recent advances have extended their capabilities to video generation, producing visually compelling results OpenAI (2024); Yang et al. (2024b); HaCohen et al. (2024); Wang et al. (2025a); Kong et al. (2024).
However, such advances incur substantial computational costs. Unlike image synthesis, the temporal dimension in video synthesis greatly increases token counts compared to images, and the quadratic scaling of self-attention with context length renders training and inference on long videos computationally prohibitive, restricting model scalability.
Several prior works tried to mitigate this challenge using sparse attention. For instance, as illustrated in Figure 3(a), Sparse VideoGen (SVG) Xi et al. (2025) employs an online profiling strategy that classifies each attention head as either spatial or temporal and then applies the corresponding sparse mask. While this can accelerate inference, it poses challenges during training, especially for longer videos. The profiling may misclassify attention heads on unseen data distributions, whose error can be reinforced during optimization, leading to degraded performance. Other approaches replace the softmax attention with linear alternatives Wang et al. (2025b); Xie et al. (2025), but these often require substantial architectural changes, where modest fine-tuning is typically insufficient to recover the original video quality.
In physics, it is well known that signals and waves lose energy as they propagate through space and time. Inspired by this principle, we observe a similar phenomenon in attention: post-softmax attention scores between tokens diminish as their spatial or temporal distance increases (see Figure 4(b)). We term this phenomenon Spatiotemporal Energy Decay and model the decay as an exponential function of both spatial and temporal distances. Based on this model, we unify spatial and temporal attention heads in SVG Xi et al. (2025) into Radial Attention, a scalable sparse attention mechanism with 𝒪(nlogn) computational complexity (see Figure 2). Radial Attention employs a static sparse attention mask to translate the concept of energy decay into computation density decay. The mask design is simple yet effective: each token attends to others at similar spatial locations, while the attention window shrinks exponentially with temporal distance, as illustrated in Figure 3(b).
Moreover, since Radial Attention only prunes unimportant token relations without modifying the underlying softmax attention mechanism, it enables efficient adaptation of pre-trained video diffusion models to longer sequences using lightweight fine-tuning, such as LoRA Hu et al. (2022). Compared to full-parameter fine-tuning with dense attention, it achieves better video quality, as LoRA focuses on updating parameters most critical for temporal coherence and visual fidelity. The length-extension LoRA is also compatible with existing style LoRAs (see Section 5.2).
Figure 3: Attention pipelines of SVG Xi et al. (2025) and our Radial Attention. Softmax is omitted for clarity. (a) SVG dynamically selects either a spatial or temporal attention for each head to speed up inference. However, it does not overcome the original model’s length limitation and cannot be trained on unseen distributions like longer videos. (b) Our Radial Attention uses a static mask that unifies spatial and temporal attention with 𝒪(nlogn) computational complexity. This static design enables efficient longer-video adaptation.
When generating videos at the default length, Radial Attention accelerates leading video diffusion models of Wan2.1-14B Wang et al. (2025a), HunyuanVideo Kong et al. (2024) by up to 1.9× speedup. When generating 4× longer videos, Radial Attention reduces tuning costs by up to 4.4× and accelerates inference by up to 3.7× without sacrificing quality. Some visual examples on HunyuanVideo can be found in Figure 1.
2 Related Work
Video diffusion models. Diffusion models have achieved state-of-the-art (SOTA) results in image synthesis Ho et al. (2020); Rombach et al. (2022); Podell et al. (2024); Xie et al. (2025). Researchers further extend them to the video domain. Early approaches Ho et al. (2022); Blattmann et al. (2023b); Blattmann et al. (2023a); Guo et al. (2024b) adapted 2D UNets Ho et al. (2020); Ronneberger et al. (2015) to handle frame sequences by adding temporal modules. Ever since the advent of Sora OpenAI (2024), the community has largely shifted to use DiT Peebles and Xie (2023) as the backbone. Latte Ma et al. (2024a) first proposed decoupled spatial and temporal attention for modeling video sequences. To better capture long-range dependencies and jointly model spatial-temporal dynamics, recent SOTA models have adopted 3D dense attention Lin et al. (2024); Peng et al. (2025); Yang et al. (2024b); Team (2024); Kong et al. (2024); Wang et al. (2025a); HaCohen et al. (2024). However, dense attention is computationally intensive due to the additional temporal dimension, and its cost scales quadratically with the number of frames, posing substantial challenges for both training and deployment.
Efficient video generation. Many techniques developed to accelerate image diffusion models—such as timestep distillation Yin et al. (2025); Li et al. (2024a), caching Ma et al. (2024b); Liu et al. (2025), quantization Li et al. (2023); Zhao et al. (2025b); Li et al. (2025), and distributed inference Li et al. (2024b); Fang et al. (2024b); Fang et al. (2024a)—also apply to video diffusion. However, video models often rely on 3D dense attention, shifting the bottleneck from feedforward to attention layers. Recent works like SageAttention Zhang et al. (2025c); Zhang et al. (2025a); Zhang et al. (2025b); Zhang et al. (2025e) and FlashAttention-3 Shah et al. (2024) show that quantizing attention can significantly speed up inference. In large language models (LLMs), sparse attention has been widely applied for acceleration Xiao et al. (2024); Chen et al. (2024); Jiang et al. (2024); Zhang et al. (2025d); Tang et al. (2024); Xu et al. (2025); Yang et al. (2025); Chen et al. (2025); Xiao et al. (2025). For instance, Long LoRA Chen et al. (2024) combines two local sparse attention patterns with shifting to achieve a global receptive field in video understanding. PowerAttention Chen et al. (2025) restricts attention to power-of-two token distances, yielding 𝒪(nlogn) complexity. However, these methods ignore the inherent spatial and temporal structure in video data, making them suboptimal for video generation (see Section 5.2). To better exploit this structure, several video-specific sparse attention methods have been proposed Xi et al. (2025); Zhang et al. (2025f); Xia et al. (2025); Tan et al. (2025). For example, STA Zhang et al. (2025f) uses sliding 3D windows for local attention, and SVG (Xi et al., 2025) dynamically selects spatio-temporal patterns for each head. Both improve efficiency but struggle with long videos: STA’s fixed receptive field limits long-range dependencies, while SVG’s runtime profiling becomes unreliable for unseen long video distributions. In contrast, our Radial Attention employs a static 𝒪(nlogn) pattern across all heads, accelerating both training and inference and enabling efficient extension to longer videos.
Long video generation. Due to the quadratic cost of dense attention, training and inferernce on long videos remain highly expensive. RIFLEx Zhao et al. (2025a) extends video length by modifying RoPE Su et al. (2024) frequencies to tackle temporal repetition and motion deceleration, allowing 2× extrapolation with the pre-trained models. However, it still suffers from poor video quality (e.g., blurring) when generating longer videos. Dalal et al. generate short video segments and stitch them together via test-time training layers Dalal et al. (2025). Framepack Zhang and Agrawala (2025) adopts an autoregressive strategy, generating short clips sequentially based on context frames that are encoded into a fixed number of tokens. Other approaches replace dense attention with linear attention Xie et al. (2025); Wang et al. (2025b); Liu et al. (2024b); Han et al. (2023); Liu et al. (2024a); Katharopoulos et al. (2020); Yang et al. (2024a); Dao and Gu (2024), offering faster computation and global receptive fields. However, linear attention struggles to capture local details Cai et al. (2023), often degrading quality. Our Radial Attention strikes a middle ground between 𝒪(n2) dense attention and 𝒪(n) linear attention, achieving 𝒪(nlogn) complexity while preserving the visual fidelity. Moreover, it can be efficiently fine-tuned from existing models using LoRA Hu et al. (2022), enabling scalable longer-video generation with minimal overhead.
Attention with 𝒪(nlogn) complexity. Preliminary efforts in this direction include Reformer Kitaev et al. (2020), which approximates dense attention via locality-sensitive hashing; H-Transformer Zhu and Soricut (2021), which imposes a hierarchical structure on the attention matrix; Multi-resolution attention Zeng et al. (2022), which recursively refines high-attention regions; Fast Multipole Attention Kang et al. (2024), which adapts the classical fast multipole method; and LogSparse Transformer Li et al. (2019) for time-series forecasting, which restricts each token to attend to 𝒪(logn) positions per layer. However, these methods are often hardware-unfriendly and exhibit limited scalability. In contrast, our approach employs a simple, block-friendly static mask that scales efficiently while maintaining strong modeling capacity.
3 Preliminary
Diffusion models synthesize videos by sampling Gaussian noise 𝑿T∼𝒩(𝟎,𝑰) in a latent space and progressively denoising it through a neural network to produce a clear latent 𝑿0, which is subsequently decoded into the final video using a pre-trained decoder. Compared to images, videos introduce an additional temporal dimension, significantly increasing the number of latent tokens. For instance, generating a 5-second 720p video in HunyuanVideo Kong et al. (2024) requires approximately 110K tokens. Excessive latent compression degrades video quality Rombach et al. (2022), limiting token reduction.
To capture spatiotemporal correlation in video generation, recent models Wang et al. (2025a); Kong et al. (2024); Team (2024); Yang et al. (2024b) use 3D dense attention, which computes interactions between all token pairs. Given n tokens with embedding dimension d, attention is computed as:
Attention(𝑸,𝑲,𝑽)=softmax(𝑸𝑲⊤d)𝑽, (1)
where 𝑸,𝑲,𝑽∈ℝn×d are the query, key, and value matrices. The computation of 𝑸𝑲T incurs 𝒪(n2) time and memory complexity. While FlashAttention Dao et al. (2022); Dao (2024) series reduce memory overhead, the quadratic time complexity remains a bottleneck, especially for long or high-resolution videos. Thus, designing more efficient attention mechanisms is vital for scaling video diffusion models.
To mitigate this computational burden, sparse attention restricts interactions to a subset of token pairs. Formally, this is achieved by adding a sparsity mask 𝑴∈{−∞,0}n×n to the attention logits:
SparseAttention(𝑸,𝑲,𝑽)=softmax(𝑸𝑲⊤+𝑴d)𝑽. (2)
Entries set to −∞ are ignored in the softmax computation. Various schemes have been proposed to construct the mask. Static methods, such as STA Zhang et al. (2025f), use fixed sparsity patterns but are less expressive. In contrast, dynamic schemes like SVG Xi et al. (2025) does dynamic sparse pattern based on input content to improve fidelity. However, dynamic masking introduces online mask decision overhead and does not apply to training. Can we design a static attention pattern that matches the expressiveness of dynamic methods and can also be used in training?
4 Method
The key insight of Radial Attention is that attention scores between tokens decay with increasing spatial and temporal distance. This motivates us to allocate computation based on the inherent spatiotemporal correlations. In Section 4.1, we characterize the spatiotemporal energy decay phenomenon in attention. In Section 4.2, we formally define Radial Attention, which translates energy decay into corresponding compute density reduction, enabling speedup. We also analyze its complexity and approximation error, showing that the complexity is 𝒪(nlogn) and the effectiveness of our mask. Finally, in Section 4.3, we show how to extend pre-trained models to longer videos using Radial Attention.
Figure 4: (a) Example spatial and temporal attention maps from HunyuanVideo (defined in Section 4.1). (b) Attention score distributions. (b1): Average score between tokens at the same spatial location decreases with temporal distance (b2): Average attention score within a frame decreases with spatial distance. Spatial and Temporal Attention refer to the distributions derived from the corresponding maps in (a). Average means averaging over multiple random maps and diffusion steps. The plots indicate that spatial attention shows a high temporal decay and relatively low spatial decay, while temporal attention exhibits the opposite.
4.1 Spatiotemporal Energy Decay in Attention
In Figure 4(a), we show two post-softmax attention maps from HunyuanVideo Kong et al. (2024). Following the terminology in SVG Xi et al. (2025), the left map is referred to as spatial attention, where each token primarily attends to nearby tokens within adjacent frames. The right map represents temporal attention, where each token focuses on tokens at the same spatial location across different frames. Figure 4(b) illustrates the attention score distributions of these two maps, along with a third curve of averaged attention scores over multiple heads and diffusion steps. In Figure 4(b1), we show the average attention score between tokens at the same spatial location but with increasing temporal distance. In Figure 4(b2), we show the average score between tokens in the same frame as spatial distance increases. In both cases, attention scores exhibit a clear decay pattern with increasing distance between the query and key tokens. We refer to this phenomenon as Spatiotemporal Energy Decay. Moreover, regression analysis suggests that this decay closely follows an exponential distribution (see Section 5.3).
Specifically, following the notation from Section 3, assume the video latent consists of f frames, each containing s tokens (in total n=fs tokens). Consider a query token located at the k0-th spatial position of the i0-th frame. The corresponding attention score after applying softmax, denoted by 𝒑∈[0,1]n, is given by 𝒑=softmax(𝑸i0s+k0𝑲⊤). Then there exist constants α,β>0 and Crel>0, for each key token at spatial position l in frame j satisfying
pjs+l≤Crele−α|j−i0|−β|l−k0|pi0s+k0. (3)
Parameters α and β control temporal and spatial decay, respectively. High β (strong spatial locality) and low α model temporal attention, while high α and low β capture spatial attention, as shown in the empirical plots in Figure 4(b). This motivates our unified sparsity pattern that leverages both spatial and temporal decay in a principled manner.
4.2 Radial Attention: Convert the Energy Decay to Compute Density Decay
Radial Attention simulates energy decay through compute density decay to save computation.
Temporal density decay. Along the temporal dimension, Radial Attention applies an exponential decay rule: the compute density between tokens in frame i and frame j is (12)⌊log2(max(|i−j|,1))⌋. This forms a structured pattern as illustrated in Figure 5(a) with 2⌈log2(max(f,2))⌉−1 diagonal bands centered on the main diagonal (band 0). Bands above and below the diagonal are indexed as 1,2,3,… and −1,−2,−3,…, respectively. Each band’s width doubles relative to the previous one, ensuring that the total computation per band remains bounded by a constant. The attention from tokens in frame i to frame j lies in band sign(j−i)⋅⌊log2max(|i−j|,1)⌋. The central band (band 0) retains 100% compute density, while each successive band moving outward has half the compute density of the preceding one – producing a radial decay effect with progressively lighter colors.
Spatial density decay. As observed in Figure 4 and formalized in Equation 3, most attention energy is concentrated on tokens at similar spatial locations across frames. We preserve these high-energy interactions, which yield diagonal-like structures within each frame-to-frame attention block. Due to temporal decay, the computed diagonal width of these blocks shrinks as the temporal distance between frames increases. Specifically, as shown in Figure 5(b), the diagonal width for attention between frame i and frame j is given by ⌊s2⌊log2max(|i−j|,1)⌋⌋. If it falls below 1, instead of further narrowing the diagonal, we reduce the frequency of diagonals. Specifically, we only retain diagonals in those blocks where |i−j|mod⌈2⌊log2max(|i−j|,1)⌋s⌉=0 to keep the same amortized attention density decay.
Figure 5: (a) The compute density pattern. The attention map is divided into 2⌈log2(max(f,2))⌉−1 bands (here, the number of frames f=12) based on the temporal distance between tokens. The central band has full compute density, while each successive outer band has half the density of the previous one. Except for band ±1, each band also doubles the diagonal width of its predecessor. (b) The corresponding attention mask for (a). The compute density is reflected in the compute diagonal width of each frame-to-frame block. When the diagonal width drops below 1, we reduce the frequency of diagonals. We additionally add an attention sink. (c) An example mask used in HunyuanVideo, illustrating the final sparsity pattern in practice.
Formal definition. Here we formally define the Radial Attention mask. We construct a 4D attention mask 𝑴~∈{−∞,0}f×f×s×s, where each element M~i,j,k,l=0 indicates that the token at spatial position k in frame i is permitted to attend to the token at position l in frame j. Conversely, M~i,j,k,l=−∞ denotes that attention between the token pair is suppressed. The mask is constructed according to:
M~i,j,k,l={0,if 2⌊log2max(|i−j|,1)⌋≤s and |k−l|+1≤s2⌊log2max(|i−j|,1)⌋0,if |i−j|mod⌈2⌊log2max(|i−j|,1)⌋s⌉=0 and k=l−∞.otherwise (4)
The final attention mask 𝑴∈{−∞,0}n×n used in the attention operation of Equation 2 is obtained by flattening frame and spatial indices: Mis+k,js+l=M~i,j,k,l. For better quality, we incorporate an attention sink Xiao et al. (2024); Xi et al. (2025) as the first frame’s attention is crucial. Figure 5(c) shows a example mask we use in HunyuanVideo for generating a 253-frame 720p video. This strategy keeps spatial interactions with high temporal proximity while using sparse sampling for distant frames to maintain efficiency.
Relation to SVG. Radial Attention unifies spatial and temporal attention in SVG Xi et al. (2025) using a single attention mask. Specifically, the central band (band 0 in Figure 5(a)) of our mask already captures dense spatial interactions, effectively subsuming the spatial attention in SVG. For temporal attention, SVG overlooks temporal decay, allocating unnecessary computation to distant frames with low relevance. In contrast, Radial Attention reduces attention to these regions and reallocates the budget toward tokens nearer in time, achieving both improved efficiency and enhanced modeling.
Complexity analysis. The computational cost of our method is proportional to the number of zeros in the attention mask 𝑴~. When the number of frames f is large, we derive the following upper bound:
#zeros in 𝑴~ ≤4s2f⏟central band and sink+∑r=1⌊log2s⌋2r+1f2s22r⏟diagonal width≥1+∑r=⌊log2s⌋+1⌈log2f⌉−12⌊log2s⌋+1fs⏟diagonal width<1 (5)
≤4s2flog2f=4sn(log2n−log2s). (6)
A detailed derivation of Equation 5 can be found in Appendix A.1. From Equation 6, we find that for long videos (i.e., large f) with fixed resolution s, the computational complexity scales as 𝒪(nlogn). Empirical results on HunyuanVideo, shown in Figure 2, confirm this trend. Notably, our Radial Attention reduces attention computation by 9× compared to dense attention for 4× longer videos.
Error analysis. Following Equation 3, we derive an error bound for the attention score corresponding to a query token at position k0 in the i0-th frame. 𝒑~=softmax(𝑸i0s+k0𝑲⊤+𝑴~i0s+k0) denotes the masked attention score. The ℓ1 attention error of our approximated attention is bounded as follows:
‖p~−p‖1≤Crel[8e−β(s2+1)(1−e−α)(1−e−β)+ 41+e−β1−e−βe−α(s+1)1−e−α]=O(Crele−min(β/2,α)s). (7)
Proof details are provided in Appendix A.2. As Equation 7 shows, the error decreases exponentially with larger decay rates α and β. In Section 5.3, we further empirically compare this error bound to that of SVG, showing that Radial Attention achieves smaller errors, thereby validating its effectiveness.
Hardware-friendly block sparsity. To ensure efficient execution on modern hardware, attention is computed over 128×128 blocks rather than individual 1×1 tokens Li et al. (2022); Xi et al. (2025); Jiang et al. (2024); Xu et al. (2025); Yang et al. (2025); Dao et al. (2022).
4.3 Low-Rank Adaptation for Long Videos
Although we employ an efficient attention mechanism, the pre-trained model was originally trained on short videos. Recent works Zhao et al. (2025a) have explored training-free methods for extending generation to longer videos, but their performance remains limited due to length distribution mismatch. Training directly on long videos, meanwhile, is computationally prohibitive. Radial Attention alleviates this challenge by reducing the training time complexity to 𝒪(nlogn). Importantly, it preserves critical inter-token relations in the softmax attention, allowing the original pre-trained weights to remain largely intact. Thus, only minimal fine-tuning is required. To further minimize training overhead, we incorporate low-rank adapters (LoRA) Hu et al. (2022); Chen et al. (2024) into the attention mechanism. Specifically, LoRA is applied to the query, key, value, and output projections of the attention layers, enabling efficient fine-tuning with significantly reduced memory and computational costs. Empirically, we find that LoRA fine-tuning with Radial Attention not only minimizes overhead but also improves video quality by refining only the most critical weights and attention more effectively. See Section 5.3 for detailed results.
5 Experiments
5.1 Setups
Models. We benchmark our method on three popular text-to-video diffusion models: Mochi 1 Team (2024), HunyuanVideo Kong et al. (2024), and Wan2.1 Wang et al. (2025a), which contain 10, 13, and 14 billion parameters, respectively. Mochi 1 can generate up to a 5-second video with 480p resolution and 162 frames. HunyuanVideo can generate up to a 5-second video with 720p resolution and 125 frames. Wan2.1-14B can generate up to a 5-second video with 720p and 81 frames.
Table 1: Quantitative results at the default video length. Under the same computation budget, our method consistently outperforms STA and PA in PSNR, SSIM, and LPIPS, matches the video fidelity of SVG, and achieves 1.8× speedup on HunyuanVideo and Wan2.1-14B on a single H100 GPU.
Model Method PSNR (↑) SSIM (↑) LPIPS (↓) Vision Reward (↑) PFLOPs Latency (s) Speedup
Original – – – 0.141 612 1649 –
Hunyuan STA (FA3) 26.7 0.866 0.167 0.132 331 719 2.29×
Video PA 22.1 0.764 0.256 0.140 339 1002 1.65×
(117 frames) SVG 27.2 0.895 0.114 0.144 340 867 1.90×
Ours 27.3 0.886 0.114 0.139 339 876 1.88×
Wan2.1-14B (69 frames) Original – – – 0.136 560 1630 –
STA (FA3) 22.9 0.830 0.171 0.132 322 812 2.01×
PA 22.4 0.790 0.176 0.126 324 978 1.67×
SVG 23.2 0.825 0.202 0.114 324 949 1.71×
Ours 23.9 0.842 0.163 0.128 323 917 1.77×
Table 2: Quantitative results at the extended (2× and 4×) video lengths. With minimal fine-tuning, our method maintains the quality regarding Vision Reward and multiple VBench dimensions (Subject Consistency, Aesthetic Quality, and Image Quality) when the length grows. It also achieves high sparsity, reducing training costs by up to 4.4× and delivering up to 3.7× inference speedup.
Model #Frames Method Sparsity Training Time (h) Training Speedup Inference Time (s) Inference Speedup Vision Reward (↑) VBench
S.C. A.Q. I.Q.
Hunyuan Video 125 (1×) Original 0.00% – – 225 – 0.119 0.959 0.643 0.672
253 (2×) Original 0.00% – – 797 1.00× 0.122 0.953 0.603 0.611
RIFLEx 0.00% – – 797 1.00× 0.128 0.969 0.622 0.614
Spatial 80.5% 16.0 2.81× 335 2.38× 0.054 0.979 0.607 0.670
Temporal 80.7% 16.2 2.78× 338 2.36× 0.104 0.963 0.620 0.658
Long LoRA 80.6% 16.6 2.71× 363 2.20× 0.112 0.958 0.620 0.685
PA Chen et al. (2025) 80.4% 16.7 2.69× 334 2.39× 0.109 0.967 0.608 0.653
SANA – 12.8 3.52× 285 2.80× -0.205 0.907 0.300 0.442
Full 0.00% 45.0 1.00× 797 1.00× 0.124 0.955 0.616 0.648
Ours 80.8% 16.2 2.78× 339 2.35× 0.126 0.968 0.623 0.663
509 (4×) Original 0.00% – – 2895 1.00× 0.054 0.988 0.545 0.451
RIFLEx 0.00% – – 2895 1.00× 0.037 0.989 0.539 0.456
Spatial 88.3% 20.7 4.52× 755 3.83× 0.112 0.922 0.598 0.664
Temporal 88.2% 21.1 4.44× 774 3.74× 0.083 0.972 0.597 0.646
Long LoRA 88.4% 20.9 4.48× 803 3.61× 0.130 0.936 0.618 0.689
PA Chen et al. (2025) 88.2% 21.8 4.29× 766 3.78× 0.128 0.950 0.590 0.648
Full 0.00% 93.6 1.00× 2895 1.00× 0.133 0.977 0.590 0.635
Ours 88.3% 21.4 4.37× 781 3.71× 0.134 0.973 0.623 0.672
Mochi 1 163 (1×) Original 0.00% – – 112 – 0.071 0.973 0.623 0.672
331 (2×) Original 0.00% – – 302 1.00× 0.040 0.937 0.551 0.466
Spatial 76.1% 8.57 1.75× 186 1.62× 0.088 0.935 0.596 0.595
Temporal 76.3% 8.54 1.76× 189 1.60× 0.075 0.936 0.591 0.593
Long LoRA 76.0% 9.07 1.65× 210 1.44× 0.095 0.950 0.596 0.630
PA Chen et al. (2025) 77.8% 8.53 1.76× 183 1.65× 0.101 0.946 0.610 0.626
SANA – 8.22 1.82× 166 1.82× -0.201 0.905 0.334 0.568
Full 0.00% 15.0 1.00× 302 1.00× 0.095 0.923 0.610 0.594
Ours 76.4% 8.43 1.78× 185 1.63× 0.110 0.951 0.615 0.602
667 (4×) Original 0.00% – – 992 1.00× -0.091 0.916 0.383 0.322
Spatial 85.2% 17.4 2.83× 382 2.60× 0.091 0.930 0.611 0.585
Temporal 85.4% 17.6 2.80× 393 2.52× 0.028 0.931 0.556 0.536
Long LoRA 86.0% 19.0 2.59× 426 2.33× 0.086 0.944 0.584 0.543
PA Chen et al. (2025) 86.5% 17.3 2.84× 381 2.60× 0.107 0.956 0.633 0.650
Full 0.00% 49.2 1.00× 992 1.00× 0.099 0.934 0.613 0.613
Ours 85.5% 17.4 2.83× 386 2.57× 0.113 0.958 0.618 0.638
Wan2.1 -14B 81 (1×) Original 0.00% – – 1630 – 0.135 0.973 0.623 0.672
161 (2×) Original 0.00% – – 5735 1.00× 0.109 0.946 0.598 0.614
Full 0.00% 28.0 1.00× 5735 1.00× 0.150 0.966 0.590 0.689
Ours 73.6% 14.5 1.93× 2847 2.01× 0.145 0.981 0.607 0.677
Benchmarks. We use Vision Reward Xu et al. (2024) (higher is better) to approximate the human rating of the generated videos. For pre-trained models evaluated at their default video lengths, we further report PSNR and SSIM to quantify numerical similarity, and LPIPS Zhang et al. (2018) to assess perceptual differences between the outputs of the original models and the benchmarked methods. For longer-video generation, we additionally use VBench-longZheng et al. (2025) to evaluate our fine-tuned models. Specifically, we report metrics of subject consistency, aesthetic quality, and imaging quality, where the original models exhibit notable degradation.
Baselines. We compare Radial Attention against the following methods:
SVG Xi et al. (2025): Accelerates video models with sparse attention by dynamically classifying attention heads as spatial or temporal and applying corresponding masks.
Spatial/Temporal: The respective attention masks used in SVG’s spatial and temporal heads, as described in Section 4.1.
STA Zhang et al. (2025f): Applies sliding window attention to capture spatially and temporally local dependencies.
PowerAttention (PA) Chen et al. (2025): A sparse attention mechanism with 𝒪(nlogn) complexity for LLMs, attending only to tokens at power-of-two distances.
LongLoRA Chen et al. (2024): Uses shifted local attention to efficiently extend the context window of LLMs.
SANAXie et al. (2025): An efficient diffusion model backbone with linear attention. We replace softmax attention with SANA’s for adapting to longer videos.
RIFLExZhao et al. (2025a): Training-free video length extrapolation by adjusting the frequency of RoPE Su et al. (2024).
Implementation details. In terms of Radial Attention implementation, we use FlashInfer Ye et al. (2025) for inference and Block-Sparse-Attention Guo et al. (2024a) with FlashAttention-2 Dao (2024) backend during training. For default-length inference, we evaluate HunyuanVideo on 117 frames at 768p resolution (768×1280), and Wan2.1 on 69 frames at the same resolution. Following SVG, we apply dense attention during the first 12 steps as a warm-up phase for all models. Additionally, we keep dense attention in the first DiT block to maintain quality. We measure all the latencies with a single NVIDIA H100 GPU.
For longer-video generation, we fine-tune the model with videos that are 2∼4× longer than the default length from OpenVid-1M Nan et al. (2025). Specifically, we sample 2k top-scoring videos in aesthetic and motion scores for each extended length. We use 8 H100 GPUs for training, which takes around 16∼21 hours for HunyuanVideo, 8∼17 hours for Mochi 1, and 15 hours for Wan 2.1. Inference latency for Wan 2.1 is measured on a single H100, while HunyuanVideo and Mochi 1 are evaluated using 8 H100s. See Appendix B for more details.
5.2 Main Results
Figure 6: Examples of videos generated by Radial Attention and the original Wan2.1-14B in the default video length. Radial Attention mirrors the video quality of the original model.
Training-free inference acceleration. Table 1 presents a quantitative comparison of Radial Attention against three strong sparse attention baselines on HunyuanVideo Kong et al. (2024) and Wan2.1-14B Wang et al. (2025a) at their default generation lengths. Corresponding visual results are provided in Figure 6 and Appendix C.1. Under the same compute budget (measured in PFLOPs), Radial Attention preserves the video quality of dense attention while consistently outperforming STA and PA on similarity metrics (PSNR, SSIM, LPIPS), and matches the quality of SVG. While PA shares a similar 𝒪(nlogn) complexity with our design, it ignores the spatio-temporal locality inherent in video data, making it suboptimal in practice.
Regarding efficiency, we adopt the same system optimization used in SVG Xi et al. (2025). Specifically, on a single H100, our Radial Attention achieves 1.9× and 1.8× end-to-end speedups for HunyuanVideo and Wan 2.1, respectively, matching the theoretical compute budget savings (1.8× and 1.7× fewer PFLOPs). Although STA yields slightly higher speedup by using FlashAttention-3 (FA-3) Shah et al. (2024), it suffers from noticeably degraded visual quality. Our current implementation uses FA2 Dao (2024). Upgrading to FA3 is orthogonal to our algorithm and is left as future work.
Long video generation. Table 2 provides results for video generation at 2× and 4× the original lengths, with visualizations available in Figure 7 and Appendix C.2. For Wan2.1-14B, only 2× extrapolation is reported due to its significantly higher computational and memory costs. To ensure fairness, all sparse attention baselines use similar sparsity ratios.
Figure 7: Visual comparison of HunyuanVideo with 4× length extension (509 frames). LoRA fine-tuned models using Radial Attention achieve higher vision rewards, outperforming dense attention baselines, while achieving a 3.7× speedup and reducing tuning costs by 4.4×.
When generating longer videos, the original models without further tuning exhibit significant quality degradation, especially for 4× video length extension. While RIFLEx improves performance at 2× length extrapolation, its quality deteriorates beyond that, indicating limited extension capability. Spatial and temporal sparse attentions suffer from limited reception fields; on the other hand, LongLoRA and PA, though with a global reception field, fail to capture spatial-temporal correlations, resulting in degraded quality. Interestingly, PA exhibits a large gain in Vision Reward after fine-tuning, suggesting that its original sparse pattern misaligns with the pre-trained attention distribution. Fine-tuning allows the model to adapt to the imposed attention sparsity, improving alignment and quality. SANA, which replaces softmax attention with linear attention, requires massive retraining and fails under fine-tuning-based video length extension. In contrast, Radial Attention achieves quality on par with LoRA fine-tuned dense attention models. Notably, it even slightly improves the Vision Reward over the pre-trained model at the default video length.
Thanks to 𝒪(nlogn) complexity, Radial Attention delivers substantial inference and training speedups over dense attention, as detailed in Table 2 and Figure 2. For instance, when generating 4× longer videos, we can save up to 4.4× training costs and get up to 3.7× inference speedup.
Compatibility with existing LoRAs. A key advantage of Radial Attention is its seamless compatibility with pre-trained task-specific LoRAs (e.g., artistic style transfer). We observe that Radial Attention is compatible with existing style LoRAs in both the default length settings, and the longer video settings by directly merging the LoRA weights of Radial Attention in long videos and the style LoRAs. Visualizations and further analysis can be found in Appendix C.3.
5.3 Ablation Study & Analyses
Effectiveness of Low-Rank Adaptation. Figure 8(a) compares Vision Reward between full fine-tuning and LoRA as video length increases. For dense attention, LoRA fine-tuning lags behind full fine-tuning until 4× length extension. However, with our proposed Radial Attention, LoRA fine-tuning matches or even outperforms full fine-tuning, suggesting that Radial Attention not only scales better computationally, but also makes the model easier to adapt to longer-video generation.
Figure 8: (a) Radial Attention outperforms dense attention in generation quality. When combined with LoRA, it further improve the quality while significantly reducing training costs. (b) We model the attention decay curves using the exponential function y=exp(−ax+b). It fits the data well, achieving R2>0.985.
Attention error. We evaluate the average attention output Mean Squared Error (MSE, lower is better) of Radial Attention on Wan2.1-14B, comparing it to SVG Xi et al. (2025) and STA Zhang et al. (2025f). Radial Attention achieves an MSE of 3.9×10−3, significantly lower than 4.4×10−3 for SVG and 1.5×10−2 for STA, demonstrating the effectiveness of our mask in preserving attention fidelity.
Regression results. We perform regression analysis using the model y=exp(−ax+b) to fit the average attention decay curves in Figure 4. As illustrated in Figure 8, the fitted curves achieve an R2 value of over 0.985, indicating that the exponential functions can well model the decay.
More ablations on Radial Attention design choice. Please refer to Appendix D for more details.
6 Conclusion & Discussion
In this work, we propose Radial Attention, an 𝒪(nlogn) sparse attention for efficient long video generation. We observe Spatiotemporal Energy Decay in video diffusion models, which motivates a unified attention pattern with sub-quadratic complexity. At default video length, Radial Attention achieves up to a 1.9× speedup with high fidelity. For videos up to 4× longer, Radial Attention preserves quality and delivers up to 4.4× and 3.7× speedups in training and inference, respectively, with minimal LoRA fine-tuning. This work contributes toward scalable, high-quality video generation and offers a foundation for efficient long-range attention in broader sequence modeling tasks.
Limitations. The assumption of exponential decay for attention scores (Equation 3) simplifies the complex spatiotemporal dependencies in natural video data. While aiding theoretical analysis, future work could improve efficiency and performance by more deeply understanding and modeling the underlying data structure. As shown in Equation 6, our method still has quadratic complexity with respect to resolution. Future work should explore more efficient attention mechanisms and pre-training strategies, as in NSA Yuan et al. (2025) and MoBA Lu et al. (2025), to better support long, high-resolution videos.
Acknowledgments
We thank MIT-IBM Watson AI Lab, National Science Foundation, Hyundai, and Amazon for supporting this research.
Changelog
V1 Initial preprint release.
V2 NeurIPS 2025 camera-ready version. Compress the paper to 10 pages, and rearrange the sections.
References
[1] W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. (2024) Hunyuanvideo: a systematic framework for large video generative models. arXiv preprint arXiv:2412.03603. Cited by: Appendix B, §C.1, §C.2, Appendix F, Figure 1, Figure 1, §1, §1, §2, §3, §3, §4.1, §5.1, §5.2.
[2] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. NeurIPS. Cited by: §1, §2.
[3] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In CVPR, Cited by: §1, §2, §3.
[4] OpenAI (2024) Sora: video generation models as world simulators. External Links: Link Cited by: §1, §2.
[5] Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, et al. (2024) Cogvideox: text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072. Cited by: §1, §2, §3.
[6] Y. HaCohen, N. Chiprut, B. Brazowski, D. Shalem, D. Moshe, E. Richardson, E. Levin, G. Shiran, N. Zabari, O. Gordon, P. Panet, S. Weissbuch, V. Kulikov, Y. Bitterman, Z. Melumian, and O. Bibi (2024) LTX-video: realtime video latent diffusion. arXiv preprint arXiv:2501.00103. Cited by: §1, §2.
[7] A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, J. Wang, J. Zhang, J. Zhou, J. Wang, J. Chen, K. Zhu, K. Zhao, K. Yan, L. Huang, M. Feng, N. Zhang, P. Li, P. Wu, R. Chu, R. Feng, S. Zhang, S. Sun, T. Fang, T. Wang, T. Gui, T. Weng, T. Shen, W. Lin, W. Wang, W. Wang, W. Zhou, W. Wang, W. Shen, W. Yu, X. Shi, X. Huang, X. Xu, Y. Kou, Y. Lv, Y. Li, Y. Liu, Y. Wang, Y. Zhang, Y. Huang, Y. Li, Y. Wu, Y. Liu, Y. Pan, Y. Zheng, Y. Hong, Y. Shi, Y. Feng, Z. Jiang, Z. Han, Z. Wu, and Z. Liu (2025) Wan: open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314. Cited by: Appendix B, §C.1, §C.2, Appendix F, §1, §1, §2, §3, §5.1, §5.2.
[8] H. Xi, S. Yang, Y. Zhao, C. Xu, M. Li, X. Li, Y. Lin, H. Cai, J. Zhang, D. Li, et al. (2025) Sparse videogen: accelerating video diffusion transformers with spatial-temporal sparsity. In ICML, Cited by: §D.1, Appendix F, Figure 3, Figure 3, §1, §1, §2, §3, §4.1, §4.2, §4.2, §4.2, 1st item, §5.2, §5.3.
[9] H. Wang, C. Ma, Y. Liu, J. Hou, T. Xu, J. Wang, F. Juefei-Xu, Y. Luo, P. Zhang, T. Hou, et al. (2025) LinGen: towards high-resolution minute-length text-to-video generation with linear computational complexity. In CVPR, Cited by: §1, §2.
[10] E. Xie, J. Chen, J. Chen, H. Cai, H. Tang, Y. Lin, Z. Zhang, M. Li, L. Zhu, Y. Lu, et al. (2025) Sana: efficient high-resolution image synthesis with linear diffusion transformers. ICLR. Cited by: §1, §2, §2, 6th item.
[11] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) LoRA: low-rank adaptation of large language models. In ICLR, Cited by: §1, §2, §4.3.
[12] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach (2024) SDXL: improving latent diffusion models for high-resolution image synthesis. In ICLR, Cited by: §2.
[13] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet (2022) Video diffusion models. NeurIPS. Cited by: §2.
[14] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis (2023) Align your latents: high-resolution video synthesis with latent diffusion models. In CVPR, Cited by: §2.
[15] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al. (2023) Stable video diffusion: scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127. Cited by: §2.
[16] Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang, Y. Qiao, M. Agrawala, D. Lin, and B. Dai (2024) AnimateDiff: animate your personalized text-to-image diffusion models without specific tuning. In ICLR, Cited by: §2.
[17] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241. Cited by: §2.
[18] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In ICCV, Cited by: §2.
[19] X. Ma, Y. Wang, G. Jia, X. Chen, Z. Liu, Y. Li, C. Chen, and Y. Qiao (2024) Latte: latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048. Cited by: §2.
[20] B. Lin, Y. Ge, X. Cheng, Z. Li, B. Zhu, S. Wang, X. He, Y. Ye, S. Yuan, L. Chen, et al. (2024) Open-sora plan: open-source large video generation model. arXiv preprint arXiv:2412.00131. Cited by: §2.
[21] X. Peng, Z. Zheng, C. Shen, T. Young, X. Guo, B. Wang, H. Xu, H. Liu, M. Jiang, W. Li, Y. Wang, A. Ye, G. Ren, Q. Ma, W. Liang, X. Lian, X. Wu, Y. Zhong, Z. Li, C. Gong, G. Lei, L. Cheng, L. Zhang, M. Li, R. Zhang, S. Hu, S. Huang, X. Wang, Y. Zhao, Y. Wang, Z. Wei, and Y. You (2025) Open-sora 2.0: training a commercial-level video generation model in $200k. arXiv preprint arXiv:2503.09642. Cited by: §2.
[22] G. Team (2024) Mochi 1. GitHub. Note: https://github.com/genmoai/models Cited by: Appendix B, §C.2, Appendix F, §2, §3, §5.1.
[23] T. Yin, Q. Zhang, R. Zhang, W. T. Freeman, F. Durand, E. Shechtman, and X. Huang (2025) From slow bidirectional to fast autoregressive video diffusion models. CVPR. Cited by: §2.
[24] J. Li, W. Feng, T. Fu, X. Wang, S. Basu, W. Chen, and W. Y. Wang (2024) T2V-turbo: breaking the quality bottleneck of video consistency model with mixed reward feedback. In NeurIPS, Cited by: §2.
[25] X. Ma, G. Fang, and X. Wang (2024) DeepCache: accelerating diffusion models for free. In CVPR, Cited by: §2.
[26] F. Liu, S. Zhang, X. Wang, Y. Wei, H. Qiu, Y. Zhao, Y. Zhang, Q. Ye, and F. Wan (2025) Timestep embedding tells: it’s time to cache for video diffusion model. In CVPR, Cited by: §2.
[27] X. Li, Y. Liu, L. Lian, H. Yang, Z. Dong, D. Kang, S. Zhang, and K. Keutzer (2023) Q-diffusion: quantizing diffusion models. In ICCV, Cited by: §2.
[28] T. Zhao, T. Fang, E. Liu, W. Rui, W. Soedarmadji, S. Li, Z. Lin, G. Dai, S. Yan, H. Yang, et al. (2025) ViDiT-q: efficient and accurate quantization of diffusion transformers for image and video generation. In ICLR, Cited by: §2.
[29] M. Li, Y. Lin, Z. Zhang, T. Cai, J. Guo, X. Li, E. Xie, C. Meng, J. Zhu, and S. Han (2025) SVDQuant: absorbing outliers by low-rank component for 4-bit diffusion models. In ICLR, Cited by: §2.
[30] M. Li, T. Cai, J. Cao, Q. Zhang, H. Cai, J. Bai, Y. Jia, M. Liu, K. Li, and S. Han (2024) Distrifusion: distributed parallel inference for high-resolution diffusion models. In CVPR, Cited by: §2.
[31] J. Fang, J. Pan, J. Wang, A. Li, and X. Sun (2024) PipeFusion: patch-level pipeline parallelism for diffusion transformers inference. arXiv preprint arXiv:2405.14430. Cited by: §2.
[32] J. Fang, J. Pan, X. Sun, A. Li, and J. Wang (2024) XDiT: an inference engine for diffusion transformers (dits) with massive parallelism. arXiv preprint arXiv:2411.01738. Cited by: §2.
[33] J. Zhang, J. Wei, P. Zhang, J. Zhu, and J. Chen (2025) SageAttention: accurate 8-bit attention for plug-and-play inference acceleration. In ICLR, Cited by: §2.
[34] J. Zhang, H. Huang, P. Zhang, J. Wei, J. Zhu, and J. Chen (2025) Sageattention2: efficient attention with thorough outlier smoothing and per-thread int4 quantization. In ICML, Cited by: §2.
[35] J. Zhang, J. Wei, P. Zhang, X. Xu, H. Huang, H. Wang, K. Jiang, J. Zhu, and J. Chen (2025) SageAttention3: microscaling fp4 attention for inference and an exploration of 8-bit training. arXiv preprint arXiv:2505.11594. Cited by: §2.
[36] J. Zhang, X. Xu, J. Wei, H. Huang, P. Zhang, C. Xiang, J. Zhu, and J. Chen (2025) SageAttention2++: a more efficient implementation of sageattention2. arXiv preprint arXiv:2505.21136. Cited by: §2.
[37] J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao (2024) Flashattention-3: fast and accurate attention with asynchrony and low-precision. NeurIPS. Cited by: §2, §5.2.
[38] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis (2024) Efficient streaming language models with attention sinks. In ICLR, Cited by: §2, §4.2.
[39] Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia (2024) LongLoRA: efficient fine-tuning of long-context large language models. In ICLR, Cited by: §2, §4.3, 5th item.
[40] H. Jiang, Y. Li, C. Zhang, Q. Wu, X. Luo, S. Ahn, Z. Han, A. Abdi, D. Li, C. Lin, et al. (2024) Minference 1.0: accelerating pre-filling for long-context llms via dynamic sparse attention. NeurIPS. Cited by: §2, §4.2.
[41] J. Zhang, C. Xiang, H. Huang, J. Wei, H. Xi, J. Zhu, and J. Chen (2025) Spargeattn: accurate sparse attention accelerating any model inference. In ICML, Cited by: §2.
[42] J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han (2024) QUEST: query-aware sparsity for efficient long-context llm inference. In ICML, Cited by: §2.
[43] R. Xu, G. Xiao, H. Huang, J. Guo, and S. Han (2025) Xattention: block sparse attention with antidiagonal scoring. In ICML, Cited by: §2, §4.2.
[44] S. Yang, J. Guo, H. Tang, Q. Hu, G. Xiao, J. Tang, Y. Lin, Z. Liu, Y. Lu, and S. Han (2025) Lserve: efficient long-sequence llm serving with unified sparse attention. In MLSys, Cited by: §2, §4.2.
[45] L. Chen, D. Xu, C. An, X. Wang, Y. Zhang, J. Chen, Z. Liang, F. Wei, J. Liang, Y. Xiao, et al. (2025) PowerAttention: exponentially scaling of receptive fields for effective sparse attention. arXiv preprint arXiv:2503.03588. Cited by: §2, 4th item, Table 2, Table 2, Table 2, Table 2.
[46] G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y. Fu, and S. Han (2025) DuoAttention: efficient long-context llm inference with retrieval and streaming heads. In ICLR, Cited by: §2.
[47] P. Zhang, Y. Chen, R. Su, H. Ding, I. Stoica, Z. Liu, and H. Zhang (2025) Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04507. Cited by: §C.1, Appendix F, §2, §3, 3rd item, §5.3.
[48] Y. Xia, S. Ling, F. Fu, Y. Wang, H. Li, X. Xiao, and B. Cui (2025) Training-free and adaptive sparse attention for efficient long video generation. arXiv preprint arXiv:2502.21079. Cited by: §2.
[49] X. Tan, Y. Chen, Y. Jiang, X. Chen, K. Yan, N. Duan, Y. Zhu, D. Jiang, and H. Xu (2025) DSV: exploiting dynamic sparsity to accelerate large-scale video dit training. arXiv preprint arXiv:2502.07590. Cited by: §2.
[50] M. Zhao, G. He, Y. Chen, H. Zhu, C. Li, and J. Zhu (2025) RIFLEx: a free lunch for length extrapolation in video diffusion transformers. In ICML, Cited by: §2, §4.3, 7th item.
[51] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu (2024) Roformer: enhanced transformer with rotary position embedding. Neurocomputing. Cited by: §2, 7th item.
[52] K. Dalal, D. Koceja, G. Hussein, J. Xu, Y. Zhao, Y. Song, S. Han, K. C. Cheung, J. Kautz, C. Guestrin, et al. (2025) One-minute video generation with test-time training. arXiv preprint arXiv:2504.05298. Cited by: §2.
[53] L. Zhang and M. Agrawala (2025) Packing input frame context in next-frame prediction models for video generation. arXiv preprint arXiv:2504.12626. Cited by: §2.
[54] S. Liu, W. Yu, Z. Tan, and X. Wang (2024) LinFusion: 1 gpu, 1 minute, 16k image. arXiv preprint arXiv:2409.02097. Cited by: §2.
[55] D. Han, X. Pan, Y. Han, S. Song, and G. Huang (2023) Flatten transformer: vision transformer using focused linear attention. In ICCV, Cited by: §2.
[56] S. Liu, Z. Tan, and X. Wang (2024) CLEAR: conv-like linearization revs pre-trained diffusion transformers up. arXiv preprint arXiv:2412.16112. Cited by: §2.
[57] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret (2020) Transformers are rnns: fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine Learning Research, Vol. 119, pp. 5156–5165. External Links: Link Cited by: §2.
[58] S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim (2024) Gated linear attention transformers with hardware-efficient training. In Forty-first International Conference on Machine Learning, External Links: Link Cited by: §2.
[59] T. Dao and A. Gu (2024) Transformers are SSMs: generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, External Links: Link Cited by: §2.
[60] H. Cai, J. Li, M. Hu, C. Gan, and S. Han (2023) Efficientvit: lightweight multi-scale attention for high-resolution dense prediction. In ICCV, Cited by: §2.
[61] N. Kitaev, L. Kaiser, and A. Levskaya (2020) Reformer: the efficient transformer. In International Conference on Learning Representations, External Links: Link Cited by: §2.
[62] Z. Zhu and R. Soricut (2021) H-transformer-1d: fast one-dimensional hierarchical attention for sequences. External Links: 2107.11906, Link Cited by: §2.
[63] Z. Zeng, S. Pal, J. Kline, G. M. Fung, and V. Singh (2022) Multi resolution analysis (mra) for approximate self-attention. External Links: 2207.10284, Link Cited by: §2.
[64] Y. Kang, G. Tran, and H. D. Sterck (2024) Fast multipole attention: a divide-and-conquer attention mechanism for long sequences. External Links: 2310.11960, Link Cited by: §2.
[65] S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y. Wang, and X. Yan (2019) Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems 32. Cited by: §2.
[66] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré (2022) FlashAttention: fast and memory-efficient exact attention with IO-awareness. In NeurIPS, Cited by: §3, §4.2.
[67] T. Dao (2024) FlashAttention-2: faster attention with better parallelism and work partitioning. In ICLR, Cited by: §3, §5.1, §5.2.
[68] M. Li, J. Lin, C. Meng, S. Ermon, S. Han, and J. Zhu (2022) Efficient spatially sparse inference for conditional gans and diffusion models. In NeurIPS, Cited by: §4.2.
[69] J. Xu, Y. Huang, J. Cheng, Y. Yang, J. Xu, Y. Wang, W. Duan, S. Yang, Q. Jin, S. Li, et al. (2024) Visionreward: fine-grained multi-dimensional human preference learning for image and video generation. arXiv preprint arXiv:2412.21059. Cited by: §C.2, §5.1.
[70] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, Cited by: §5.1.
[71] D. Zheng, Z. Huang, H. Liu, K. Zou, Y. He, F. Zhang, Y. Zhang, J. He, W. Zheng, Y. Qiao, and Z. Liu (2025) VBench-2.0: advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2311.17982. Cited by: §5.1.
[72] Z. Ye, L. Chen, R. Lai, W. Lin, Y. Zhang, S. Wang, T. Chen, B. Kasikci, V. Grover, A. Krishnamurthy, and L. Ceze (2025) FlashInfer: efficient and customizable attention engine for llm inference serving. In MLSys, Cited by: §5.1.
[73] J. Guo, H. Tang, S. Yang, Z. Zhang, Z. Liu, and S. Han (2024) Block Sparse Attention. GitHub. Note: https://github.com/mit-han-lab/Block-Sparse-Attention Cited by: §5.1.
[74] K. Nan, R. Xie, P. Zhou, T. Fan, Z. Yang, Z. Chen, X. Li, J. Yang, and Y. Tai (2025) OpenVid-1m: a large-scale high-quality dataset for text-to-video generation. In ICLR, Cited by: §5.1.
[75] J. Yuan, H. Gao, D. Dai, J. Luo, L. Zhao, Z. Zhang, Z. Xie, Y. Wei, L. Wang, Z. Xiao, et al. (2025) Native sparse attention: hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089. Cited by: §6.
[76] E. Lu, Z. Jiang, J. Liu, Y. Du, T. Jiang, C. Hong, S. Liu, W. He, E. Yuan, Y. Wang, et al. (2025) MoBA: mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13189. Cited by: §6.
Appendix A Derivations and Proofs
A.1 Complexity
In this section, we provide the detailed complexity derivation in Equation 5, which scales as 𝒪(nlogn). Since the computational cost of masked attention is proportional to the number of zeros in the attention mask 𝑴~, we only need to derive an 𝒪(nlogn) upper bound for the latter.
Central band&attention sink. Firstly, recall from Figure 5(a) that we apply dense attention on these frame-to-frame attention blocks within the central band and attention sink. The attention sink refers to the pattern that every token attends to all tokens in the first frame. Using the same notation as in Section 4, where n is the total number of tokens, s is the number of tokens per frame, and f is the number of frames (so n=fs), we define the attention mask for this region as 𝑴~(1)∈{−∞,0}f×f×s×s:
M~i,j,k,l(1)={0,if |i−j|≤1 or j=0−∞.otherwise (8)
Here, M~i,j,k,l(1) indicates whether the k-th token in frame i is allowed to attend to the l-th token in frame j, with 0 denoting allowed attention and −∞ indicating disallowed attention. Since the attention sink spans f blocks and the central band includes at most 3f blocks, the total number of nonzero entries in this region is bounded by:
#zeros in 𝑴~(1)≤4⋅f⋅s2=4s2f. (9)
Bands with diagonal width ≥1. The second part is those bands with diagonal width ≥1, except the central band. The mask for this region can be defined as 𝑴(2)∈{−∞,0}f×f×s×s:
M~i,j,k,l(2)={0,if 2⌊log2max(|i−j|,1)⌋≤s and |k−l|+1≤s2⌊log2max(|i−j|,1)⌋−∞.otherwise (10)
Thus, since there are at most ⌊log2s⌋ bands in this region, the number of zeros in these bands is bounded by:
#zeros in 𝑴~(2) ≤∑r=1⌊log2s⌋2r+1sn⏟area bound for band ±r⋅2/2r⏟compute density bound of band ±r (11)
≤∑r=1⌊log2s⌋2r+2s2f2r (12)
=4s2f⋅⌊log2s⌋. (13)
Bands with diagonal width <1. The last part is those bands with s2⌊log2max(|i−j|,1)⌋<1, where we reduce the frequency of diagonals. The mask for this region 𝑴(3)∈{−∞,0}f×f×s×s is given by:
M~i,j,k,l(3)={0,if |i−j|mod⌈2⌊log2max(|i−j|,1)⌋s⌉=0 and k=l−∞.otherwise (14)
Since there are at most (⌈log2f⌉−1)−(⌊log2s⌋+1) bands satisfying this condition, we have the number of zeros in these bands bounded by:
#zeros in 𝑴~(3) ≤∑r=⌊log2s⌋+1⌈log2f⌉−12⌊log2s⌋+1⏟number of diagonals⋅n⏟area bound of each diagonal (15)
≤(⌊log2f⌋−⌊log2s⌋)4s2f. (16)
Combining Equation 9, Equation 13, and Equation 16 together, we have the aggregate upper bound of the number of zeros in Radial Attention’s mask:
# of zeros in 𝑴~ ≤4s2f⋅⌊log2f⌋≤4s⋅n(log2n−log2s), (17)
which scales 𝒪(nlogn) with the number of frames f for long video generation.
A.2 Error Bound
The design of Radial Attention is inspired by the spatial-temporal structure in video. In this section, we formulate this intuition by theoretically bounding the asymptotic approximation error of Radial Attention with respect to the decay speed of the attention value in the spatial and temporal dimensions.
We focus on bounding the approximation error of a single row of the attention matrix. We fix a reference query token at position k0 of frame i0, and write the unnormalized row of the attention matrix as
aj,l=exp(𝑸i0s+k0𝑲js+l⊤).
where 𝑸i0s+k0 refers to the query vector at position k0 in frame i0, 𝑲js+l refers to the key vector at position l in frame j, and s refers to the number of tokens per frame.
Assumptions
Relative exponential decay. To capture the intuition that the closer frames have a stronger correlation and each token typically attends to tokens in other frames at similar spatial positions, we assume there exist Crel>0 and (α,β)>0 such that
0≤aj,l≤Crele−α|j−i0|−β|l−k0|a0,a0:=ai0,k0>0.
where α characterizes the temporal decay rate and β characterizes the spatial decay rate.
Infinite temporal grid & finite spatial grid. To conduct asymptotic analysis, we let j∈ℤ (temporal) but keep l∈{1,…,s} (spatial). Extending j to ℤ only enlarges the sums we bound.
Notation
Z:=∑j∈ℤ∑l=1saj,l,Zkeep:=∑(j,l):𝑴~i0,j,k0,l=0aj,l,Zout:=Z−Zkeep.
Exact and masked softmax rows: pj,l=aj,l/Z,p~j,l=aj,l𝟏{𝑴~=0}/Zkeep. The total variation error can be calculated as follows by standard algebraic argument,
‖p~−p‖1= 2ZoutZ.
Because a0 itself is in the sum, Z≥a0. Hence
ZoutZ≤Zouta0.
Mask geometry
For a temporal offset Δt:=|j−i0|≥0, define the bandwidth
w(Δt):=s2⌊log2max(Δt,1)⌋∈{1,2,4,…,s}.
The mask keeps a spatial index l iff |l−k0|≤w(Δt) and the frame is one of the sub-sampled frames; otherwise 𝑴~i0,j,k0,l=−∞.
Two kinds of approximation errors, therefore, appear:
(i) Spatial tails inside kept frames
For each Δt, the discarded spatial part satisfies
∑d>w(Δt)e−βd≤e−β(w(Δt)+1)1−e−β.
Because w(Δt)≥s2 when Δt≤s,
T1:=2Crela0∑Δt≥0e−αΔt∑d>w(Δt)e−βd≤4Crela0(1−e−α)(1−e−β)e−β(s2+1).
(ii) Frames skipped by the subsampling rule
For Δt>s, only every K(Δt)=⌈2⌊log2Δt⌋/s⌉ frame is kept; the remainder contributes
T2:=2Crela01+e−β1−e−β∑Δt>se−αΔt≤2Crela0(1+e−β)(1−e−β)(1−e−α)e−α(s+1).
Total variation error
Combine all equations above:
‖p~−p‖1≤Crel[8e−β(s2+1)(1−e−α)(1−e−β)+ 41+e−β1−e−βe−α(s+1)1−e−α]=O(Crele−min{β/2,α}s).
This characterizes how the decay rates affect the approximation error.
Appendix B Additional Implementation Details
In terms of our LoRA fine-tuning for longer-video generation, we fine-tune HunyuanVideo Kong et al. (2024) and Mochi 1 Team (2024) at a global batch size of 1 with sequence parallelism, and train Wan 2.1 Wang et al. (2025a) with a global batch size of 8. All tuning experiments are conducted on 8 H100 GPUs. During training, we keep the first two DiT blocks with dense attention. Since there are 60, 48, and 40 blocks for HunyuanVideo, Mochi 1, and Wan2.1-14B, this only incurs negligible overhead. We train HunyuanVideo for 2× and 4× length video generation for 2400 and 1200 steps, respectively. We train Mochi 1 for 5000 steps for both 2× and 4× length video generation. We train Wan2.1-14B for 2500 steps for 2× length video generation. The LoRA rank is 128 for all training tasks.
Appendix C Visualization of the generated videos
In this section we compare Radial Attention against various baselines in video quality, and list our speedup in both training and inference.
C.1 Default Video Length
We provide a visual comparison between the original dense attention, STA Zhang et al. (2025f), and our Radial Attention on HunyuanVideo Kong et al. (2024) and Wan2.1-14B Wang et al. (2025a). We conduct experiments under 768p, 117 frames settings for HunyuanVideo, and 768p, 69 frames settings for Wan2.1-14B. As shown in Figure A and Figure B, Radial Attention has higher PSNR compared to STA Zhang et al. (2025f), effectively maintaining the high fidelity of the original videos.
C.2 Longer-video Length
We provide a visual comparison between the aforementioned baselines and Radial Attention on HunyuanVideo Kong et al. (2024), Mochi 1 Team (2024), and Wan2.1-14B Wang et al. (2025a). We conduct experiments under the default resolution settings, which are 720p for HunyuanVideo and Wan2.1-14B, and 480p for Mochi 1. Moreover, we generate videos at 4× longer length for HunyuanVideo (21 seconds, 509 frames) and Mochi 1(22 seconds, 667 frames), and 2× longer length for Wan2.1-14B (10 seconds, 161 frames). We use Vision Reward Xu et al. (2024) to evaluate the generated videos. Figure C, Figure D, and Figure E demonstrate that Radial Attention achieves the highest average Vision Reward score compared to the baselines, well preserving the video quality even at longer-video settings.
C.3 LoRA Compatibility Visual Results
As illustrated in Figure F, combining our extended-length LoRA with existing style LoRAs preserves visual quality while enabling longer-video generation. We observe that the content style generated by the merged LoRA exhibits subtle differences from the original LoRAs. This discrepancy is primarily attributed to the relatively small dataset used for training the extended-length LoRA, which may introduce a slight style bias that interacts with the style LoRA. We expect that training the length-extension LoRA on a more comprehensive dataset would help mitigate this issue.
Table A: We compare Radial Attention against another 𝒪(nlogn) attention baseline, Harmonic Series (HS). Radial Attention consistently outperforms it across all metrics.
Model Method PSNR (↑) SSIM (↑) LPIPS (↓) VisionReward (↑)
HunyuanVideo (117 frames) HS 27.0 0.881 0.119 0.136
Ours 27.3 0.886 0.114 0.139
Appendix D Ablations on Initial Dense-Attention Layers and Steps
We provide additional ablation studies to justify our design choices, including how many dense-attention timesteps and blocks we use to deliver the best video quality with the same computation budget, as well as the design of our 𝒪(nlogn) attention mask.
D.1 Ablation on Initial Dense-Attention Steps
For default-length video generation, we follow SVG (Xi et al., 2025) to apply full attention to the first 25% of timesteps (12 steps) as a warmup for all methods. We ablate this setting in Table B. For fair comparison, we match the overall computation of all settings by adjusting the sparsity of our Radial Attention mask. The 12-step warmup achieves the best video quality across all metrics.
Table B: Ablation on the number of initial full-attention (warmup) steps for default-length video generation of Wan2.1-14B model. The 12-step warmup achieves the best performance across all metrics.
Model #Warmup Steps PSNR (↑) SSIM (↑) LPIPS (↓)
Wan2.1-14B (69 frames) 0 12.8 0.486 0.522
4 18.5 0.693 0.267
8 21.7 0.778 0.183
11 23.2 0.813 0.151
12 (Ours) 23.6 0.823 0.146
13 23.5 0.819 0.150
For 4× longer video generation, we apply full attention to the first 2 steps as a warmup during inference. The impact of different warmup steps on HunyuanVideo is shown in Table C. Computation is matched across all configurations. Using 2 warmup steps achieves the highest Vision Reward.
D.2 Ablation on Initial Dense-Attention Layers
To better capture global information, we keep the first two layers as full attention during training. Table D reports results on HunyuanVideo when using 0, 1, 2, or 3 dense layers, under the same computation budget. Our choice of using 2 full-attention layers delivers the best video quality.
Table C: Ablation on the number of warmup steps for 4× longer video generation. Two warmup steps yield the best Vision Reward.
Model #Warmup Steps Vision Reward (↑)
HunyuanVideo (117 frames) 0 0.154
1 0.160
2 (Ours) 0.163
3 0.157
Table D: Ablation on the number of initial full-attention (dense) layers during training. Using two full-attention layers yields the highest Vision Reward.
Model #Dense Layers Vision Reward (↑)
HunyuanVideo (117 frames) 0 0.139
1 0.156
2 (Ours) 0.163
3 0.157
D.3 Comparison with Other 𝒪(nlogn) Sparsity Patterns
We additionally conduct an ablation study to validate the effectiveness of the sparsity pattern in our proposed 𝒪(nlogn) attention mask. Specifically, we compare Radial Attention with the Harmonic Series Decay Attention (HS), which features a computed diagonal width inversely proportional to its distance from the main diagonal. Table A presents quantitative results comparing Radial Attention with HS, demonstrating the superiority of Radial Attention.
Appendix E Broader Impacts
Radial Attention significantly reduces computational costs for video diffusion models, enabling longer-video generation with minimal fine-tuning efforts while maintaining quality. This paves the way for high-quality video creation tools for education and creative arts. Since Radial Attention accelerates self-attention to 𝒪(nlogn) complexity, it can accelerate video diffusion models and decrease energy consumption, leading to greener AI applications. This also helps the popularization of generative models. However, malicious users can misuse our method to create deepfakes and spread misinformation. The technology may also exacerbate the digital divide between those with and without access to the minimal necessary computational resources. To address these concerns, we advocate for responsible deployment, adherence to ethical standards, and the development of effective detection methods. We encourage the research community to continue advancing both efficient generation techniques and safeguards to ensure these powerful tools benefit society while minimizing potential harms. We will explicitly specify the usage permission of our code and models with proper licenses.
Appendix F License
Here, we show all the licenses for our used assets. Wan 2.1 Wang et al. (2025a), Mochi 1 Team (2024), Diffusers, and STA Zhang et al. (2025f) are under Apache-2.0 license. The license of HunyuanVideo Kong et al. (2024) is here. SVG Xi et al. (2025) and OpenVid-1M do not have an explicit license.
Figure A: Comparison of Dense Attention and Radial Attention on HunyuanVideo Text-to-Video generation at the default length (5 seconds, 117 frames, 768p).
Figure B: Comparison of Dense Attention and Radial Attention on Wan2.1-14B Text-to-Video generation at the default length (4 seconds, 69 frames, 768p).
Figure C: Comparison of all baselines and Radial Attention at 4× default length (21 seconds, 509 frames) Text-to-Video video generation from HunyuanVideo. Radial Attention achieves the best Vision Reward score with good visual quality and consistency. In contrast, Original HunyuanVideo and RIFLEx generate blurred videos with poor visual quality. Temporal Head generates distorting figures. Spatial Head, Long LoRA, and PowerAttention generate temporally inconsistent video backgrounds. Dense Attention generates less dynamic videos.
Figure D: Comparison of all baselines and Radial Attention at 4× default length (22 seconds, 667 frames) Text-to-Video video generation from Mochi 1. Radial Attention achieves the highest Vision Reward score because it has excellent visual quality and consistency. In contrast, Original Mochi 1 generates blurred videos with poor visual quality. Spatial Head, Temporal Head, Long LoRA, PowerAttention, and Dense Attention generate videos with either inconsistent backgrounds or inconsistent figures.
Figure E: Comparison of all baselines and Radial Attention at 2× default length (10 seconds, 161 frames) Text-to-Video video generation from Wan2.1-14B. Radial Attention achieves the highest Vision Reward score original Wan2.1-14B generates blurred videos and Dense Attention generates videos with inconsistent figures.
Figure F: Radial Attention LoRA is compatible with existing style LoRAs. On HunyuanVideo, it extends video length by 4× while maintaining a vision reward comparable to that of the original-length LoRA video.
By Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang, Yujun Lin, Lvmin Zhang, Songlin Yang, Jinbo Hu, Kelly Peng, Maneesh Agrawala, Ion Stoica, Kurt Keutzer, Song Han · Text edition under CC BY 4.0 · Original paper
