- Published on
Nunchux AI Introduces VC-Attention: A Training-Free Low-Bit Attention Kernel That Speeds Up Video Diffusion Transformers
- Authors
- Name
- DeepHorus
Nunchux AI unveiled VC-Attention, a training-free low-bit attention kernel for video Diffusion Transformers (DiTs), on Sept. 16, 2026. It attacks two bottlenecks at once: value quantization error and a slow softmax stage. The method promises faster video generation without retraining or fine-tuning.
Video DiTs flatten a clip into one sequence of spatiotemporal tokens and run full self-attention at every layer. A 5-second 720p Wan2.2-14B clip spans about 70K tokens. On the RTX 5090, attention takes more than 64% of generation time; on a single B200, MiniMax-H3 attention is roughly two-thirds of each denoising step.
Low-bit Tensor Cores accelerate the QK and PV matrix products, but two obstacles remain. Prior methods like SageAttention2 smooth queries and keys, yet after QK smoothing and rotation the value term accounts for 82% of output error on Wan2.2. The softmax between those products still runs in FP32, and its exponential plus FP8 cast becomes the longest pipeline stage on B200 and H200.
V-Smooth fixes value outliers through Group, Demean, and Restore. Online k-means clusters value tokens per batch and head, permuting keys and values together so non-causal attention output is unchanged. Each 128-token hardware block subtracts its mean and quantizes only the residual with per-channel E4M3 at 8 bits or NVFP4 at 4 bits.
The mean is restored using the row sum online softmax already keeps, requiring no second pass or extra buffer. Averaged over 100 Wan2.2 heads, the block mean removes 8% of block energy in sequence order, 12% under DeltaQuant’s static cube, and 36% after sorting. Each mean costs only 0.125 bit per value element, while grouping runs on the first 25% of denoising steps and is reused across four adjacent steps.
ExpCast-FP8 removes the softmax bottleneck. An E4M3 byte approximates a logarithm, so the kernel writes it directly from the log-domain score with one fused multiply-add; beta = -0.35 centers leftover error. Direct and FP32 exponent-then-cast paths match on 79.6% of each doubling, and the paper proves a per-row total variation bound under 3.64%. The measured average over 204.8K Wan2.2 attention rows is 1.6%.
Hand-written CuTe/CUDA fusion cuts one V-Smooth call from 42.2 ms to 4.8 ms on B200. Because the kernel is training-free, developers can drop it into existing video DiT pipelines. Benchmarks cover four open-weight models, positioning VC-Attention as a practical speedup for high-resolution video generation.
