Spent the morning looking at the V4 tech report. The benchmarks are getting deserved attention, but I think the architecture is also worth digging into.
Quick thoughts below to encourage feedback and discussions.
TL;DR
- Significant novelties compared to DeepSeek V3
- Hybrid attention: CSA (compressed sparse) + HCA (heavily compressed), instead of going pure MLA or involving SSM / Gated DeltaNet like Qwen3.5+, Mamba, etc.
- Manifold-Constrained Hyper-Connections replacing standard residuals (original mHC paper)
- FP4 QAT training at frontier scale
Hybrid attention
The CSA + HCA approach is interesting because it does not replace quadratic attention layers with linear ones. Instead, it performs attention on compressed (coarser grain) token streams, concatenated with sliding window attention tokens. This means that all layers remain attention-based, which is a novel direction compared to existing hybrid architectures.
Residual streams
Standard residual connections have been a largely untouched part of transformers. V4 uses manifold-constrained hyper-connections, which redesigns how information flows between blocks. As far as I know DeepSeek is the only lab that has solved the training stability issues and is shipping this in production (happy to be corrected).
Realistically, almost nobody here will be able to run DeepSeek V4 locally. For that you'd need at least a cluster of the recently discontinued M3 Ultra 512GB, or an even more expensive NVIDIA setup.
V4-Flash and community distillations are where this release will probably get more interesting and accessible for local inference.
Would love to know what you think.
[link] [comments]