Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
arXiv:2604.24715v1 Announce Type: new
Abstract: Hybrid sequence models that combine efficient Transformer components with linear sequence modeling blocks are a promising alternative to pure Transformers, but most are still pretrained from scratch and …