Author name: Ruida Zhou, Chao Tian, Suhas Diggavi

An Information-Theoretic Approach to Understanding Transformers’ In-Context Learning of Variable-Order Markov Chains

Ruida Zhou, Chao Tian, Suhas Diggavi / April 1, 2026

arXiv:2410.05493v3 Announce Type: replace
Abstract: We study transformers’ in-context learning of variable-length Markov chains (VOMCs), focusing on the finite-sample accuracy as the number of in-context examples increases. Compared to fixed-order Mar…

cs.IT, cs.LG, math.IT

Transformers learn variable-order Markov chains in-context

Ruida Zhou, Chao Tian, Suhas Diggavi / March 31, 2026

arXiv:2410.05493v2 Announce Type: replace
Abstract: We study transformers’ in-context learning of variable-length Markov chains (VOMCs), focusing on the finite-sample accuracy as the number of in-context examples increases. Compared to fixed-order Mar…