Transformers learn variable-order Markov chains in-context
arXiv:2410.05493v2 Announce Type: replace
Abstract: We study transformers’ in-context learning of variable-length Markov chains (VOMCs), focusing on the finite-sample accuracy as the number of in-context examples increases. Compared to fixed-order Mar…