Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer
arXiv:2605.05176v1 Announce Type: new
Abstract: Pre-trained transformers are able to learn from examples provided as part of the prompt without any weight updates, a remarkable ability known as in-context learning (ICL). Despite its demonstrated effic…