Training Transformers in Cosine Coefficient Space
arXiv:2604.04440v1 Announce Type: cross
Abstract: We parameterize the weight matrices of a transformer in the two-dimensional discrete cosine transform (DCT) domain, retaining only the lowest-frequency coefficients. At each forward pass the full weigh…