Token-Level Entropy Reveals Demographic Disparities in Language Models
arXiv:2501.19337v3 Announce Type: replace
Abstract: We ask whether demographic identity, signaled by a name alone, systematically reshapes the generative distribution of a language model. Measuring full-vocabulary Shannon entropy at temperature zero a…