Author name: Nathan Helm-Burger

I’ve been looking into the Monet/PEER sparse expert papers. I think there’s a lot of potential in these ideas for interpretability-by-design.
Some of what I’ve done so far:

Quantization experiments: PEER can be losslessly distilled to int8 and distil…

Interpretable by Construction: A Research Bet on SAE-like Expert Architectures
The Bet
You can build a language model architecture whose native decomposition is already close to what sparse autoencoder researchers are trying to recover post-hoc: a larg…