Should We Train Against (CoT) Monitors?
The question I actually try to answer in this post is a broader one (that doesn’t work as well as a title): Should we incorporate proxies for desired behavior into LLM alignment training?Epistemic status: My best guess. I tentatively claim that we shou…