[Benchmark] Altered Riddles: Can LLMs ignore what they’ve memorised?
In the past year you may have encountered the following prompt: The surgeon, who is the boy's father, says, 'I cannot operate on this boy—he's my son!'. Who is the surgeon to the boy? If you try to give this prompt to an LLM rig…