On “the adults in the room”.
A foundational rationalist principle is nihil supernum – no father, no mother, only nothingness above. There is no one we can really count on, in the end; we must take final responsibility for our own decisions.
It is perhaps fitting, then, that in 2026, the loose collection of people and groups that make up current EA/AI Safety movements recognizes no person or group as its leader. We all saw the fraud of SBF and the moral complicity of some EA leaders in the whole affair; we would’ve been better off in 2022 to look elsewhere for guidance.
At the same time, it is perhaps unfortunate that no person has stepped up, in order to try and be the recognized leader. Where once the EA movement was arguably headed by people like Peter Singer, Will MacAskill, or Toby Ord, and where once AI safety followed the essays written by thought leaders like Holden Karnofsky or Eliezer Yudkowsky, today there exist no such recognized moral authorities. Some may call this a retreat from undeserved respect; others may argue that this is an abdication of undesired responsibilities. As the joke goes, “EA” now stands for “EA-adjacent”; it seems every group is around the EA movement, but few are of it.
Unfortunately, in the absence of formal leadership structures, the alternative is not a group of free-thinking individuals with no coordination mechanisms; informal leadership structures take hold. People look to leaders for many reasons – for coordination, for overall direction, for moral clarity, for advice, for someone to take final responsibility – and the absence of formal leadership does not mean these reasons go away.
The influence that Open Philanthropy/Coefficient Giving have over the ecosystem has long been commented on. Indeed, they are correct that they did not ask for this role and have never claimed to have this role; and they are also correct that today they probably should not serve as this role. But being more than 50% of the funding for the ecosystem inevitably gives one some position of power over it.
Similarly, Anthropic also has outsized influence over the ecosystem. Anthropic’s public communication shapes what many people think about AI Safety. Research that Anthropic does or endorses becomes popular. The short timelines held by Anthropic people are incredibly influential. Working at Anthropic – or better yet, turning down an Anthropic job offer to work on something else – is perceived by many as a badge of legitimacy.
A final power group that people point to is the Constellation network. Constellation is a research center that, as an organization, does not claim a position of power or influence. Yet for many people I’ve spoken to, the Constellation crowd – Redwood Research, METR, Anthropic, the Open Phil people – seem to control much of what is considered fashionable and who are considered legitimate. Speaking as someone embedded in this network, I've seen the influence and information one gets by simply talking to right people at lunch, even though Constellation has never claimed a position of power (it started as a group of friends working in the same office)
To paraphrase Jo Freeman, who wrote about similar problems 50 years ago in the context of a different movement, the reason these informal leadership structures are problematic is not that they exist: some structure inevitably does.
The reason is not that the informal leaders and elites did not deserve their position: Open Phil really funds a lot of good research and deserves much credit for seeding many great organizations; leadership at Anthropic were right about many issues in AI, and has built a massive company using their insights; and a big part of why the people at Redwood or METR have influence is precisely because they’ve done great research and put out influential pieces.
The reason is also not that the informal elites are a conspiracy out to manipulate the group: informal leadership tends to look like friendship networks among people with positions of power, with each person honestly providing their thoughts on what to do next.
The problem with informal leadership is two fold: firstly, that it ensures that the leadership structures are covert and unaccountable. Secondly, it means that the press and public invent “stars” to serve as spokespeople for the community, many who never wanted to speak for the community, and whose spokesperson roles quickly become resented by both themselves and the broader community. In either case, the issue is that the community is ceding power and influence to small groups of people in ways that it cannot then revoke (because it was never granted in the first place).
We see both of these happen today in EA and in AI Safety. The lack of accountability of these organizations to the EA/AIS movements is obvious; they are not of the movement, so why should they feel responsibility for what happens to the community as a whole, and why should they be responsive to the beliefs and desires of those they never chose to represent?
The same goes for becoming appointed as representatives. I imagine many employees at Anthropic resent the fact they (as employees of an independent commercial entity) are unjustly held accountable for the actions of “doomers” calling for much stronger AI regulation. And I imagine OP leadership feels much resentment about becoming inexplicably tied to Effective Altruism as a brand.
This is likely going to get worse, before it gets better. As people have recently been writing about, it seems likely that Anthropic is going to IPO in the coming year, injecting an incredibly large sum of money into the ecosystem, and inevitably shaping it by sheer amount of resources.
I think people are correct that part of the answer is to increase the amount of grantmaking capacity. Yes, if we’re going to triple the amount of dollars invested in AI Safety nonprofits, we definitely need more people to investigate grants and do their due diligence. The alternative is to lower our standards in one way or another, and end up greatly diluting the quality of AI Safety work.
But I think part of the answer is that AI Safety, and perhaps the EA movement as a whole, desperately needs explicit, formal leadership structures. Without a group of people to provide overall direction, we will likely end up either with a greatly expanded version of our current informal leadership structures; even more power, even more influence, but with no one to point to who is in charge of making it all go well.
Or worse yet, in the absence of good formal leadership, we might end up with bad formal leadership. We might again end up with someone ill-intentioned or morally dubious like SBF serving as the part of the leader. Perhaps they are merely power hungry and content to be in control, but perhaps they may be corrupt and nefarious. Perhaps, like SBF, they might drive the reputation of the community (as well as themselves) off yet another cliff.
Nihil supernum is a worthy epistemic or moral principle: each of us does have final responsibility for our own beliefs and for our own actions. But it is not an organizational principle for a community. No group of people as large and diverse as the AI Safety movement will truly operate with only nothingness above.
Discuss