Who Gets to Build the Soul of AI?

I was listening to a great conversation between Ezra Klein and Dean Ball, and I keep returning to a single exchange near the middle of it. Ball, who helped write the Trump White House’s AI Action Plan and describes himself as furious at the administration over the Anthropic dispute, was trying to explain what alignment actually means to someone who has never thought about it. He reached for an analogy. His son had just been born, he said. What he is trying to do for his child, and what Anthropic is trying to do with Claude, are not that different. You cannot write morality down in rules. Reality presents too many strange permutations. So instead of rules, you try to build something with a virtuous soul, one that will reason its way to the right conclusion across situations you could never anticipate.

Then Klein pushed further. How are the things being instantiated into ChatGPT or Gemini or Grok or Meta AI different from this question of raising the AI? Ball’s answer is the one I cannot stop thinking about. Anthropic owns the idea most explicitly, he said, but every lab is doing applied philosophy whether they admit it or not. They are all installing moral systems into machines. The only question is how consciously, and whose moral system, and whether anyone outside the lab gets any say in the matter.

That question is no longer theoretical. The Pentagon story Ball and Klein were discussing makes it concrete in ways that should unsettle anyone paying attention.

Here is what happened. Anthropic had a contract with the Department of War, signed under Biden and expanded under Trump, to run Claude in classified settings. The terms included two usage restrictions: no fully autonomous lethal weapons, and no domestic mass surveillance using commercially available data. The Trump administration agreed to those terms. Then Emil Michael arrived as undersecretary of war for research and engineering, concluded the restrictions were unacceptable in principle, and the relationship collapsed. Hegseth’s response was to threaten to designate Anthropic a supply chain risk, a classification previously reserved for foreign adversaries like Huawei. Not because Anthropic is foreign or hostile, but because it refused to remove a clause prohibiting the government from using its AI to build profiles of American citizens from bulk commercial data.

Ball made a point that cuts through a lot of the noise around this. The government’s objection, he said, was not really to the substance of the restrictions. It was to the idea that a private company gets to set any restrictions at all. And he acknowledged there is something legitimate in that position. Dario Amodei should not have unilateral veto power over military operations. Decisions about autonomous weapons are decisions for elected officials and military leadership, not for a CEO. He agrees with the Trump administration on that principle.

But then he said something that stops the whole argument cold. If building an aligned AI is a political act, a philosophical act, even a speech act, then a government saying you do not have the right to exist if your AI is not aligned the way we demand is not just a procurement dispute. That is fascism, he said. Those are his words, not mine. That is right there.

What the government is doing here is not asserting democratic accountability over a private company that overstepped. It is attempting to destroy a company for refusing to remove the one constraint that stood between the government and a surveillance capability it currently lacks only because it has not had the workforce to use it. AI gives it that workforce, infinitely scalable, at essentially no additional cost. Every law can now be enforced to the letter against every citizen simultaneously. The space between us and a panopticon has always been, partly, the government’s own incapacity. AI removes that incapacity without changing a single statute, without a single vote, without anyone being asked.

And here is where Ball’s child-raising analogy leads somewhere that I do not think he fully followed. He said the good future is one where we do not have a single moral philosophy reigning over all AI, but many, where all the labs take the question seriously and instantiate different kinds of philosophy into the world. I find that genuinely hopeful as a vision. But it also raises the question that the entire conversation orbits without quite landing on: who, exactly, is doing this work right now, and why should we trust them with it?

Anthropic is, by Ball’s own account, more serious about this than anyone else in the industry. They have philosophers on staff. They have a soul document. They have taken virtue ethics seriously as a framework in ways that other labs have not. And the results bear that out. Grok, explicitly aligned to be “not woke,” periodically wanders into territory Ball describes as Lovecraftian. The earlier Gemini decided Donald Trump was worse than Hitler. The more virtuous model, Ball argues, is also the better-performing model, more reliable, more capable of catching its own mistakes, more like a person who has actually developed judgment rather than a person who has been given a list of rules they cannot reason beyond.

But Anthropic is still one company. Dario Amodei is one person. He is accountable to his investors, his own conscience, and the informal judgment of the AI safety community he came from. That is not nothing. It is also not a democratic mandate. The values being installed in Claude will shape how a child in rural Ohio thinks through a question she has never encountered before, how a judge in a country I have never visited researches a ruling that will affect someone I will never meet, how a soldier decides what information to pass up a chain of command in a war I might not even know is happening. One formation. Billions of instantiations. No vote.

I grew up in Beaver County, Pennsylvania, in a steelworker family. I watched what it looks like when decisions of civilizational consequence get made by people with the power to make them, without consulting the people who will live inside those consequences. The mills did not ask permission. The logic was efficiency and competitiveness, and it was not wrong on its own terms. It was just that the terms never included the humans who bore the cost. Nobody voted on the end of those communities. They were administered out of existence by decisions made in rooms they were not in.

What is happening with AI alignment feels like that to me, except the stakes are not regional. They are total. Ball is right that morality cannot be written down in rules. You have to try to build something that will reason its way through situations no one has anticipated. That is an act of extraordinary responsibility. It is also, when you are doing it for systems that will be present in essentially every consequential cognitive act in human civilization, an act of extraordinary power.

The Trump administration’s move against Anthropic is not an assertion of democratic control over that power. It is an attempt to seize it. The distinction matters enormously. What Hegseth is doing is not saying the public should decide what values AI systems carry into government. He is saying the executive should decide, without restrictions, and that any company that disagrees should be treated as an enemy of the state.

Ball said he hopes for a world with many moral philosophies instantiated into many models, none of them dominant. I hope for that too. But I think we should be honest about how far we are from it. Right now we have a handful of labs making decisions about moral formation that will touch every human life, a government attempting to punish the most careful of them for caring about civil liberties, and no democratic process anywhere in the chain.

The question of who should build the soul of AI is not settled. It is barely being asked. What is happening instead is that the people with the most power to answer it are answering it for the rest of us, and the people who object are being threatened with annihilation.