Between Caution and Code: Why Anthropic’s Co-Founder Left OpenAI and Still Fears the Future

Ben Mann, co-founder of Anthropic and one of the early minds behind OpenAI, has spent years at the frontier of artificial intelligence. But in a recent interview, Mann revealed something few tech leaders openly admit: even he is afraid of what’s coming next.

As generative AI races toward Artificial General Intelligence (AGI), Mann’s thoughtful reflections offer a rare blend of excitement, regret, and existential concern.

Why He Left OpenAI

Mann was part of the original team at OpenAI—before the ChatGPT explosion, before the multibillion-dollar Microsoft deal. He left when he felt OpenAI was drifting from its safety-first mission.

“There was a vibe shift,” he said. “The pressure to ship faster started to override the internal culture of caution.”

His departure wasn’t a condemnation of OpenAI’s talent. In fact, he praises their technical brilliance. But he believed there needed to be an alternative: a lab that kept AI alignment, interpretability, and safety at its core.

That alternative became Anthropic.

“We Are Building Something We Don’t Fully Understand”

Despite his optimism, Mann is clear-eyed about the risks. What keeps him up at night? That we’re building systems with intelligence we can’t fully control, and potentially values we don’t fully align with.

“I’m not scared of what we have now,” he said. “I’m scared of what comes next.”

He stresses that AI alignment—getting machines to behave in ways that truly reflect human values—is still an unsolved problem, and likely will be even at the AGI stage.

Don’t Wait for AGI to Get Serious

Mann warns against focusing only on a hypothetical AGI. He believes the real risk is cumulative deployment—the slow, subtle integration of powerful but unaligned models into society before governance and oversight can catch up.

In his view, the most likely path to harm isn't a rogue superintelligence—it’s the compound impact of unchecked, poorly-understood systems embedded into institutions, infrastructure, and daily life.

“We Need a New Moral Compass for the Machine Age”

Anthropic’s north star is constitutional AI—models trained with explicit ethical principles encoded into their decision-making. But Mann admits the industry still doesn’t know how to guarantee safe outcomes at scale.

What we do know, he says, is that more transparency, slower timelines, and robust collaboration across labs and regulators will be essential.

Because once we build it, there may be no going back.