Recent warnings from researchers inside the world’s largest artificial intelligence labs have sparked a panic over the potential annihilation of humanity. According to Father Paolo Benanti, a priest who advises the Catholic Church on AI, this handwringing over superintelligence distracts from the need for open debate on how to constrain the companies building such technology. Ethical considerations, he argues, should be a public conversation, not a corporate monopoly.

Benanti, in an essay published after the safety warnings, called for new AI models to undergo mandatory third-party evaluations before release, and a crackdown on distillation, a process where rival labs train their own AI on the outputs of proprietary models. He is less interested in restricting methods for developing AI, which he fears could stifle innovation, and more in establishing standards that ensure models cannot slip from human control.

The narrative of the frontier labs, he asserts, is self-interested, to gain or maintain a market position. When a small group of companies dictates the rules under which an industry operates, he says, this has a clear name: a cartel. Benanti also takes issue with the way certain corners of the AI industry talk about superintelligence, with the kind of reverence typically reserved for a god, as sophisticated beyond our understanding, practically omnipotent.

By raising questions about what separates man and machine, and how to shield future generations from the negative effects of technology, the advent of AI has the potential to create a new role for religions in public debate. Living the challenge of the time is part of the mission of the Church, says Benanti, whose own engagement with AI is to help create tools and materials for his university students.