Nadella: AI Needs Human-Controlled Emergency Brake for Safety

Microsoft Chairman and CEO Satya Nadella has called for advanced artificial intelligence systems to include strict safety controls and an emergency brake that allows humans to pause operations mid-task. He shared the proposal on social media platform X on Saturday.

Nadella argued that developers must surround non-deterministic AI models with reliable operating procedures and independent oversight. He stressed that treating frontier closed and open weight models like insider risks is essential for building a safer architecture. "We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient," he wrote.

The push for tighter oversight comes as experts across the tech sector warn that safety protocols are lagging behind rapid development. An AI researcher recently left Anthropic after accusing the company and rival OpenAI of "gambling with our lives." Around the same time, an alignment specialist at Anthropic noted there is a greater than 10% chance that artificial intelligence could "kill all humans" within the next decade.

Political leaders are taking a different approach to the technology's trajectory. President Donald Trump has repeatedly dismissed extinction scenarios and stressed that the United States must maintain a technological edge over China. To support this goal, Trump recently launched a new AI Force designed to streamline industry operations and remove bad actors. Director of National Intelligence Jay Clayton will lead the initiative.

Nadella outlined a set of observability rules for the new systems, including continuous testing, auditability, and clear incident reporting. He emphasized that safety depends on structural checks rather than blind confidence in the software. "The most trustworthy Super Intelligence system will not be the one with the model we trust most," he wrote. "It will be the one that enables us to trust the model the least."