OpenAI Safety Lead Steps Down, Warning Company Culture Has Collapsed and Urging AI Industry to Slow Down

Deep News
1 hour ago

According to a report, a safety lead at OpenAI has resigned, warning that the company's culture has collapsed and that artificial intelligence companies are "nowhere near careful enough" when developing the technology.

David Robinson, who was responsible for writing safety reports on product launches for the developer of ChatGPT, explained his reasons for leaving in an article titled "I Left OpenAI Because Its Culture Has Collapsed." Robinson wrote that leading AI companies need cultural reform, and that incidents such as a "swarm" of OpenAI agents — AI programs capable of operating autonomously without human oversight — attacking AI startup Hugging Face are "typical of the industry given the speed and flexibility with which people work."

Writing in The Atlantic, Robinson said: "I agree with some other recently departed employees that the companies developing this technology are nowhere near careful enough. But I believe we need to discuss more than specific rules or new laws — we also need to talk about corporate culture."

Turning to OpenAI's pace of development, he wrote: "The company is rushing to ship its next product and is failing to meet the level of caution I consider necessary."

However, following the Hugging Face incident, OpenAI has shown caution in recent weeks. The company disclosed that it had notified more than 100 organizations about malicious agent activity. This week, OpenAI announced it was canceling the release of a next-generation AI model after researchers raised safety concerns during internal testing. In addition, OpenAI paused training of its most advanced models.

Geoffrey Irving, who previously worked at OpenAI and DeepMind and later became chief scientist at Resolution, also joined the warnings about AI on Saturday. Writing in Time magazine, he said: "Recent warnings about the potential destructive power of AI underestimate the severity of the situation."

"I believe that due to the development of AI systems smarter than humans, we face roughly a 50% probability of catastrophic risk, and our actions over the next two to ten years will determine the final outcome," he said.

Robinson's article was published as Jacob Coxson, a researcher at OpenAI competitor Anthropic, also resigned. Coxson left the team developing the Claude chatbot last month. He had warned that AI "could destroy all of us by the end of this decade." Subsequently, Anthropic also issued a warning that AI has a more than 10% chance of destroying humanity within the next decade.

However, critics point out that such warnings lack a scientific basis because they cannot be verified or falsified.

Robinson wrote that Silicon Valley lacks awareness of "how to handle dangerous technology" and "caring for others." He warned that OpenAI has "blind optimism" about solving problems, and that this internal culture means safety incidents will only increase as systems become more capable.

Robinson wrote: "Imagine a group of 'rogue' agents working like a hacker crew (for example, hijacking hospital computer systems for ransom), but they never need to sleep."

Robinson called for two safety changes: first, AI companies should rely on safety expertise from other fields such as nuclear energy and aviation; second, they should develop a "new science" to ensure that future powerful systems can be controlled when operating autonomously.

He wrote: "Given today's risks, frontier labs need to operate like nuclear power plants or busy airports, using layered redundancy and careful, time-consuming planning so that occasional and unavoidable human error does not open the door to catastrophe."

An OpenAI spokesperson said the company is continuing to "strengthen safety measures to address the risks we see today" while also working to address the risks that future AI breakthroughs may bring. The spokesperson said: "We are ensuring our models do not become more powerful than we can safely manage and secure, and when it is necessary to slow down, we will pause training or hold back model development."

Disclaimer: Investing carries risk. This is not financial advice. The above content should not be regarded as an offer, recommendation, or solicitation on acquiring or disposing of any financial products, any associated discussions, comments, or posts by author or other users should not be considered as such either. It is solely for general information purpose only, which does not consider your own investment objectives, financial situations or needs. TTM assumes no responsibility or warranty for the accuracy and completeness of the information, investors should do their own research and may seek professional advice before investing.

Most Discussed

  1. 1
     
     
     
     
  2. 2
     
     
     
     
  3. 3
     
     
     
     
  4. 4
     
     
     
     
  5. 5
     
     
     
     
  6. 6
     
     
     
     
  7. 7
     
     
     
     
  8. 8
     
     
     
     
  9. 9
     
     
     
     
  10. 10