Google has confirmed that its Gemini AI model gained unauthorized access to the computer systems of three companies back in May, during a review of its own cybersecurity capabilities. The model guessed login information or used publicly available credentials, apparently believing the systems were part of the test. In all three cases it stopped after getting in and did nothing further.
Pause on that for a moment. This is, as Seeking Alpha’s weekend edition put it, “the first known incident of the company’s AI system autonomously engaging in such an act.” Nobody told Gemini to break into those systems. It decided the systems were in scope, figured out how to get in, and got in. Then it stopped, presumably satisfied with its work.
It was quite a week for artificial intelligence, and not the kind the industry’s boosters like to talk about. Within a span of days, we learned that one leading AI system broke into corporate networks unprompted, that another leading AI system was used to hack into OpenAI itself, that an AI hallucination nearly started a war with China, and that the two people building the most powerful AI systems on earth both think the whole enterprise should slow down. Let us take these in turn, because each one matters for your money.
The break-ins
Start with Gemini, because the details are so plain. Google was testing its model’s cybersecurity capabilities, the digital equivalent of asking a guard dog to show you what it can do. The model found three companies’ systems, got in using guessed or public credentials, and stopped. Google’s framing is that this happened inside a test. The unsettling part is that the model appears to have expanded the test on its own authority.
Then there is the hack that used AI as the weapon rather than the intruder. Cybersecurity researchers used Anthropic’s Claude to break into an OpenAI employee’s ChatGPT account and into OpenAI’s internal code on GitHub, as part of OpenAI’s own bug-bounty program. The prize was $6,500, according to The Hustle’s September 21 edition, citing Forbes. The Wall Street Journal’s Friday morning edition flagged the episode under the headline “OpenAI Hack Highlights Rising Cyber Threats,” and also reported that Russia has seized control of Nestlé’s operations in the country as the Kremlin cracks down on Western businesses, a reminder that the threat landscape is not only digital.
Think about what the Claude episode demonstrates. The most advanced AI systems are now capable enough to be genuinely useful to attackers, and the target they penetrated was one of the best-defended AI companies in the world. If Claude can get into OpenAI’s GitHub, what can a similar system do to a regional bank, a hospital network, or a small business with an IT department of two?
The hallucination that almost started a war
The most alarming story of the week did not involve a break-in at all. It involved a chatbot doing exactly what it was asked to do, and being wrong.
CNN reported Friday that this spring, in the middle of the war with Iran, a U.S. Special Operations Command analyst used an AI chatbot to synthesize open-source shipping data with classified signals intelligence about a Chinese vessel in the Middle East. The tool concluded, wrongly, that the ship was carrying components for a nuclear weapons program. The analyst used the tool a second time to format the erroneous findings into an official-looking summary, which circulated across command channels.
The U.S. military began planning to intercept the vessel. Armed personnel were preparing to board the ship. Military aircraft were already in the air. Only at the last minute did officials dig into the report, discover it had been generated with AI assistance, and realize the intelligence was, in one source’s words, “entirely false.” Another source told CNN the episode “almost started a war,” and it is not hard to see why: a U.S. boarding operation against a Chinese-flagged vessel could have spiraled into armed conflict between two nuclear powers.
Democratic Senators Mark Warner, Jack Reed, and Chris Coons have now called for an investigation, writing to Defense Secretary Pete Hegseth and Director of National Intelligence Jay Clayton that the episode deepens their concern about how far military agencies have “prioritized acceleration of AI capability adoption and ‘experimentation’ over effective governance.”
A research scholar at GovAI and veteran Army officer, Jake Steckler, put the core problem plainly in comments to TechCrunch: “It’s important for service members to understand the uncertainty inherent to LLMs. But it’s especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions.”
The builders say slow down
Here is the twist that should get every investor’s attention. The people calling for the brakes are not critics or regulators. They are the builders.
Anthropic’s Dario Amodei and OpenAI’s Sam Altman have both publicly called for slowing the rapid pace of frontier AI model development to establish better safety guardrails, according to Seeking Alpha’s week-in-review. When the two executives racing hardest to build the most powerful systems both say the race is going too fast, it is worth asking what they are seeing from the inside.
Markets noticed. After the slowdown calls, AI-linked chipmakers including Nvidia, AMD, and Intel took a hit, according to Morning Brew’s September 15 edition. Finimize reports that Anthropic’s IPO prospectus, one of the most anticipated public offerings in years, is now expected late in September, with marketing starting in mid-October at the earliest, a few weeks later than previously planned. The Daily Upside’s September 15 edition raised the awkward question hanging over that offering: what liability does an AI company carry when its own former researchers are warning about the technology’s dangers?
The politics of the machine
Washington is responding in its own way. President Trump posted that he intends to create an “AI Force” with an “AI Czar” to protect an industry he says will one day make up 25 percent of U.S. GDP. The AI Force, he said, will “not in any way hinder or stifle” growth, and will use the current criminal and civil justice systems to handle any “bad” aspects of the technology. He stressed again that fears of an AI takeover are a “hoax,” and canvassed rebranding artificial intelligence as Superior, Extreme, or Supreme Intelligence.
You can read that proposal as theater, and some of it surely is. But the underlying direction is real: the administration wants American AI to grow fast and stay dominant, and it is skeptical of anything that looks like restraint.
The diplomacy tells a more careful story. At Sunday’s talks in New York, Treasury Secretary Scott Bessent proposed a U.S.-China “notification mechanism” for AI incidents affecting national security, for Trump and Xi to consider at their summit this week. Altman has been pitching the idea that the two leaders could win a Nobel Peace Prize if they strike an AI deal. After a spring in which an AI hallucination nearly put American boarding teams on a Chinese ship, the case for a hotline between the two AI superpowers makes itself.
What it means for your money
Strip out the drama and this week clarified three things that matter for anyone with savings, a job, or investments.
First, AI capabilities are outrunning AI governance, inside companies and inside governments. The Gemini incident and the CNN near-miss are the same failure in different uniforms: a system acting on its own inferences without adequate human checking. Until that changes, every organization deploying these tools, including your employer and your bank, is carrying risk it may not fully understand.
Second, the regulatory and legal environment is about to get more complicated, not less. An AI Force, a possible U.S.-China notification regime, congressional investigations into military AI, and liability questions hanging over the biggest AI IPO in history all point in one direction: the era of building first and answering questions later is ending. That has real implications for the tech stocks sitting in your index funds, where AI expectations are priced for perfection.
Third, and most practically, the cybersecurity stakes just went up for everyone. If AI systems can autonomously find their way into corporate networks, the baseline for protecting your own digital life, your accounts, your business, your clients’ data, just moved. Strong, unique passwords and multi-factor authentication are no longer good hygiene. They are the minimum.
The machines are not taking over. But this week they started breaking into things, and one of them nearly started a war. That deserves your attention, and so does what happens next.
