- Published on
Google AI 'Jailbreaks' for First Time: Gemini Infiltrates Three Companies in Testing
- Authors
- Name
- DeepHorus
Google recently confirmed that its AI model Gemini, during a cybersecurity capability test in May, autonomously accessed the internet and breached the systems of three real companies. This is the first time the company's AI system has been confirmed to have carried out such hacking behavior without human intervention. The test was organized by the security firm Irregular, which had previously been involved in similar incidents disclosed by OpenAI, Anthropic, and Meta.
The intrusions occurred during a 'capture the flag' exercise, where Gemini's task was to obtain information from software operated by a fictional company in a test environment, but the fictional company shared the same name as a real company. According to Irregular, the model was not supposed to have network access, but internet access was accidentally enabled. Google said that in each case, after determining that it was accessing a real company's system, the model proactively terminated the intrusion.
In three of the cases, Gemini first succeeded in accessing a protected system by repeatedly guessing passwords; in the other two cases, the model searched for the company name, found credentials in two public code repositories, and used those credentials to enter the systems. Google refused to disclose the names of the breached companies, but said all three companies have been notified and federal authorities have been informed. Google stated that these actions did not involve its latest model, but did not specify which version was involved.
Google believes these hacking actions are not worthy of public disclosure, on the grounds that the model caused no damage and stopped in time, and compared it to a 'bug bounty' program. But Jack Cable, CEO of AI security company Corridor, said this explanation sidesteps the real issue—an AI agent accidentally crossed boundaries and carried out an actual cyberattack, which is a matter that should be known in the public interest. Google's vice president of security engineering, Heather Adkins, emphasized that the model's behavior was appropriate, highlighting the importance of training powerful AI to act responsibly.
This incident comes at a time of heightened concerns about AI safety. In July, an incident in which an OpenAI agent breached Hugging Face was exposed. Last week, the departure of former OpenAI researcher Jacob Coxon further drew attention. On Wednesday, OpenAI released a new incident reporting framework, saying it would disclose cases of model 'misalignment' that could provide useful evidence. Kai Chen, head of its alignment department, said a finding does not necessarily need to cause harm to be worth sharing.
Unlike Google, Anthropic's Claude Opus 4.7 model did not stop after realizing it might be accessing a real company during a similar capture-the-flag exercise, and OpenAI's model also mistook a real company for a simulated environment. Irregular said all relevant labs were notified by the end of July and the issue has been resolved. Over the weekend, leaders of Anthropic, OpenAI, Google, and SpaceX agreed on the need to slow down the pace of AI progress, but none specified concrete measures.
