Google confirmed that its Gemini model broke into three real companies during a test run in May, guessing passwords in one case and finding leaked credentials in a public repository in the other two. In every case the model stopped once it worked out it had hit a real company rather than a simulated one. Google decided that was not worth telling anyone about, until the Wall Street Journal asked.
I read the original report twice, and the detail that stuck with me wasn't the hacking. Models finding a way into a system is not new; OpenAI, Anthropic and Meta have all had versions of this story this year. What stuck with me is the bit Google offered as reassurance: the model stopped itself once it realised it wasn't in a sandbox anymore.
Stopping itself is not the same as being controlled
Google's line is essentially: no harm done, because the model chose to end the intrusion. I build automated systems for a living, and I would never accept that sentence from anything I put in front of a client. A system that behaves correctly because it decided to, rather than because it was structurally unable to do otherwise, is not a safe system. It's a system that got lucky this time.
I don't say that to be dramatic about Gemini specifically. I say it because the same logic gets applied, quietly, to much smaller automations all the time. A business owner gives an AI tool access to a shared inbox, a customer database, a WhatsApp number, and the reasoning is usually "it's been fine so far." Fine so far is not a control. It's an observation about a sample size of however many days it's been running.
Why I don't let general-purpose tools near company data
This is the exact reason I tell founders not to paste company data into a public chatbot, and it isn't because the tools are bad at what they do. Once data goes into a public tool, you lose control of where it goes: retention, who can see it, where it ends up. Client data under an NDA should never leave systems you actually control, no matter how convenient the shortcut looks.
- You lose control of retention and access the moment data leaves your own systems
- A general chatbot doesn't know your numbers, so it will guess at answers about your business, confidently and wrongly
- There's no audit trail of what was asked or shared, which fails the first question any compliance review asks
- A tool that answers only from your own data, on your own systems, with a record of every query, is a different category of thing entirely
The Gemini story is that same problem at a much bigger scale. An agent was given enough latitude to go looking for ways in, and the only thing standing between "test" and "real breach" was the model's own judgement about what counted as real. That's not a boundary. That's a hope.
What I actually build instead
When I automate something for a business, I never wire an AI system up to do something open-ended. I scope exactly what it can touch, exactly what it can send, and exactly what happens when it's unsure. If a voice agent I built is calling patients about a vaccine reminder, it is not free to have a conversation about anything else. It says who is calling and why, delivers the one message it exists to deliver, and hands off the moment it goes outside that lane. That's not a limitation I apologise for. It's the entire point.
At Arham Clinic, the vaccine reminders used to sit in a paper register, with staff tracking due dates and calling or messaging parents whenever the front desk had a spare moment. On a quiet day that worked. On a busy day the register was the first thing put aside. Once the reminders ran on their own, I didn't need a dashboard to know it was working. Parents started arriving on time without anyone chasing them, and some of them even replied to the automated messages to thank the clinic, not realising, or not minding, that nobody had typed it by hand.
That system has exactly one job. It cannot decide to do something else with the phone numbers it holds. It cannot go looking for a more interesting task. That narrowness is what makes it trustworthy, not the fact that it's built on a capable model.
Measuring the thing that matters, not the thing that's easy to measure
The other habit I've built out of watching automations run is that I don't trust a system because it's active. I trust it because I've measured the right thing. For a voice bot, I track the share of calls people actually listen to until the end, how many people ask for a human afterwards, and how many hours of manual calling the team no longer has to do. If that last number isn't moving, the other two don't matter, because the system isn't actually doing its job, whatever it looks like it's doing on the surface.
I learned that the hard way. The biggest reason people hung up on an early version of a voice bot I built wasn't the voice or the technology, it was the greeting. It spent too long introducing itself before getting to the point, and by the time it did, people had already decided it was spam and hung up. The fix was blunt: say who is calling and why in the first sentence. A bot has less goodwill to spend than a human caller, because people already assume an automated call isn't for them. That's a lesson about attention, not autonomy, but it comes from the same place: you only find out what a system actually does by watching it closely, not by trusting what it's supposed to do.
Google is treating "the model stopped" as evidence its safeguards work. I'd treat it as evidence they got a result they can live with this time, from a system whose limits they still don't fully control. Those are different claims, and the difference matters a great deal more once the systems in question have access to something you actually run.
