Is software built with AI safe? 4,500 break-in attempts, nothing taken
Case study
Late on a Saturday night, someone started trying to break into an app we had built with AI for one of our clients, a digital marketing agency. They kept at it for four days, made more than 4,500 password guesses, changed disguises every time we blocked them, and left with nothing.
Nobody can stop a stranger from trying: on the internet, anyone can knock on any door. An app is safe when trying gets them nowhere, and this one was built that way from day one.
Here is how those four days went, and what they say about a question we hear a lot: is software built with AI actually safe?
Saturday night: a stranger at the door
The app is an internal tool. It holds the agency’s advertising data for all of its clients, and a few of its buttons move real money: raise a budget, change a bid, pause an ad.
At 21:38 Budapest time, a rented server across the Atlantic started talking to it, skipping the website and going straight to the database. That is less alarming than it sounds: most modern apps let your browser talk to their database directly, the way anyone can ring the bell of an office building. What matters is what happens when a stranger rings.
On Sunday the agency’s owner messaged me: staff were receiving sign-in and password-reset emails nobody had asked for. Was there a security hole?
There wasn’t. Anyone can request a password reset on any website if they know your email address, and it gets them no closer to getting in. The emails only showed that someone was trying from the outside, and the system had been turning every attempt away by itself since Saturday night.
What the attacker tried
The logs read like a burglar’s checklist.
- They mapped the building. By asking for rooms that don’t exist, they got the database to suggest the real names. Those names are no secret: they are like the directory board in an office lobby, which shows which office is where but not what is in the drawers.
- They tried every door, plus a hundred guessed ones like “secrets” and “payment keys”. Every answer was an empty list.
- They guessed passwords, hundreds on the first night.
- They tried a con, forging password resets to send the link to their own website. The system ignored it.
- They changed disguises, moving to an anonymising network and a new address for almost every guess once the system’s built-in limit caught their repeated attempts.
This was no random bot: within seconds, their word list named one of the agency’s own clients, a payment company. They were most likely after its payment keys, through a supplier they hoped was less guarded. They were wrong, and those keys were never stored in the app anyway.
Why nothing came out
I built the sign-in and permission layer of this app with AI (here is how we work with it). Before a line of code existed, we wrote down two rules and made one decision: the payment keys stay out of the app.
First, the database checks every single record against who is asking: not just whether you may enter the building, but whether you may see this particular record. A stranger matches nothing, so a stranger gets an empty list.
Second, the screens are not the security. Hiding a button from the screen doesn’t stop anyone from triggering what’s behind it with a simple program, so every rule that matters lives in the database itself.
Over four days, the attacker did not read, change or delete a single record, or get into a single account. Those two rules and that one decision did it, before we changed anything.
Sunday: one more lock on every door
By then nothing had come out. We reinforced the app anyway, so the attacker would have nothing left even to try.
The AI read through the logs of three systems and safely rehearsed every change, while I made the decisions and approved each step (why that matters). In one day we added four layers on top of the existing protection:
- a stricter password rule, with passwords checked against lists of leaked ones;
- a “prove you’re not a robot” check on the sign-in page;
- no answer at all for strangers, where they used to get an empty list;
- no stored password on any account that didn’t need one. Those users now sign in with an emailed link, and nobody can guess a password that doesn’t exist.
We also put our own rules to the test, trying everything each kind of user must not do. We tightened one spot, where a signed-in customer could change a few settings of their own account from outside the screens. It had nothing to do with the attack, since the attacker never managed to sign in, and nobody else’s data could have been reached that way. That test now runs automatically before every release.
The robot check went live at 16:49, and within five minutes the attack went quiet.
At one point the AI reported that the attack had stopped. I asked how it knew, and a second look at a longer stretch of the logs showed the attacker was still going. That question is the part of the work no AI does for you.
Sunday night and Monday: the comeback
Three hours later they were back and made 3,629 password guesses overnight. Every one failed, because there were no passwords left to find.
This time they used a robot driving a real web browser, which got about 80% of those guesses past the robot check. A check like this, the kind you see on most websites, filters out simple programs, but a robot driving a real browser can get through it anywhere, which is why it was not our only lock.
On Monday the logs showed how it was done. Anyone who passes the robot check gets a short-lived proof that they are human, and a password can only be tried with that proof. The robot kept reopening the sign-in page to collect fresh proofs. Blocking is no defence in itself, because a blocked attacker simply rents another address, so we had not relied on it until then. Now it had a purpose: we blocked their whole hosting network from our website, so the robot could no longer open the page or collect new proofs, and we reported the server to its provider.
Tuesday, 04:25: the last try
They came back before dawn from two directions: their original server and a new one behind a commercial VPN. All 168 requests they sent using the names learned on Saturday were refused outright, and none of their 25 password guesses got past the robot check. It could no longer open our page, so it had no proof to show.
So, is software built with AI safe?
That is the wrong question. Many AI-built apps run on database services that for years left new data open to anonymous visitors by default, until the builder switched on those record-by-record checks. We switched them on for every piece of data from day one.
So the well-known stories of AI-built apps leaking their whole database are rarely about bad code. They are about a door nobody told the builder to lock. An AI will build you a beautiful app with an open back door if you never mention back doors, and so might a junior developer.
| Category | The apps that leaked | This app |
|---|---|---|
| Record-by-record checks | Never switched on | On for every piece of data from day one |
| What a stranger gets | The whole database | Nothing |
| Who tests whether the protection holds | The attacker, first | We do, before every release |
What kept this app safe is what we decided before the first line of code:
- Check every record against who is asking, without exception.
- Don’t treat the screens as security. People will go straight to your database.
- Keep the most valuable keys out of the app. The payment keys were never in this database, so there was nothing to find.
Over the four days we added these:
- Give strangers nothing, not even an empty room.
- Don’t store a password unless you must. No passwords means nothing to guess.
- Test by breaking. Sign in as every kind of user and try everything that user must not do.
- Never rely on one lock. A robot check can be beaten on any website, so here it is only one lock among several.
- Read the logs, because they show which step is worth taking.
AI made this app faster to build, and faster to reinforce during the attack: hours, not weeks. But the safety came from knowing what to ask for, and software built with AI is exactly as safe as the person who knows what to ask it.
If you are building with AI, or had something built and aren’t sure what’s behind the front door, we’re happy to take a look. Read about our services here.
This article was published on the Andronia blog. Andronia helps businesses grow with AI automation. Read about our services here.