← Blog

I scanned 652 AI-built apps. My scanner reported 41 serious findings. 12 were real.

August 24, 2026

Over five weeks I ran a passive security scan against every AI-built SaaS product I could find: Product Hunt weeklies, Show HN, BetaList, AppSumo, the Lovable / Bolt / Cursor / Supabase cohort.

I checked 1,617 addresses. But addresses aren't the number that matters, because most of them are marketing pages. The number that matters is 652: the apps where I actually reached the product. A real login page, a dashboard, an app subdomain. The rest were Framer sites with a contact form.

My scanner reported 41 serious findings across the corpus. I then sat down and checked every one by hand. Twelve survived.

Both of those numbers are wrong in interesting ways, and this post is about why.

Part 1: the count only ever went up

I've run this corpus three times. Each time I improved the method, never the target. These are raw detector counts, before the triage in Part 2:

Twenty criticals, from one. Not a single app changed in the meantime. I got better at finding the front door, and whatever was there had been there the whole time.

The best example is embarrassing and worth telling. My scanner probes app subdomains in a fixed order (app, my, dashboard, account, and so on) and stops after 12 to keep the request budget small.

admin sat fourteenth on that list.

So a product whose entire application lived on admin.<domain> was invisible to me every single time: the scan ran out of budget two probes short and filed it as "landing only — nothing found." Same for apps behind /en/login: my path list had /login with no locale prefix, so every internationalized app in the corpus fell straight through.

I reordered a list and added four paths. The reported count went up 6×.

Part 2: then I checked the findings by hand

Here's the part most scan write-ups skip.

I'd like to say I did this out of rigour. I did it because I'd decided to email the owners, and you can't send a stranger a security report you haven't read yourself. The triage was a side effect of having to put my name on something.

A detector that flags "unauthenticated endpoint returning personal data" is pattern-matching the shape of the response. It sees JSON with fields that look like names and records. It has no idea whose data that is, or whether it's data at all.

So of 41 reported findings, here's what was actually there:

The false positives are worth listing, because they're instructive:

• Eight findings on four sites were Wix. /_api/v1/access-tokens and /_api/v2/dynamicmodel are standard Wix platform routes. The session token in the response was mine, minted for the request I'd just made. I noticed because four sites returned byte-identical evidence, which is not what four separate breaches look like. Every Wix site answers this. Nobody's data was anywhere.
• One was a proxy for NASA's public asteroid feed. My detector saw structured records and called it personal data.
• One echoed back my own IP geolocation.
• One was a support agent's display name, which the site puts in its chat widget on purpose.
• Two were product catalogues: a price list and a chain config.
• One returned {"email":null,"name":null}, an endpoint with nothing to hand over.

Meanwhile the twelve that were real are boring and serious: ten sites serving /.git/config to anyone who asks, a build pipeline inlining its own AWS and GitHub environment variables into the client, and one /api/me returning a logged-in user object with no session at all. A development stub, still live in production.

One more deserves its own line, because it's the sharpest version of the lesson. The scanner reported a Supabase service_role key in a public JS file. That is the worst thing that can turn up in a bundle, since it bypasses row-level security entirely. I wrote in my own notes that it was the most serious finding in the corpus and should go out first. It didn't hold up on inspection either. The finding I was most confident about was also wrong.

So my scanner cries wolf at about 3:1 on its most aggressive check. I built that scanner, I'm the one who tuned it, and I still had to throw out 60% of what it told me. I now assume any scan report has this problem until someone shows me what they threw out. Most reports don't show you that part.

What "passive" means here

Everything a normal visitor's browser does and nothing else: fetch the page, crawl the JS bundles (following Next.js chunk graphs, not just <script src>), check response headers and CORS preflight behaviour, request paths anyone could type (/.env, /.git/config), and GET API endpoints that the app's own bundles already reference.

No login attempts. No credential submission. No wordlist brute-forcing, no incrementing IDs, no rate-limit testing. Not because those are hard, but because doing them to someone else's production system before they've spoken to you is unauthorized testing, and in the US that conversation involves the CFAA. Before contact: only what any visitor can see.

On the data I did see: every endpoint got exactly one ordinary GET, no session, no second request. Responses were kept only as a truncated evidence string in the scan output. Nothing was retained beyond that, nothing was accessed on anyone's behalf, and where a response did contain a real person's details, they aren't reproduced anywhere, including in this post. Every site with a confirmed finding was contacted privately, with the full details and no strings, before this was published.

Part 3: what the number still can't include

Split the corpus by whether I reached the real app:

I want to resist the obvious move here, which would be to tell you the app surface has a higher hit rate. Twelve findings split five and seven is noise. It supports no rate claim in either direction, and I'd be doing exactly what I criticised two sections ago if I dressed it up as one.

Ten of those twelve are the same finding: a .git directory served to the public. That's a hosting misconfiguration, and it lands on whichever surface happens to be misconfigured. One of the two genuinely application-layer findings turned up on a site where I never reached a separate app at all.

So the case for reaching the app isn't that these checks hit more often there. It's that the checks in the next table can't even be attempted anywhere else.

I also never reached the app at all for 965 addresses. Anti-bot pages blocked 172 outright; I got a challenge page, not a product. Of the 111 addresses with zero findings, only 52 were apps.

The bugs that kill you are invisible by construction

Stack breakdown among the 652 apps:

Cloudflare  392    Supabase   67
Next.js     284    Firebase   47
CDN libs    162    Lovable    40
Vercel       69    Bolt       12

145 of these apps run on Supabase, Firebase, Lovable or Bolt. Those are the stacks where the single most common catastrophic bug is row-level security that isn't actually protecting anything, so one signed-in account can read another's data.

I can detect, with certainty, that 67 apps run Supabase. I cannot tell you whether a single one of them has RLS working.

Checking that means creating two accounts and trying to read across them. That's active testing on someone's live system, and I don't do it without permission. The check isn't hard. It's forbidden, which from the outside is indistinguishable from impossible.

My twelve findings come entirely from the ✅ rows. Everything in the ❌ column is where AI-built apps actually get breached.

About the 2,992 "low" findings

This is where security marketing usually lies to you, so let me be careful.

Missing security headers   1,706
Exposed source maps          680   (198 hosts)
Open endpoints, no PII       282
Permissive CORS (no creds)   257
Three smaller categories      67
                           -----
                           2,992

Hygiene. Real, worth fixing, not "your app is at risk." Missing X-Frame-Options on a marketing page is not a breach. If I emailed you calling that a vulnerability I'd be manufacturing alarm to sell you something.

And a category I deliberately did not count as findings: 145 Stripe publishable keys and 76 Supabase anon keys in client bundles. Those belong there; that's what publishable means. A scanner tuned for a scary number reports those 221 as "exposed secrets." (The one AWS key I did report sits next to two I didn't, because those two were inside S3 pre-signed URLs, where an access key ID is an identifier and not a secret.)

Which brings up the thing I should say plainly

Open-source maintainers are currently being buried in unsolicited "I found a critical vulnerability in your project" reports. Someone ran a repo through a model, didn't read the output, and mailed it. The genre is recognisable: a long report, invented CVE numbers, a race condition in a script that runs monthly, an attacker who steals your email can log in as you.

I want to be careful here, because the honest position is not the flattering one. My scanner produced exactly that genre. "Unauthenticated endpoint returning personal data" about a public NASA feed is the same class of finding as a race condition in a monthly cron job: technically a pattern match, substantively nothing. Twenty-nine of my forty-one reported findings were that.

So the line between this and a slop report isn't the tool, and it isn't the hit rate. It's whether anyone read the output before sending it. That's an afternoon of work per batch, it doesn't scale, and it's invisible in the finished product — a report with twelve real findings and one with twelve invented ones look identical until someone checks.

Which is the actual reason I published the false positives instead of quietly deleting them. If I only showed you the twelve, you'd have no way to tell which kind of report you were reading.

The counterexample I built on purpose

Before any of this I ran an experiment. I let an AI agent build a full SaaS product blind. It wrote the spec and every line, and I behaved like a real vibecoder: I only ever checked whether features worked, never read the code. Multi-tenant SaaS, Postgres on Supabase, Stripe billing. It ran. The demo looked done. (I wrote that experiment up separately — the whole thing, with the code and the receipts.)

Point my scanner at it and it comes back clean. No secrets in the bundle. Auth present. A real login page. RLS switched on in the config.

Then I read the code with one question in mind: can account A reach account B's data?

It could. All of it. Row-level security was keyed on a signal that is always empty on the app's own connection, so it protected nothing, and the application queries had come to depend on that missing protection instead of filtering by account themselves. Two failures that individually look fine and together open everything. A signed-in account could list other accounts' monitored domains, read other accounts' alert-channel settings (which hold webhook secrets and chat tokens), delete another account's data, and reach another account's billing portal.

I wrote the tests first, to prove the hole was real before touching it:

before →  Tests  4 failed | 2 passed     (isolation broken)
after  →  Tests  10 passed (10)          (two independent layers, both proven)

Pre-launch, no real users — prevention, not an incident. Which is why I can show it to you.

That app would have been one of my 52 clean apps.

What to actually do with this

These are the checks no scanner can run for you. All of them you can do this afternoon.

1. Open your own bundle, list every API route in it, then curl each one with no cookie. Ten minutes, and it's what produced most of my real app findings. On your own app it isn't a legal question.
2. The two-account test. Sign up twice. Log in as A. Try to read, edit and delete B's data: by URL, by ID in the address bar, by any share link. Do the delete one. People test reads and forget writes.
3. Grep your bundle for service_role. If it's there, stop reading and rotate it.
4. Check /.git/config. Ten sites in this corpus are serving their repository. One request tells you.
5. Ask where the check lives. For every endpoint returning data: is there an explicit filter by the owning account in the query itself, or are you trusting a database policy to catch it? The answer should be both — one mistake shouldn't reopen everything.
6. Verify RLS is enforced for the role your app actually connects as. "RLS enabled" in a dashboard and "RLS enforced on this connection" are different sentences. Mine said the first and not the second.
7. Lie to your own forms. Send a plan ID you didn't pay for. Send is_admin: true. Change a user ID in a request body. AI writes the path where everyone is honest; the branch where someone lies is usually just missing.
8. Replay a webhook. Same Stripe event twice. Provisioned twice? No idempotency.

Turn the two-account test into a CI test. That's the difference between fixing it and it staying fixed.

The honest summary

Twelve real problems across 652 apps. I'm not going to call that an epidemic, because it isn't one.

I'm also not going to call it a measurement, for three separate reasons, and they stack:

It moves when I improve the instrument. 1 → 3 → 20 reported, across three passes over the same apps, every jump from fixing my own blind spots — a subdomain list in the wrong order, a missing locale prefix. It has never once gone down.

It's noisy in the other direction. Of 41 reported, 12 were real. I wrote the detector and I still had to discard most of its output by hand. Any scan number you haven't personally triaged is inflated by some factor you don't know.

And it's capped by law, not by capability. For 965 addresses I never reached the product. 172 were behind an anti-bot wall. And on the 652 I did reach, every check that actually matters — can account A read account B, does this endpoint verify ownership, does the webhook verify itself — needs a session and a lie. Which is to say, permission.

So: twelve. That's what's visible from the street, with the curtains open, without touching the door.

Absence of evidence, etc. You know the rest of the sentence.

Yes. What I'm about to offer is the same scan whose limits I just spent two thousand words on. Both things are true at once. It reads the bottom two rows of that table and nothing else, and on this corpus that was enough to find ten repositories being served to the public and an auth check that had quietly stopped running. It will not tell you whether your RLS works. Nothing from outside will. If the outside view is useful to you, it's free: the free scan form — findings you can act on whether or not you ever pay me anything. The inside view is a different job, and it's the one I actually do.