Half My AI Crawler Traffic Wasn't AI Crawlers

I told people that AI assistants fetch pages from this site roughly 80 times for every click Google sends me. I believed it because I'd counted it myself: sum the requests whose user-agent string says ChatGPT-User or Claude-User, divide by Search Console clicks. Basically everyone who quotes a number like this counts it the same way.

Then I added one dimension to the query. Cloudflare's GraphQL Analytics API has a field called verifiedBotCategory, which is non-empty only when Cloudflare has confirmed the request actually came from the operator named in the user-agent string, by one of two routes: a Web Bot Auth cryptographic signature on the request, or the older method of checking the source against the operator's published IP list and reverse DNS. Empty means unconfirmed. The field works on the free plan, and adding it meant one more word in a query I was already running.

Half the traffic landed in "unconfirmed." By the time I'd finished checking, 80:1 was wrong in three compounding ways and the honest figure was about 11:1.

The number was already load-bearing by the time I checked it. It was written into aiwatch.py's docstring as the reason the tool existed, and it sat in the design doc for the whole demand-sensing system under a table row labelled "the killer metric." That same morning it had justified a day of work: I added 197 lines to another article specifically because aiwatch told me live AI conversations pulled it harder than any other page on the site.

And this post was going to open with it. The first line I had in my head was "AI assistants fetch this site eighty times for every Google click," a good opening line that was wrong by a factor of seven.

Forget my arithmetic for a second. The user-agent string is a self-declared claim, and the industry is quoting it as a measurement. Most of the "AI bot traffic is up 300%" charts I've seen sourced their numbers from UA strings alone, same as mine, while the field that separates the two sat in a free API the whole time. The exception I know of is Cloudflare Radar, which classifies by verified bot rather than by string.

Three different jobs wearing similar names

Each major operator runs several bots, and they do genuinely different work. Miss that and every number below is easy to over-read.

Per OpenAI's crawler docs, GPTBot "is used to crawl content that may be used in training our generative AI foundation models." OAI-SearchBot builds the index behind ChatGPT's search feature, and sites opted out of it "will not be shown in ChatGPT search answers, though can still appear as navigational links." Quote that sentence whole, because the second clause is the interesting one: opting out leaves you in ChatGPT as a link while taking you out of the answers. ChatGPT-User is the live one, the bot that visits a page because someone in a conversation asked something that needed it.

Anthropic's docs split the same three ways. ClaudeBot collects content "that could potentially contribute to their training," Claude-SearchBot "navigates the web to improve search result quality," and Claude-User "supports Claude AI users" by fetching pages when someone asks a question.

A training crawl tells you your text may end up in a weights file eventually; an index crawl tells you you're eligible to be cited; only the live fetch means a human is sitting there right now, waiting on an answer that includes your page. That third number is the one worth caring about, which is why it's the one people inflate. I inflated it.

The split, on one small zone

Seven days, 23 to 29 August 2026, on bumbletap.com. 65,899 total HTTP requests, 829 distinct user-agent strings. For scale, the single loudest UA on the zone was curl/8.7.1 at 4,857 requests. Everything below is a small slice of a small site.

Claimed UA	verified	unverified	% unverified
meta-external	867	55	6%
Googlebot	285	196	41%
ClaudeBot	197	100	34%
PerplexityBot	0	226	100%
GPTBot	37	187	83%
ChatGPT-User	65	157	71%
OAI-SearchBot	73	137	65%
Claude-User	22	167	88%
Perplexity-User	0	136	100%
Bytespider	25	90	78%
Amazonbot	52	58	53%
Claude-SearchBot	0	56	100%
AI operators only	1,338	1,369	51%
Including Googlebot	1,623	1,565	49%

51% of the traffic claiming an AI operator's name — the larger half of it — could not be confirmed as that operator.

One flaw in my own table before I lean on it. meta-external is a prefix match, so it silently merges Meta's meta-externalagent (training crawls) and meta-externalfetcher (live, user-triggered) into one row. That's the conflation I spent the last section warning about, committed by my own query, in the biggest row on the board. I've left it as one row because that's how I collected it, but don't read that 867 as one bot.

One row keeps this from being an AI-panic post. 41% of the traffic claiming to be Googlebot failed to verify too, and forging a user agent is much older than AI: people have been sticking Googlebot in the header to skip paywalls and rate limits since roughly forever. My hunch is that the AI names are simply the ones that currently get you waved through, so those are the names scanners wear now. That's a claim about change over time, though, and seven days of one zone can't test it. I'd rather say so than dress the hunch up as a finding.

"Unverified" is not the same as "fake"

An empty verifiedBotCategory means Cloudflare couldn't confirm the request, not that the request was forged. That's the obvious objection and it's a good one. Cloudflare's verified-bot list is maintained by humans and it lags. A new bot, a new IP range, an operator who hasn't finished the paperwork: all of those produce honest traffic in the unverified bucket.

Look at Claude-SearchBot in that table. Zero verified, 56 unverified, a clean 100%. I do not believe all 56 of those were forged. The far more likely reading is that Cloudflare's verified list doesn't cover that bot on my zone yet, and 56 perfectly legitimate requests got filed under "couldn't confirm."

Perplexity's two zeros have a different cause, and a much more specific one. Cloudflare de-listed Perplexity from Verified Bots on 4 August 2025, after publishing evidence that it was running undeclared crawlers to get around no-crawl directives: "we have de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling."

So PerplexityBot at 0/226 and Perplexity-User at 0/136 are neither list lag nor forgery. They're a policy decision. Every Perplexity request on every Cloudflare zone has been landing in the unverified bucket for a year. My first draft filed those 362 requests under "the list probably hasn't caught up," which was me not reading the news.

Take both out, Perplexity's 362 as a de-listing and Claude-SearchBot's 56 as genuine lag, and the AI unverified rate drops to 41.5%. Googlebot's is 40.7%. On this sample those are the same number.

That's a better version of "this isn't an AI story" than the raw 51-versus-41 I opened with. An AI operator's name and Google's name fail verification at rates I can't tell apart. Treating "unverified" as "fraud" would be the exact mistake I'm complaining about, run in the other direction. So the split alone proves nothing; what makes it stand up is that the two buckets behave nothing alike.

The two populations don't do the same things

I cross-tabbed the same requests against what they actually asked for. "Content" means the pages a crawler would plausibly want: /blog/*, the end-of-life calendar at /eol/*, and /blueprints/*. "Credential probes" are matched by a strict regex rather than a substring list: .env, .git/, .aws/, .ssh/, .gcloud/, credentials.json|old|bak, gcp-credentials, firebase-adminsdk, wp-config.php, /@fs/, /%40fs/, .npmrc, .gitlab-ci.yml, .openclaw, application_default_credentials.

This table is AI-branded traffic only, no Googlebot, so the columns sum to 1,338 and 1,369 rather than the wider totals from the previous section.

requested	verified	unverified
content (/blog/*, /eol/*, /blueprints/*)	133	98
root / robots.txt / sitemap	270	78
credential probes	0	662
other	935	531
TOTAL	1,338	1,369

Out of 1,338 verified requests from bots claiming an AI operator's name, none went looking for a credentials file. Out of 1,369 unverified ones, 662 did, which is 48.4% of everything that bucket asked for. Close to half of the "AI crawler traffic" I couldn't verify wasn't crawling at all; it was scanning.

Two things about that zero, because a zero is the easiest number in the world to manufacture.

My first classifier matched bare config, .yml, wp-, backup and .sql, any of which shows up in an ordinary content URL. It counted 870 probes, a shade over half of everything unverified the way I was slicing it then; the strict regex above counts 662. What I care about is that the zero survived the tightening. Anyone can manufacture a scary number by loosening a classifier. Finding out whether the scary number was real means tightening it and watching what survives.

The second thing is worse, because it was a bug. My first script tested only the unverified bucket against the probe list, if ok: … elif probe:, which made the zero true by construction: it could not have come out any other way. Both buckets get tested now. A measurement with one possible outcome isn't evidence, and I had written one without noticing.

The response codes say the same thing from the other side. Staying on the AI-only numbers, the same 1,338 and 1,369 as the table above, verified traffic got 1,291 of its requests answered with HTTP 200. That's 96%, because it asked for pages that exist. Unverified traffic collected 742 404s, 292 301s, 110 307s, 85 403s, 4 302s, 3 429s, 2 405s, one 401 — and 130 200s.

I left that last number out of the first version of this paragraph. Anyone adding up my four figures would have found a few hundred requests missing from my own stated total. It's 9% of unverified traffic getting a real page, and it belongs here. The argument survives it easily. 96% against 9% is not a close call, and those 130 successes weren't probes: the probe subset returned no 200s at all.

A crawler that indexes you gets 200s. A scanner guessing at file paths gets 404s, and it gets a lot of them. That's the confirmation: two populations sharing a user-agent string and agreeing on nothing else.

Since this is the point where someone reasonably asks: nothing here was at risk. bumbletap.com is a static export served by Cloudflare Workers Static Assets. There's no application server behind it, and no .env or .git directory to find. Of the 662 probes, 383 got a 404, 161 a 301, 107 a 307, 9 a 403 and 2 a 429. Not one of them got a 200. That's the whole security story and I'm not going to dress it up.

The tells, verbatim

This is the part I'd screenshot. Here are real requests from the dataset, paths alongside the user agent that claimed to be fetching them. None of them returned a 200.

/.git/HEAD          Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/...)
/.git/config        Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
/.env               Mozilla/5.0 (compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user)
/wp/.env            Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/...)
/laravel/.env       Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
/gcp-credentials.json     Mozilla/5.0 (compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot)
/firebase-adminsdk.json   Mozilla/5.0 (compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot)
/.github/workflows/deploy.yml   Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
/instance/config.py             Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
/.aws/credentials.old     Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.an...)
/@fs/var/www/.env         Mozilla/5.0 (compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user)
/@fs/app/.aws/credentials Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)
/.openclaw/.env           Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)
/config/runtime.exs       Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; ...)

Two of those lines, /instance/config.py and /config/runtime.exs, are credential-shaped but don't match the strict regex I counted with, so they sit in the other row rather than the 662. I'm showing them because they're real, not because they're in the tally.

Five things in those strings looked like tells to me. Two survived checking.

Handset device tokens. Some of the fakes carry things like Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; Claude-User/1.0...), and there are Pixel 7 and SM-S918B variants in there too. A platform token by itself isn't suspicious. OpenAI's real published OAI-SearchBot string includes (Macintosh; Intel Mac OS X 10_15_7). Then I wrote that no operator publishes a specific phone. Flatly false, and the counterexample was sitting in my own table further up the page. Google's crawler docs give Googlebot Smartphone as Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html). A Nexus 5X right there in the documented string, and the first thing a reader would have checked.

What's left of it is narrower and it isn't a tell. No AI operator documents a handset token, and iPhone 18_4, Pixel 7 and SM-S918B appear in no published OpenAI or Anthropic string I can find. That's a prior about two specific companies: useful for eyeballing, not something I'd put in a WAF rule.

Three GPTBot versions in one week. GPTBot/1.2, GPTBot/1.3 and GPTBot/1.4 all showed up against this zone inside seven days, and OpenAI documents 1.4. My read is that one operator doesn't run three fleet versions simultaneously at this volume against one small site. So those are several different tools, each pinned to whatever string its author copied off a blog post on whatever day they wrote it.

That's an inference, though, and it's the weaker of the two survivors. OpenAI publishes no version history, so I can't check when 1.2 and 1.3 were current. My own data cuts against me here too. OAI-SearchBot/1.3 turns up in the same week that 1.4 is the documented string, and the whole pattern is equally consistent with stale strings simply circulating. Both stories fit what I can see.

/@fs/ and /%40fs/ paths. The strong one, and it gets its own section.

What /@fs/ is, and why nothing good asks for it

If you don't use Vite: @fs is a special route its development server exposes so it can serve files that live outside your project root, which is a normal thing to need in a monorepo where a package imports from a sibling directory. Vite's server options docs describe server.fs.allow as restricting "files that could be served via /@fs/", with server.fs.deny as a higher-priority blocklist whose shipped defaults cover .env, .env.*, *.{crt,pem,key,p12,pfx,cer,der}, .npmrc, .yarnrc.yml and **/.git/**.

Those defaults exist because this exact attack exists. A dev server accidentally bound to a public interface with a loose server.fs.allow is a file-read primitive, and /@fs/var/www/.env is somebody typing that primitive straight into the URL bar. The %40fs variant is the same request with the @ percent-encoded, which is what you send when you suspect the target filters on the literal character.

Nothing legitimate requests it across the public internet: not a browser pointed at your production domain, and certainly not a crawler that has just told you it works for OpenAI. A Vite dev app on localhost hits /@fs/ constantly, because that's the feature; anywhere else, the request has no innocent reading.

The tells that didn't survive checking

There are two more, on top of the handset one I've just walked back. The second is the embarrassing one, because I had the polarity inverted.

My first pass listed "broken parenthesis placement" as a tell: the real OpenAI strings close the paren at the end (...+https://openai.com/bot)), while the fakes close it early ((KHTML, like Gecko); compatible; ...), leaving the bot token dangling outside the comment group. The theory was tidy right up until I read OpenAI's actual bots page: the string OpenAI publishes today is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, paren closed early. The form I'd written down as the giveaway is the documented one, and both forms appear in my spoofed traffic anyway, which means paren placement separates nothing.

The second one I would have published with a completely straight face. I had +Claude-User@anthropic.com down as generated, on the theory that somebody had seen the real claudebot@anthropic.com and pattern-matched the local part to the bot's name. It was backwards on all three counts. <botname>@anthropic.com appears to be Anthropic's genuine convention, and multiple independent crawler databases record Claude-User's real string as ending +Claude-User@anthropic.com. The address Anthropic actually publishes for site owners is crawlers@anthropic.com, not claudebot@anthropic.com. And Anthropic publishes no complete user-agent strings at all, so no primary source could have settled it either way. I took a real address, labelled it forged, and built a tell on top of the label.

Two honest tells beat a list of five shaky ones, and the wreckage is more useful than the list I thought I was writing. The user-agent string isn't even reliable evidence about its own forgery. Half of what looks like a giveaway turns out to be an operator's documented string, or your own guess about their conventions dressed up as knowledge. Check the docs page before you match on any of these, and when the string and the IP disagree, believe the IP. That's all verifiedBotCategory does, and it's why one field beat an afternoon of my pattern-matching.

Which numbers survive

Here's what I'd still put my name to, on this zone, for this week.

Throw out any per-bot total that came from user-agent strings alone. That includes the top-user-agents view in most analytics dashboards, most WordPress bot plugins, and the number I'd been carrying around. A total like that is two unrelated quantities added together, and there's no way to tell from it which one moved.

Keep the verified counts, with one asterisk. Verified AI-operator requests came to 1,338 against 285 verified Googlebot hits. About 4.7 to 1. The asterisk: 867 of that 1,338 is meta-external on its own, 65% of the total, and a reader I've just spent two thousand words teaching to distrust aggregates is going to check that first. Excluding Meta, it's 1.65 to 1. Both numbers are true, and the second one is the one I'd quote.

And the corrected headline. Here is how badly I had it.

The number I'd been carrying around was about 80 to 1. It came from 411 requests whose user-agent claimed ChatGPT-User or Claude-User, divided by 5 Google Search clicks. Three things were wrong with it, and they multiplied.

One, it counted claims. Across the same seven days only 87 of those 411 requests verify. That's 65 ChatGPT-User and 22 Claude-User.

Two, the windows didn't match, which I hadn't noticed at all. Search Console covered 20 to 27 August; Cloudflare covered 23 to 29. On the six days both sources cover, 23 to 28 August 2026, the verified figure is 82 requests and Search Console shows 4 clicks, not 5.

Three, it counted /robots.txt and / as though they were article reads. Claude-User shows that damage best: 22 verified requests, of which 17 were /robots.txt and 3 were content pages. Strip the housekeeping out of the 82 and the live fetchers pulled 43 content pages.

Matched window, verified only, content pages only: 43 fetches against 4 clicks. About 11 to 1.

That's a sevenfold correction to my own headline. The survivor has a problem of its own: it divides machine requests by human clicks, which is not a like-for-like comparison and never will be, because a click is a person choosing to arrive and a fetch is a model reading. I use the ratio as a direction rather than a measurement, and you should read it the same way.

11 to 1 is still lopsided enough to justify the tool. I wrote tools/aiwatch.py to watch this channel because this site's acquisition really is AI assistants rather than Google, and I'd rather have the small true number than the big false one.

What the live fetchers pulled is more interesting than the totals anyway. On verified traffic the most-requested article was how Chrome extensions actually run your code, at 6 ChatGPT-User fetches, while Claude-User's busiest page was the Manifest V2 removal post at 3. And that robots.txt number is worth sitting with: the live fetcher checked whether it was allowed more than five times for every content page it took. A Disallow line still means something to the half of this traffic that is who it says it is.

The correction changed almost nothing about what I do. I kept the tool and the channel and went on writing for it, because eleven to one points exactly where eighty to one pointed — and if the honest number had come back at 2:1 I'd have kept going anyway, since 43 machine reads a week against 4 human clicks is not a close call either. A ratio that survives a sevenfold haircut and still says the same thing was never really doing the work I thought it was.

What changed is underneath. verified is part of the primary key in the table now, so the two buckets can't be summed by accident. Both buckets get tested against the probe list. The design doc no longer says these bots self-identify, because that sentence was the root of all of it. And the one decision the bad number had already paid for turned out fine: I expanded that extensions article because aiwatch said live conversations pulled it more than anything else, and on verified-only traffic it is still the most-pulled page on the site, so the direction held. My confidence in it shouldn't have been as cheap as it was.

Run it on your own zone in ten minutes

If your site is behind Cloudflare, you can reproduce all of this today. It's one query.

query ($zone: String!, $start: Time!, $end: Time!) {
  viewer { zones(filter: {zoneTag: $zone}) {
    httpRequestsAdaptiveGroups(
      filter: {datetime_geq: $start, datetime_lt: $end,
               OR: [{userAgent_like: "%GPTBot%"},
                    {userAgent_like: "%ChatGPT-User%"},
                    {userAgent_like: "%Claude%"},
                    {userAgent_like: "%Googlebot%"}]},
      limit: 5000, orderBy: [count_DESC]) {
      count
      dimensions { userAgent verifiedBotCategory clientRequestPath edgeResponseStatus }
    } } } }

With variables shaped like this, one day per request:

{
  "zone": "<32-character zone id>",
  "start": "2026-08-23T00:00:00Z",
  "end": "2026-08-24T00:00:00Z"
}

Time wants exactly that format, RFC 3339 with the trailing Z, and it will not meet you halfway. POST the whole thing to https://api.cloudflare.com/client/v4/graphql with an Authorization: Bearer <token> header. The token needs one permission and no more: Zone → Analytics → Read. Add more userAgent_like clauses to the OR array for whichever operators you care about.

Before you go looking for the field itself: verifiedBotCategory is not in Cloudflare's public dataset documentation. I found it by introspection, so if you search for it you'll find roughly nothing and reasonably start wondering whether I invented it. Ask the schema instead:

{ __type(name: "ZoneHttpRequestsAdaptiveGroupsDimensions") { fields { name } } }

That returns every dimension the dataset exposes on your zone, documented or not. One thing about the values it hands back: Cloudflare changed the bot taxonomy on 1 July 2026 and no longer distinguishes "AI Search" from ordinary search. Both are Search now, with the old value kept alive for rule compatibility. If your categories don't look like mine, check which side of that date your data sits on.

Three collection constraints bit me, and all three are worth knowing before you start.

Retention and the query window are per-zone values, not a plan policy. Cloudflare doesn't publish fixed per-plan figures for these. On my free zone the settings discovery node reported 7 days (notOlderThan) and a 24-hour maximum window (maxDuration), so a week of data was seven queries summed in my own code. Ask for your own rather than trusting mine:

query ($zone: string) {
  viewer { zones(filter: {zoneTag: $zone}) {
    settings { httpRequestsAdaptiveGroups {
      enabled maxDuration maxNumberOfFields maxPageSize notOlderThan availableFields
    } } } } }

maxPageSize is the ceiling on limit, and availableFields is the quick way to find out what your plan doesn't have before you write the query around it.

limit truncates silently. With orderBy: [count_DESC], everything past your limit is simply gone, with no error and no flag in the response. I collected the first pass at limit: 1000 and re-ran the recount at 5,000, and on a zone busier than this one you can lose the entire tail without ever being told. If a row count comes back exactly equal to your limit, assume you're truncated and raise it.

clientAsn and clientASNDescription are not available. Ask for either and you get:

zone '<id>' does not have access to the field 'clientasn'

Note the lowercase clientasn in the error, which is what you'll want to paste into a search box. This is also the reason there's nothing in this post about who the scanners are or where they connect from. I couldn't see the network they came from, and guessing would be inventing. verifiedBotCategory itself is available on free, which is the only field this post needed.

One ask, for anyone writing about AI crawler volume: run the query before you publish the chart. Running it took me under an hour. Correcting myself afterwards took a great deal longer, and it's most of what you've just read.

Caveats

• n = 1 zone. One small site, 7 days, 65,899 requests. The shape of this generalises: UA strings are claims, the two buckets behave differently, tells exist. The percentages don't. Your zone's 51% will be a different number.
• The long tail is cut by my query, not by sampling. I originally blamed this on Cloudflare's adaptive sampling, which was the wrong mechanism in the wrong direction. My own limit combined with orderBy: count_DESC is what truncated the tail of those 829 user agents. Where sampling does apply, Cloudflare extrapolates the aggregates, so counts come back as estimates with error in both directions rather than as floors. At 65,899 requests for the week this zone was almost certainly unsampled anyway.
• "Unverified" stays unproven. I've argued the behavioural split makes forgery the best explanation for most of that bucket, not all of it. Claude-SearchBot at 0/56 is my own example of a case where I think the verification list, not the traffic, is what's wrong. Perplexity's 362 has a documented cause that is neither.
• The article-level counts are verified-only. The per-page fetch numbers in "Which numbers survive" come from verified traffic alone, which means any genuine request sitting in the unverified bucket because of list lag or a de-listing isn't in them. Read them as a floor on live-assistant interest, not a total.
• Published UA strings change. Every string quoted here is what the operator documented, or what my logs recorded, in August 2026. If you match on them in a WAF rule, match loosely, and re-check the source pages rather than trusting this post a year from now.