Fake AI Crawlers: How We Verified a Week of Bot Traffic
We matched 23,962 requests from 15 crawler names to the IP lists OpenAI, Google, Anthropic and others publish. 11.1% came from elsewhere, and the fakes went for .env files.

In short: Don't trust a crawler's name. Check the address against the list its operator publishes before you count, allow or block it, and keep secrets out of anything your web server can reach. Then look at which pages the verified crawlers read, because that is what search engines and AI answers will know about you.
On this page12
Overview
Every site gets visits from bots that say they are Googlebot, GPTBot or ClaudeBot. The name is a line of text in the request, and anyone can type it. So how many of these visits are real?
We checked. We took one week of request logs from vandatateam.com, the site of Van Data Team, the company behind Vanaxity, and matched every request that named a crawler against the address list its operator publishes. This article shares what we found: how many requests were fake, what the fakes wanted, and what the real AI crawlers read.
Disclosure: Vanaxity builds AI agents for marketing, and we want AI crawlers to read our pages. The data comes from our own logs, and we describe every step of the method. This article was drafted with help from Claude, which is made by Anthropic, one of the companies whose crawlers we measured.
Key takeaways
- From November 9 to 15, 2026, 23,962 requests to vandatateam.com named one of 15 crawlers whose operators publish an address list.
- 21,303 of them (88.9%) came from the operator's addresses. 2,659 (11.1%) came from somewhere else.
- Seven addresses sent 627 of those outsiders under as many as 15 crawler names. One sent 155 requests under 15 names in seven seconds.
- 321 requests from outside the lists went to paths only a scanner asks for, such as .env files and .git folders. Verified crawlers asked for none of them.
- Verified crawlers read mostly our AI model benchmark pages, and Meta's crawlers read 1,086 different pages in the week.
- Verified AI crawlers sent 16,718 requests, 4.3 times as many as Googlebot and Bingbot together.
- No major AI or search crawler asked for our llms.txt file all week.
Book a free fit checkMap your SEO, GEO and AEO workflow before you build.
Book a consultationHow we checked each crawler
vandatateam.com runs on Firebase App Hosting, whose request log records the user agent and the client address of every request. We pulled the week of November 9 to 15, 2026 (UTC) and kept the requests whose user agent named a known crawler.
Most operators publish the addresses their crawlers use and explain how to check them. We downloaded each list on November 16, 2026 and checked every request against the list for the name it gave:
| Crawler names | Operator | How we verified them |
|---|---|---|
| Googlebot | Google's list for its common crawlers, generated November 13, 2026 | |
| Bingbot | Microsoft | bingbot.json, last generated January 3, 2024 |
| GPTBot, OAI-SearchBot, ChatGPT-User | OpenAI | A separate list for each bot |
| ClaudeBot, Claude-User, Claude-SearchBot | Anthropic | One list, bots.json, dated October 20, 2026 |
| PerplexityBot, Perplexity-User | Perplexity | A separate list for each bot |
| Applebot | Apple | applebot.json |
| Amazonbot | Amazon | The address page for Amazonbot |
| CCBot | Common Crawl | ccbot.json |
| Meta-ExternalAgent, Meta-WebIndexer | Meta | The routes Meta's network (AS32934) announces, the check Meta's crawler page gives |
| DuckAssistBot | DuckDuckGo | DuckDuckGo's published list |
Bytespider, ByteDance's crawler, sent 1,377 requests, but we found no published list for it, so it is left out of the check. SEO tools such as AhrefsBot and SemrushBot and link previews such as facebookexternalhit are left out too.
A request counts as verified when its address falls inside the operator's list. We publish totals only, never addresses, and the counts behind every chart are in our data file.
How many crawler requests were real
23,962 requests named one of the 15 crawlers. 21,303 of them, or 88.9%, came from the operator's own addresses. The other 2,659, or 11.1%, did not.
The share varied a lot by name. OpenAI's search crawler was the cleanest, at 1.1%. Two names never came from a verified address at all:
Requests from outside the operator's published list, November 9 to 15, 2026
Share of the crawler's requests
| Crawler | Requests | From the list | Not from the list | Share not from the list |
|---|---|---|---|---|
| Meta's crawlers | 10,260 | 8,492 | 1,768 | 17.2% |
| OAI-SearchBot | 3,064 | 3,031 | 33 | 1.1% |
| PerplexityBot | 2,184 | 2,156 | 28 | 1.3% |
| Googlebot | 2,179 | 2,062 | 117 | 5.4% |
| Bingbot | 1,830 | 1,795 | 35 | 1.9% |
| Amazonbot | 1,080 | 1,054 | 26 | 2.4% |
| Applebot | 756 | 728 | 28 | 3.7% |
| ChatGPT-User | 570 | 530 | 40 | 7.0% |
| CCBot | 516 | 472 | 44 | 8.5% |
| GPTBot | 504 | 465 | 39 | 7.7% |
| ClaudeBot | 363 | 261 | 102 | 28.1% |
| Claude-User | 299 | 0 | 299 | 100% |
| DuckAssistBot | 293 | 257 | 36 | 12.3% |
| Perplexity-User | 38 | 0 | 38 | 100% |
| Claude-SearchBot | 26 | 0 | 26 | 100% |
| All 15 | 23,962 | 21,303 | 2,659 | 11.1% |
A high share doesn't always mean a busy faker. ClaudeBot's 28.1% is 102 requests, while Meta's 17.2% is 1,768. The next section splits the outsiders by who sent them.
Who sent the fake requests
Not every request from outside a list was an attack. We split the 2,659 by the address that sent them and what it asked for:
- 627 came from just seven addresses that used more than one crawler name.
- 1,713 used a Meta crawler name from 821 addresses outside Meta's network. They asked for normal pages, and we cover them below.
- 266 were Claude-User requests that look like real use, also covered below.
- The last 53 came mostly from addresses posing as Googlebot (43).
Scanners in costume
The seven multi-name addresses were scanners. One sent 394 requests in 40 minutes under 15 crawler names, from Googlebot to Perplexity-User. Another sent 155 requests under 15 names in seven seconds. A third used eight names.
They were after secrets. 321 requests from outside the lists went to paths that only a scanner asks for: .env files, .git folders, cloud key files, WordPress logins and debug pages. 308 of those came from the seven multi-name addresses. No verified crawler asked for a secret file; the only verified requests that matched these patterns were two Bingbot visits to old WordPress-style links that now redirect.
None of the 321 found anything: 314 got a 404 and 7 a redirect, because the site is a static build with no such files. On a site with a stray .env file, a rule that lets "GPTBot" or "Googlebot" through by name would have let these scanners through too.
Fake Googlebots
117 requests claimed to be Googlebot from 24 addresses that appear on none of Google's lists, including its lists for special crawlers and user-triggered fetchers. 74 came from the multi-name scanners, and 48 went to scanner paths.
Meta-named requests we can't explain
1,713 requests used a Meta crawler name from 821 addresses outside the routes Meta's network announces, almost all with the short user agent "meta-webindexer/1.1". Most addresses sent a single request. Together they asked for only 39 pages, mostly our most-shared blog posts, and the busiest post drew 557 of them. By contrast, 8,469 of the 8,492 verified Meta requests used a long user agent that starts like a desktop browser.
We can't tell who sent them. They didn't probe for secrets, but they didn't come from Meta's documented network either, so we count them as unverified and leave it there.
Claude-User comes from people's own computers
None of Claude-User's 299 requests came from Anthropic's list, and that doesn't make them fake. 109 carried a Claude Code version in the user agent, from 94 addresses. Claude Code fetches pages from the computer it runs on, so those addresses can't be on any list. Another 157 came from 43 addresses and read ordinary pages. The last 33 came from the multi-name scanners.
What the real AI crawlers read
Next we kept only verified requests, to see what each crawler actually read. The sitemap listed 1,261 URLs at the end of the week: 925 AI model benchmark pages (73%), 266 blog posts (21%), 52 service and portfolio pages (4%) and 18 other pages (1%).
Most crawlers read the benchmark pages even more than their share of URLs would suggest:
Share of each verified crawler's page requests that went to benchmark pages
Benchmark pages, share of page requests
| Verified crawler | Page requests | Different pages | Benchmark pages | Blog posts |
|---|---|---|---|---|
| Meta's crawlers | 6,295 | 1,086 | 44% | 49% |
| OAI-SearchBot | 2,419 | 464 | 87% | 12% |
| PerplexityBot | 1,961 | 763 | 65% | 28% |
| Bingbot | 1,419 | 584 | 83% | 12% |
| Amazonbot | 959 | 937 | 76% | 20% |
| Googlebot | 839 | 393 | 59% | 27% |
| ChatGPT-User | 528 | 150 | 23% | 20% |
| CCBot | 469 | 469 | 86% | 7% |
| Applebot | 346 | 213 | 72% | 27% |
| DuckAssistBot | 207 | 114 | 56% | 43% |
| ClaudeBot | 157 | 148 | 72% | 20% |
| GPTBot | 57 | 53 | 42% | 44% |
A few habits stood out:
- Meta's crawlers read 1,086 different pages in seven days, most of the site. They leaned toward the blog, at 49% of their page requests against 21% of URLs.
- OAI-SearchBot, which feeds ChatGPT search, sent 87% of its page requests to benchmark pages, the kind of page people ask ChatGPT about.
- GPTBot, OpenAI's training crawler, made only 57 page requests. Most of its 465 verified requests were for images, scripts and styles.
- ChatGPT-User, which fetches a page when someone in ChatGPT needs it, opened our home page 274 times, more than any other page.
- Service and portfolio pages, the ones that sell, drew about 5% of any crawler's page requests at most.
AI crawlers now outnumber search crawlers
Counting verified requests only, AI crawlers sent 16,718 requests in the week: Meta's crawlers, OpenAI's three bots, PerplexityBot, Amazonbot, CCBot, ClaudeBot and DuckAssistBot. Googlebot and Bingbot together sent 3,857, so the AI crawlers sent 4.3 times as many. Applebot, which serves both Apple's search and its AI features, is in neither group.
Verified requests per crawler, November 9 to 15, 2026
Requests from the operator's addresses
Meta's crawlers alone sent 8,492 verified requests, about four times Googlebot's 2,062. Next came OAI-SearchBot with 3,031, PerplexityBot with 2,156, Bingbot with 1,795, Amazonbot with 1,054, Applebot with 728, ChatGPT-User with 530, CCBot with 472, GPTBot with 465, ClaudeBot with 261 and DuckAssistBot with 257.
Requests aren't readers, and a crawler that reads a page once may cite it a thousand times. But the split shows where the reading happens now: most of the machines that read our site in that week were building AI answers, not a list of blue links.
Who read robots.txt and llms.txt
Verified crawlers fetched robots.txt often. Googlebot fetched it 254 times in the week, DuckAssistBot 50, PerplexityBot 43, OAI-SearchBot 41 and ClaudeBot 32. A change to robots.txt reaches these crawlers quickly.
The fakes ignore robots.txt anyway. That is why a rule keyed on a crawler's name is weak protection in either direction: an allow rule lets a fake in, and a block rule only stops the crawlers that were honest to begin with.
llms.txt is a proposed file that gives AI tools a map of a site, and vandatateam.com publishes one. In the whole week it drew 42 requests, and none came from OpenAI, Anthropic, Google, Microsoft, Perplexity, Meta, Apple or Amazon. They came from SEO tools, small AI agents and test browsers: SEOJuice's bot sent 9, agentwebmap 6 and Andi 5.
Keep an llms.txt if it costs you little. Don't expect the big crawlers to read it.
How to check the crawlers on your own site
- Export a week of request logs with the time, path, status, user agent and client address. Behind a CDN, use the client address the CDN passes on.
- Download the operators' lists on the day you check, because they change.
- Group requests by the crawler name in the user agent, and check each address against the list for that name.
- Read the unverified requests by path. Requests for .env, .git and key files mean a scanner in costume.
- Read the verified requests by page. That is what search engines and AI answers know about your site.
Most lists share one format, so a few lines of Python do the matching. Meta is the exception: its check uses the routes its network announces, which you get from a routing registry rather than a JSON file.
import ipaddress, json, urllib.request
def load(url):
data = json.load(urllib.request.urlopen(url))
return [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
for p in data["prefixes"]]
gptbot = load("https://openai.com/gptbot.json")
def verified(ip, networks):
addr = ipaddress.ip_address(ip)
return any(addr in net for net in networks if net.version == addr.version)What to change
- Don't allow or block a crawler by name alone. If you filter at all, filter by verified address.
- Keep secrets out of anything your web server can serve. Scanners ask every site for .env and .git.
- Leave the real AI crawlers in if you want to show up in AI answers. vandatateam.com blocks none of them.
- Make the pages the crawlers read worth citing. On our site that means the benchmark pages, so that is where we keep the data freshest.
- Count bots in your analytics separately. Our own GA4 had a bot problem too: see how one visitor made 88% of our sessions.
What this data can't tell you
- It covers one week on one site. A site with other content will draw other crawlers in other proportions.
- Lists change. Bing's list was last generated in January 2024, and a crawler may use a new address before its operator lists it.
- User-triggered fetchers such as Claude-User can run on people's own computers, so an address missing from a list doesn't prove a fake.
- Meta's check rests on the routes its network announces, as Meta's own page advises, and it can't explain the 1,713 short-form requests.
- Bytespider has no list we could find, so its 1,377 requests can't be checked at all.
Fake AI crawlers FAQ
How do I know if Googlebot is real?
Check the request's address against the list Google publishes for its common crawlers, or run a reverse DNS lookup that ends in googlebot.com or google.com, then a forward lookup that returns the same address. In our week, 117 requests named Googlebot from addresses on none of Google's lists.
Do OpenAI and Anthropic publish crawler IP addresses?
Yes. OpenAI publishes a list for each of GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic publishes one list for its crawlers. Google, Microsoft, Perplexity, Apple, Amazon, Common Crawl and DuckDuckGo publish lists too. We found none for Bytespider.
Why do bots fake AI crawler user agents?
To get past rules that trust those names. In our logs the fakes went for .env files, .git folders and cloud keys, and two addresses switched between 15 crawler names.
Do AI crawlers read llms.txt?
Not on our site. In one week, none of the major AI or search crawlers asked for vandatateam.com's llms.txt. Its 42 requests came from SEO tools, small agents and test browsers.
Should I block AI crawlers?
Only if you don't want AI tools to read or cite your pages. If you do block them, use robots.txt for the honest crawlers and verified addresses for everything else, since the fakes ignore robots.txt.
Sources and data
The counts behind this article are in our data file: one row per crawler name, totals only. The operators' pages on their crawlers and lists:
- Google: Verifying Google's crawlers and fetchers
- Microsoft: Bingbot address list
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Perplexity: Perplexity crawlers
- Apple: About Applebot
- Amazon: Amazonbot
- Common Crawl: CCBot
- Meta: Meta web crawlers
- DuckDuckGo: DuckAssistBot
- The llms.txt proposal

