Fake AI Crawlers: How We Verified a Week of Bot Traffic

We matched 23,962 requests from 15 crawler names to the IP lists OpenAI, Google, Anthropic and others publish. 11.1% came from elsewhere, and the fakes went for .env files.

3D illustration of a neon path of cyan light motes running across a dark mirror floor and through an arch of cyan light, while a dim amber branch leads three amber motes wearing dark visors to a glowing amber barrier where they stop.

In short: Don't trust a crawler's name. Check the address against the list its operator publishes before you count, allow or block it, and keep secrets out of anything your web server can reach. Then look at which pages the verified crawlers read, because that is what search engines and AI answers will know about you.

On this page12

Overview

Every site gets visits from bots that say they are Googlebot, GPTBot or ClaudeBot. The name is a line of text in the request, and anyone can type it. So how many of these visits are real?

We checked. We took one week of request logs from vandatateam.com, the site of Van Data Team, the company behind Vanaxity, and matched every request that named a crawler against the address list its operator publishes. This article shares what we found: how many requests were fake, what the fakes wanted, and what the real AI crawlers read.

Disclosure: Vanaxity builds AI agents for marketing, and we want AI crawlers to read our pages. The data comes from our own logs, and we describe every step of the method. This article was drafted with help from Claude, which is made by Anthropic, one of the companies whose crawlers we measured.

Key takeaways

  • From November 9 to 15, 2026, 23,962 requests to vandatateam.com named one of 15 crawlers whose operators publish an address list.
  • 21,303 of them (88.9%) came from the operator's addresses. 2,659 (11.1%) came from somewhere else.
  • Seven addresses sent 627 of those outsiders under as many as 15 crawler names. One sent 155 requests under 15 names in seven seconds.
  • 321 requests from outside the lists went to paths only a scanner asks for, such as .env files and .git folders. Verified crawlers asked for none of them.
  • Verified crawlers read mostly our AI model benchmark pages, and Meta's crawlers read 1,086 different pages in the week.
  • Verified AI crawlers sent 16,718 requests, 4.3 times as many as Googlebot and Bingbot together.
  • No major AI or search crawler asked for our llms.txt file all week.

Book a free fit checkMap your SEO, GEO and AEO workflow before you build.

Book a consultation

How we checked each crawler

vandatateam.com runs on Firebase App Hosting, whose request log records the user agent and the client address of every request. We pulled the week of November 9 to 15, 2026 (UTC) and kept the requests whose user agent named a known crawler.

Most operators publish the addresses their crawlers use and explain how to check them. We downloaded each list on November 16, 2026 and checked every request against the list for the name it gave:

Crawler namesOperatorHow we verified them
GooglebotGoogleGoogle's list for its common crawlers, generated November 13, 2026
BingbotMicrosoftbingbot.json, last generated January 3, 2024
GPTBot, OAI-SearchBot, ChatGPT-UserOpenAIA separate list for each bot
ClaudeBot, Claude-User, Claude-SearchBotAnthropicOne list, bots.json, dated October 20, 2026
PerplexityBot, Perplexity-UserPerplexityA separate list for each bot
ApplebotAppleapplebot.json
AmazonbotAmazonThe address page for Amazonbot
CCBotCommon Crawlccbot.json
Meta-ExternalAgent, Meta-WebIndexerMetaThe routes Meta's network (AS32934) announces, the check Meta's crawler page gives
DuckAssistBotDuckDuckGoDuckDuckGo's published list

Bytespider, ByteDance's crawler, sent 1,377 requests, but we found no published list for it, so it is left out of the check. SEO tools such as AhrefsBot and SemrushBot and link previews such as facebookexternalhit are left out too.

A request counts as verified when its address falls inside the operator's list. We publish totals only, never addresses, and the counts behind every chart are in our data file.

How many crawler requests were real

23,962 requests named one of the 15 crawlers. 21,303 of them, or 88.9%, came from the operator's own addresses. The other 2,659, or 11.1%, did not.

The share varied a lot by name. OpenAI's search crawler was the cleanest, at 1.1%. Two names never came from a verified address at all:

Requests from outside the operator's published list, November 9 to 15, 2026

Share of the crawler's requests

  1. Claude-SearchBot100.0%
  2. Perplexity-User100.0%
  3. ClaudeBot28.1%
  4. Meta's crawlers17.2%
  5. DuckAssistBot12.3%
  6. CCBot8.5%
  7. GPTBot7.7%
  8. ChatGPT-User7.0%
  9. Googlebot5.4%
  10. Applebot3.7%
  11. Amazonbot2.4%
  12. Bingbot1.9%
  13. PerplexityBot1.3%
  14. OAI-SearchBot1.1%
Claude-User is left out of the chart because most of its requests come from people's own computers (see below). Claude-SearchBot and Perplexity-User sent no request from a verified address that week.Source: Vanaxity: vandatateam.com request logs, November 9 to 15, 2026
CrawlerRequestsFrom the listNot from the listShare not from the list
Meta's crawlers10,2608,4921,76817.2%
OAI-SearchBot3,0643,031331.1%
PerplexityBot2,1842,156281.3%
Googlebot2,1792,0621175.4%
Bingbot1,8301,795351.9%
Amazonbot1,0801,054262.4%
Applebot756728283.7%
ChatGPT-User570530407.0%
CCBot516472448.5%
GPTBot504465397.7%
ClaudeBot36326110228.1%
Claude-User2990299100%
DuckAssistBot2932573612.3%
Perplexity-User38038100%
Claude-SearchBot26026100%
All 1523,96221,3032,65911.1%

A high share doesn't always mean a busy faker. ClaudeBot's 28.1% is 102 requests, while Meta's 17.2% is 1,768. The next section splits the outsiders by who sent them.

Who sent the fake requests

Not every request from outside a list was an attack. We split the 2,659 by the address that sent them and what it asked for:

  • 627 came from just seven addresses that used more than one crawler name.
  • 1,713 used a Meta crawler name from 821 addresses outside Meta's network. They asked for normal pages, and we cover them below.
  • 266 were Claude-User requests that look like real use, also covered below.
  • The last 53 came mostly from addresses posing as Googlebot (43).

Scanners in costume

The seven multi-name addresses were scanners. One sent 394 requests in 40 minutes under 15 crawler names, from Googlebot to Perplexity-User. Another sent 155 requests under 15 names in seven seconds. A third used eight names.

They were after secrets. 321 requests from outside the lists went to paths that only a scanner asks for: .env files, .git folders, cloud key files, WordPress logins and debug pages. 308 of those came from the seven multi-name addresses. No verified crawler asked for a secret file; the only verified requests that matched these patterns were two Bingbot visits to old WordPress-style links that now redirect.

None of the 321 found anything: 314 got a 404 and 7 a redirect, because the site is a static build with no such files. On a site with a stray .env file, a rule that lets "GPTBot" or "Googlebot" through by name would have let these scanners through too.

Fake Googlebots

117 requests claimed to be Googlebot from 24 addresses that appear on none of Google's lists, including its lists for special crawlers and user-triggered fetchers. 74 came from the multi-name scanners, and 48 went to scanner paths.

Meta-named requests we can't explain

1,713 requests used a Meta crawler name from 821 addresses outside the routes Meta's network announces, almost all with the short user agent "meta-webindexer/1.1". Most addresses sent a single request. Together they asked for only 39 pages, mostly our most-shared blog posts, and the busiest post drew 557 of them. By contrast, 8,469 of the 8,492 verified Meta requests used a long user agent that starts like a desktop browser.

We can't tell who sent them. They didn't probe for secrets, but they didn't come from Meta's documented network either, so we count them as unverified and leave it there.

Claude-User comes from people's own computers

None of Claude-User's 299 requests came from Anthropic's list, and that doesn't make them fake. 109 carried a Claude Code version in the user agent, from 94 addresses. Claude Code fetches pages from the computer it runs on, so those addresses can't be on any list. Another 157 came from 43 addresses and read ordinary pages. The last 33 came from the multi-name scanners.

What the real AI crawlers read

Next we kept only verified requests, to see what each crawler actually read. The sitemap listed 1,261 URLs at the end of the week: 925 AI model benchmark pages (73%), 266 blog posts (21%), 52 service and portfolio pages (4%) and 18 other pages (1%).

Most crawlers read the benchmark pages even more than their share of URLs would suggest:

Share of each verified crawler's page requests that went to benchmark pages

Benchmark pages, share of page requests

  1. Share of sitemap URLs73%
  2. OAI-SearchBot87%
  3. CCBot86%
  4. Bingbot83%
  5. Amazonbot76%
  6. ClaudeBot72%
  7. Applebot72%
  8. PerplexityBot65%
  9. Googlebot59%
  10. DuckAssistBot56%
  11. Meta's crawlers44%
  12. GPTBot42%
  13. ChatGPT-User23%
Page requests from verified addresses only: images, scripts, robots.txt and scanner paths are left out. Benchmark pages were 73% of the sitemap's URLs.Source: Vanaxity: vandatateam.com request logs, November 9 to 15, 2026
Verified crawlerPage requestsDifferent pagesBenchmark pagesBlog posts
Meta's crawlers6,2951,08644%49%
OAI-SearchBot2,41946487%12%
PerplexityBot1,96176365%28%
Bingbot1,41958483%12%
Amazonbot95993776%20%
Googlebot83939359%27%
ChatGPT-User52815023%20%
CCBot46946986%7%
Applebot34621372%27%
DuckAssistBot20711456%43%
ClaudeBot15714872%20%
GPTBot575342%44%

A few habits stood out:

  • Meta's crawlers read 1,086 different pages in seven days, most of the site. They leaned toward the blog, at 49% of their page requests against 21% of URLs.
  • OAI-SearchBot, which feeds ChatGPT search, sent 87% of its page requests to benchmark pages, the kind of page people ask ChatGPT about.
  • GPTBot, OpenAI's training crawler, made only 57 page requests. Most of its 465 verified requests were for images, scripts and styles.
  • ChatGPT-User, which fetches a page when someone in ChatGPT needs it, opened our home page 274 times, more than any other page.
  • Service and portfolio pages, the ones that sell, drew about 5% of any crawler's page requests at most.

AI crawlers now outnumber search crawlers

Counting verified requests only, AI crawlers sent 16,718 requests in the week: Meta's crawlers, OpenAI's three bots, PerplexityBot, Amazonbot, CCBot, ClaudeBot and DuckAssistBot. Googlebot and Bingbot together sent 3,857, so the AI crawlers sent 4.3 times as many. Applebot, which serves both Apple's search and its AI features, is in neither group.

Verified requests per crawler, November 9 to 15, 2026

Requests from the operator's addresses

  1. Meta's crawlers8,492
  2. OAI-SearchBot3,031
  3. PerplexityBot2,156
  4. Googlebot2,062
  5. Bingbot1,795
  6. Amazonbot1,054
  7. Applebot728
  8. ChatGPT-User530
  9. CCBot472
  10. GPTBot465
  11. ClaudeBot261
  12. DuckAssistBot257
Googlebot and Bingbot are highlighted. Requests include pages, images and scripts.Source: Vanaxity: vandatateam.com request logs, November 9 to 15, 2026

Meta's crawlers alone sent 8,492 verified requests, about four times Googlebot's 2,062. Next came OAI-SearchBot with 3,031, PerplexityBot with 2,156, Bingbot with 1,795, Amazonbot with 1,054, Applebot with 728, ChatGPT-User with 530, CCBot with 472, GPTBot with 465, ClaudeBot with 261 and DuckAssistBot with 257.

Requests aren't readers, and a crawler that reads a page once may cite it a thousand times. But the split shows where the reading happens now: most of the machines that read our site in that week were building AI answers, not a list of blue links.

Who read robots.txt and llms.txt

Verified crawlers fetched robots.txt often. Googlebot fetched it 254 times in the week, DuckAssistBot 50, PerplexityBot 43, OAI-SearchBot 41 and ClaudeBot 32. A change to robots.txt reaches these crawlers quickly.

The fakes ignore robots.txt anyway. That is why a rule keyed on a crawler's name is weak protection in either direction: an allow rule lets a fake in, and a block rule only stops the crawlers that were honest to begin with.

llms.txt is a proposed file that gives AI tools a map of a site, and vandatateam.com publishes one. In the whole week it drew 42 requests, and none came from OpenAI, Anthropic, Google, Microsoft, Perplexity, Meta, Apple or Amazon. They came from SEO tools, small AI agents and test browsers: SEOJuice's bot sent 9, agentwebmap 6 and Andi 5.

Keep an llms.txt if it costs you little. Don't expect the big crawlers to read it.

How to check the crawlers on your own site

  1. Export a week of request logs with the time, path, status, user agent and client address. Behind a CDN, use the client address the CDN passes on.
  2. Download the operators' lists on the day you check, because they change.
  3. Group requests by the crawler name in the user agent, and check each address against the list for that name.
  4. Read the unverified requests by path. Requests for .env, .git and key files mean a scanner in costume.
  5. Read the verified requests by page. That is what search engines and AI answers know about your site.

Most lists share one format, so a few lines of Python do the matching. Meta is the exception: its check uses the routes its network announces, which you get from a routing registry rather than a JSON file.

import ipaddress, json, urllib.request

def load(url):
    data = json.load(urllib.request.urlopen(url))
    return [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
            for p in data["prefixes"]]

gptbot = load("https://openai.com/gptbot.json")

def verified(ip, networks):
    addr = ipaddress.ip_address(ip)
    return any(addr in net for net in networks if net.version == addr.version)

What to change

  • Don't allow or block a crawler by name alone. If you filter at all, filter by verified address.
  • Keep secrets out of anything your web server can serve. Scanners ask every site for .env and .git.
  • Leave the real AI crawlers in if you want to show up in AI answers. vandatateam.com blocks none of them.
  • Make the pages the crawlers read worth citing. On our site that means the benchmark pages, so that is where we keep the data freshest.
  • Count bots in your analytics separately. Our own GA4 had a bot problem too: see how one visitor made 88% of our sessions.

What this data can't tell you

  • It covers one week on one site. A site with other content will draw other crawlers in other proportions.
  • Lists change. Bing's list was last generated in January 2024, and a crawler may use a new address before its operator lists it.
  • User-triggered fetchers such as Claude-User can run on people's own computers, so an address missing from a list doesn't prove a fake.
  • Meta's check rests on the routes its network announces, as Meta's own page advises, and it can't explain the 1,713 short-form requests.
  • Bytespider has no list we could find, so its 1,377 requests can't be checked at all.

Fake AI crawlers FAQ

How do I know if Googlebot is real?

Check the request's address against the list Google publishes for its common crawlers, or run a reverse DNS lookup that ends in googlebot.com or google.com, then a forward lookup that returns the same address. In our week, 117 requests named Googlebot from addresses on none of Google's lists.

Do OpenAI and Anthropic publish crawler IP addresses?

Yes. OpenAI publishes a list for each of GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic publishes one list for its crawlers. Google, Microsoft, Perplexity, Apple, Amazon, Common Crawl and DuckDuckGo publish lists too. We found none for Bytespider.

Why do bots fake AI crawler user agents?

To get past rules that trust those names. In our logs the fakes went for .env files, .git folders and cloud keys, and two addresses switched between 15 crawler names.

Do AI crawlers read llms.txt?

Not on our site. In one week, none of the major AI or search crawlers asked for vandatateam.com's llms.txt. Its 42 requests came from SEO tools, small agents and test browsers.

Should I block AI crawlers?

Only if you don't want AI tools to read or cite your pages. If you do block them, use robots.txt for the honest crawlers and verified addresses for everything else, since the fakes ignore robots.txt.

Tran Tien VanFounder, Van Data Team. Runs Vanaxity, the done-for-you SEO, GEO and AEO content service for B2B companies.Researched and first drafted with AI assistance. Facts, figures and quotes link to their sources in the text.Connect on LinkedInHow we write Insights

Keep reading

View all
3D isometric dark cosmic illustration comparing a fluid topological node map with a rigid crystalline data grid.
A Map for People
AI Search StrategyHTML sitemapXML sitemap
AI Search Strategy · Oct 10, 2026

HTML Sitemap vs XML Sitemap: What Google Just Said

On Google's Search Off the Record podcast, John Mueller and Martin Splitt discussed HTML sitemaps. Mueller said an HTML sitemap is a map of a website for users and doesn't replace an XML sitemap, because it can't be submitted or processed the same way. Crawlers can still follow its links. He suggested listing key categories, not every page. For AI search, plain HTML links matter because most AI crawlers don't run JavaScript, and accurate XML sitemaps with lastmod dates help Bing, which powers many AI answers.

Data engineering behind AI search — SERP snapshots and AI answers turned into entities, citations and structured signals.
Under the hood
AI Search StrategyAI search researchSERP data
AI Search Strategy · Jul 7, 2026

The Data Engineering Behind AI Search Research: How Vanaxity Sees the SERP

A look under the hood at the data engineering that powers Vanaxity's SEO, GEO and AEO research: collecting SERP and answer-engine data, extracting competitor and citation signals from protected sources, and turning it into review-ready reporting.

A classic search results page dissolving into particles that converge on a glowing crystal inside an orbiting network.
Vanaxity launch note
AI Search StrategyVanaxityVan Data Team
AI Search Strategy · Jun 18, 2026

The 10 Blue Links Are Dead: Meet Vanaxity, the AI Agent for SEO, GEO and AEO

A senior-marketer introduction to Vanaxity, why search has moved beyond traditional SEO, and where an AI agent system can help or still needs human judgment.

XstraStar G-Power Framework cover — standardizing AI search visibility metrics for GEO and AEO.
XstraStar G-Power Framework
AI Search StrategyXstraStar G-Power framework: standardizing AI search visibility metrics for GEO and AEOreasoning and acting agents
AI Search Strategy · Jul 3, 2026

XstraStar G-Power Framework: Standardizing AI Search Visibility Metrics for GEO and AEO

How to turn presence in AI answers into a system of scores, signals, and decisions you can track.

GEO and AEO Strategy: Win the Shift From Clicks to Conversations
GEO and AEO Strategy
AI Search StrategyGEO and AEO strategyAI Overviews CTR
AI Search Strategy · Jul 9, 2026

GEO and AEO Strategy: Win the Shift From Clicks to Conversations

A practical framework for marketing teams covering research, answer content, evidence, review points, and reporting signals.

Google Search Agents: What AI Mode Changes for GEO and AEO
Google Search Agents: GEO
AI Search StrategyGoogle Search agentsGenerative UI
AI Search Strategy · Jul 10, 2026

Google Search Agents: What AI Mode Changes for GEO and AEO

How Google Search agents synthesize information, track topics, and generate answers, and the brand data you need to prepare for them.