verify AI crawlers

Vanaxity insights tagged verify AI crawlers - notes on AI content, SEO, GEO and AEO from Van Data Team.

3D illustration of a neon path of cyan light motes running across a dark mirror floor and through an arch of cyan light, while a dim amber branch leads three amber motes wearing dark visors to a glowing amber barrier where they stop.
Fake AI crawlers
AI Search Strategyfake AI crawlersverify AI crawlers
AI Search Strategy · Nov 16, 2026

Fake AI Crawlers: How We Verified a Week of Bot Traffic

We took one week of request logs from vandatateam.com, the site of Van Data Team, the company behind Vanaxity, and checked every request that named a crawler against the address list its operator publishes. From November 9 to 15, 2026, 23,962 requests named one of 15 crawlers with a published list, from Googlebot and Bingbot to OpenAI's, Anthropic's and Perplexity's bots. 21,303 of them (88.9%) came from the operator's addresses and 2,659 (11.1%) did not. Seven addresses sent 627 of the outsiders while switching between as many as 15 crawler names, and 321 requests went to paths only a scanner asks for, such as .env files and .git folders. Verified crawlers asked for none of those files. They read mostly our AI model benchmark pages, Meta's crawlers read 1,086 different pages in the week, and no major AI crawler asked for our llms.txt. A user agent is only a claim, and the address list is how you check it.