AI crawlers
Vanaxity insights tagged AI crawlers - notes on AI content, SEO, GEO and AEO from Van Data Team.
2 articles
HTML Sitemap vs XML Sitemap: What Google Just Said
On Google's Search Off the Record podcast, John Mueller and Martin Splitt discussed HTML sitemaps. Mueller said an HTML sitemap is a map of a website for users and doesn't replace an XML sitemap, because it can't be submitted or processed the same way. Crawlers can still follow its links. He suggested listing key categories, not every page. For AI search, plain HTML links matter because most AI crawlers don't run JavaScript, and accurate XML sitemaps with lastmod dates help Bing, which powers many AI answers.
Technical SEO Debt Is Now an AI Search Visibility Problem
Technical SEO debt used to cost you crawl budget and rankings, but it now costs AI search visibility too, because testing by Vercel and MERJ found that none of the major AI crawlers execute JavaScript, so anything rendered in the browser is invisible to ChatGPT, Claude, and Perplexity, and since those crawlers already fetch at roughly a quarter of Googlebot's volume, the practical answer is to reserve a fixed slice of every sprint for debt paydown, prioritize by revenue impact rather than audit severity, and check what bots actually see rather than what your browser shows.