Robots.txt validator
Most robots.txt tools were built before AI crawlers existed. This one tells you in plain language whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest can actually read your site, plus the classic Googlebot and Bingbot checks.
Fetching and parsing robots.txt...
AI crawler access
Classic search crawlers
Findings
Sitemaps declared
Raw robots.txt
The AI crawlers we check
GPTBot, OAI-SearchBot, ChatGPT-User
OpenAI's family. GPTBot trains models. OAI-SearchBot powers ChatGPT search. ChatGPT-User fetches pages when a user clicks a link in chat. Blocking one is not the same as blocking all three.
ClaudeBot, anthropic-ai
Anthropic's bots. The current ClaudeBot is the one to allow if you want to be cited by Claude in 2026.
PerplexityBot, Perplexity-User
Perplexity's crawlers. Perplexity-User fetches pages in response to a real-time query, similar to ChatGPT-User.
Google-Extended
Google's opt-out for Gemini training. Blocking it does NOT affect Google Search ranking, but does opt you out of being used as training data.
Common mistakes we look for
Disallow rules missing a leading slash, conflicting User-agent blocks, sitemaps that don't match the host, blocking Googlebot accidentally with a broad pattern.
Want an AI-crawler policy review at scale?
Part of our technical SEO audit. Includes log-file analysis to see who actually crawls you versus who you think you let in.