AI Crawler Access Checker
Nine crawlers decide most AI-search visibility. If your robots.txt blocks them, AI engines answer questions about your category from other sources — usually your competitors. Check your verdict per crawler.
The nine crawlers
GPTBot / OAI-SearchBot / ChatGPT-User feed OpenAI training, search and live browsing · ClaudeBot / anthropic-ai feed Anthropic · PerplexityBot powers Perplexity's citations · Google-Extended gates Gemini training · Bytespider feeds ByteDance (Doubao) · CCBot builds Common Crawl, the corpus behind many models.
FAQ
What happens if I block AI crawlers?
They can't read your content, so AI engines describe your category using other sources — usually competitors. If AI visibility matters, a blanket block is the most damaging robots.txt mistake.
Should I ever block them?
Some publishers block training crawlers (CCBot, Bytespider) while allowing citation crawlers (OAI-SearchBot, PerplexityBot) — a legitimate trade-off between training-data control and answer visibility. Just make it a decision, not an accident.
Go deeper
Crawler access is stage zero. The full loop — what AI engines actually say about you across 18 engines, and machine-verified fix tickets — is one command: npx fastergeo