How to Check If AI Crawlers Can Read Your Site
The short answer
To check if AI crawlers can read your site, inspect your live robots.txt for Disallow rules on AI user-agents, then test an actual fetch using each agent (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). A page can load in a browser but return 403 to a bot if a CDN or WAF blocks it. Verify both layers.
Why this is the first thing to check
If AI crawlers can't fetch your pages, no amount of content or schema will get you cited. Crawler blocks are common and silent — your site looks fine to humans while being invisible to engines.
It's also the single most common reason a brand is missing from AI answers. Because the block returns a normal-looking site to your own browser, teams spend weeks improving content that the engines will never see. Rule this out before anything else on the AEO audit checklist.
Step 1: read your robots.txt
- ▪Open yoursite.com/robots.txt.
- ▪Search for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended.
- ▪Any Disallow: / under those agents blocks the engine.
Step 2: test a real fetch
robots.txt is advisory; your CDN or WAF can hard-block bots regardless. Send a request with each AI user-agent and confirm a 200, not a 403 or challenge. Cloudflare, in particular, can inject AI-bot blocks that override your own robots file.
You can do this from a terminal with curl by setting the user-agent header to, say, GPTBot, and reading the status code. Test a real content page, not just the homepage — some WAF rules only trigger on deeper paths. A 403, 429, or an interstitial challenge all mean the crawler is blocked.
Step 3: fix and verify
Remove Disallow rules and whitelist the agents in your CDN/WAF. Then re-test. WebCited automates both checks — robots.txt parsing plus a live user-agent probe — for every major AI crawler, and flags exactly which engine is blocked. If you find GPTBot specifically blocked, follow how to fix robots.txt blocking GPTBot.
Frequently asked questions
+Can a site be blocked even if robots.txt allows AI bots?
Yes. A CDN or WAF (like Cloudflare's AI-bot controls) can block crawlers at the edge regardless of robots.txt. Always test an actual fetch with each user-agent, not just read the robots file.
+Does blocking AI crawlers help with anything?
It keeps your content out of AI training and answers, which a few publishers want for licensing or copyright reasons. But if discovery and citations matter to you, blocking is self-defeating — you can't be cited by an engine that can't read you. Decide deliberately rather than leaving a default block in place.
+Which AI crawlers should I allow?
At minimum GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), and Google-Extended (Google AI). Allowing them lets the major answer engines fetch and potentially cite your content.
+How do I test a specific crawler quickly?
Send an HTTP request with that crawler's user-agent string and check the status code, or use an AEO scanner that runs live user-agent probes for all major bots automatically.
