12 AI crawlers checked against your robots.txt in one scan. See which are allowed, which are blocked, and which rules cause each outcome.
Paste a URL and this tool fetches that site's robots.txt and checks it against 12 named AI crawlers, one by one. For each crawler it reports whether the file allows or blocks it, and which specific rule line is responsible, so you can see exactly what your robots.txt is telling AI systems right now.
Not all AI crawlers do the same job, so the tool groups them into three kinds.
CCBot (Common Crawl) gather web pages into datasets used to train language models. Blocking one stops that crawler from adding your content to a future training set.GPTBot (ChatGPT), ClaudeBot (Claude), and PerplexityBot (Perplexity) fetch pages at query time, when someone actually asks a question, to help build the answer. Blocking one of these means that engine simply cannot read your page when it matters.Google-Extended is a separate toggle from regular Googlebot. It governs whether Google can use your content for AI features like AI Overviews, without touching your normal search ranking.If an inference crawler is blocked, the assistant it belongs to cannot fetch your page when someone asks a related question, so you cannot be cited or quoted in that answer, no matter how good the content is. One honest caveat: robots.txt is a request, not a lock. Compliant crawlers respect it, but nothing forces any bot to obey it, so treat this as a strong signal of intent, not a guarantee of behavior.
That depends on your goals. Blocking training crawlers (CCBot) is common and has no impact on AI search visibility. Blocking inference crawlers (GPTBot, ClaudeBot, PerplexityBot) means those AI engines cannot read your pages when generating answers, which reduces your chance of being cited. Blocking Googlebot-related crawlers like Google-Extended affects whether Google's AI overview can reference your content.
Training crawlers collect data to train AI models (CCBot for Common Crawl, used by many LLMs). Inference crawlers fetch pages at query time to produce answers (GPTBot for ChatGPT, PerplexityBot for Perplexity, ClaudeBot for Claude). Blocking training crawlers prevents future model training. Blocking inference crawlers prevents the AI from citing your page in live answers.
Yes, if the Disallow rule covers the paths you care about. User-agent: * with Disallow: / blocks all crawlers, including all AI bots, from reading your entire site. Some sites use this accidentally, or inherit it from old WordPress defaults.
This tool reads the robots.txt file and reports the stated rules. In practice, compliance is voluntary. Most major AI companies (OpenAI, Anthropic, Google) say they respect robots.txt, but nothing enforces it. The tool tells you what the file says, not whether every crawler obeys it.