AI Crawler & llms.txt Checker

What this checker looks at

It reads your robots.txt the way crawlers must under RFC 9309, then tests 26 AI crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon and others against your home page, and against any page you enter. It also reads the page's meta robots tags and X-Robots-Tag header, your llms.txt and your sitemap. What each crawler does comes from its company's own documentation, linked in the table.

How robots.txt decides

• A crawler obeys the groups that name it, merged together. Only when no group names it does it follow User-agent: *.
• Of the rules that match an address, the longest wins. If an Allow and a Disallow are the same length, Allow wins.
• * matches any characters and $ marks the end of the address.
• A robots.txt that returns a server error (5xx) means "block everything"; a 404 means "everything allowed".

A worked example

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

GPTBot is named, so it reads only its own group and "/" is blocked. OAI-SearchBot isn't named, so it follows the * group: it can read everything except /admin/. You stay in ChatGPT search answers, and OpenAI is asked not to train on your pages.

Search crawlers and training crawlers: the trade-off

Blocking training crawlers such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot keeps your future content out of model training. It doesn't remove you from AI answers, which come from separate search crawlers like OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking those removes you from the answers themselves, and for a business that wants enquiries, being quoted usually beats being absent. That's why the score counts search crawlers and ignores training ones.

Google is the exception: AI Overviews and AI Mode come from Googlebot, so you can't leave them without leaving Google Search. And robots.txt is a request, not a lock. Fetchers that act for a person may ignore it, and a firewall or CDN bot setting can block crawlers whatever robots.txt says.

New this year

A shared opt-out, User-agent: __usage_ai-training with Disallow: /, was proposed in an IETF draft on 30 October 2026; Bing reads it from 1 February 2027. Bing is also replacing noarchive and nocache with ai-training and ai-grounding values, and stops reading the old tags for AI after 1 March 2027. The checker reports both.

Common mistakes

• Blocking Googlebot to stay out of AI Overviews, which removes the site from Google entirely.
• Naming crawlers that don't exist, such as ChatGPT, anthropic-ai or Claude-Web.
• Leaving an old Allow: / group for a crawler you now block: the groups merge and the tie goes to Allow.
• A robots.txt that answers with an error or an HTML page.

When it's worth getting help

If crawlers are allowed but you still don't show up in AI answers, the cause is usually content and structure, not robots.txt. Our technical SEO service fixes both; the AI search optimisation guide explains what gets a page quoted. While you're here, check your email records and your site speed.

Sources

OpenAI, Anthropic, Perplexity, Google's crawlers and AI features pages, Bing's October 2026 AI preferences and 2023 noarchive posts, Apple, Meta, Amazon, Mistral, DuckDuckGo, Common Crawl, AI2, RFC 9309, the training-preference draft, llmstxt.org and Cloudflare's Content Signals. All read on 9 November 2026.