← All checks

Check reference

Which crawlers can reach your homepage.

Four of the ten high-severity checks read one file: robots.txt. Together they are worth 20 of the 90 points. With the 1-point check that robots.txt is available, 21 of the 90 points rest on that one file, more than on sitemap.xml (3 points) or llms.txt (1 point). Each check asks whether a named search crawler may fetch the root path of your site. Three crawlers that control model training or AI use are reported beside them and never change the score.

The four crawlers that count

OpenAI search crawler can access the homepage

AI discoveryHigh severity5 of 90 pointsid: crawler-openai

Repair shown in the report: Update robots.txt so OAI-SearchBot can access the public pages you want discovered.

OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. It recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges, and notes that a robots.txt change can take about 24 hours to reach its systems. GPTBot is a separate, independent setting for training (OpenAI, overview of OpenAI crawlers).

Claude search crawler can access the homepage

AI discoveryHigh severity5 of 90 pointsid: crawler-anthropic

Repair shown in the report: Update robots.txt so Claude-SearchBot can access the public pages you want discovered.

Anthropic says disabling Claude-SearchBot prevents its system from indexing your content for search optimization, which may reduce your visibility and accuracy in user search results. ClaudeBot, the training crawler, and Claude-User, the user-initiated fetcher, are separate tokens (Anthropic, does Anthropic crawl data from the web).

Perplexity crawler can access the homepage

AI discoveryHigh severity5 of 90 pointsid: crawler-perplexity

Repair shown in the report: Update robots.txt so PerplexityBot can access the public pages you want discovered.

Perplexity says PerplexityBot surfaces and links websites in Perplexity search results and is not used to crawl content for AI foundation models. It recommends allowing the bot in robots.txt and permitting its published IP ranges, and says changes can take up to 24 hours (Perplexity, Perplexity crawlers).

Google discovery crawler can access the homepage

AI discoveryHigh severity5 of 90 pointsid: crawler-google

Repair shown in the report: Update robots.txt so Googlebot can access the public pages you want discovered.

To be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible for a snippet in Google Search. Google's first listed fundamental is ensuring that crawling is allowed in robots.txt and by any CDN or hosting infrastructure (Google, AI features and your website).

How the scanner reads robots.txt

1. Fetch /robots.txt at the root of the final homepage origin, following up to five redirects and reading at most 256,000 bytes.
2. Split the file into groups: one or more User-agent lines followed by Allow and Disallow rules. Comments and other fields, such as Sitemap and Crawl-delay, are ignored.
3. Select every group whose User-agent equals the crawler's token, ignoring case, and combine their rules. If none match, use the groups for *.
4. Test the root path /. A rule applies when / starts with the rule's path, read up to its first * or $. So Disallow: /, Disallow: /* and Disallow: * all apply, while Disallow: /admin does not. Empty Disallow lines are ignored.
5. The longest applicable rule, as written, wins. On a tie, Allow beats Disallow.
6. A winning Disallow reports Blocked and fails the check. Anything else reports Open and passes.

Steps 3 and 5 follow RFC 9309, which tells crawlers to obey the matching group or else the * group, to use the most specific match, and to prefer Allow when an Allow and a Disallow rule are equivalent (RFC 9309, Robots Exclusion Protocol).

When robots.txt is missing or unreadable

• 2xx: the rules are applied as above.
• 404: no file is published, so all four checks pass as Open. The separate robots.txt availability check on the discovery files page fails.
• Anything else, including 401, 403, 410, 429, a 5xx, a timeout, too many redirects or a file over 256,000 bytes: the report states that robots.txt could not be inspected, so crawler access is not confirmed. Policy shows as Unknown and all four checks fail.

RFC 9309 tells crawlers to assume complete disallow when robots.txt is unreachable because of a server error. Google stops crawling a site for the first 12 hours when its robots.txt returns a 5xx, then falls back to the last good copy for up to 30 days (Google, how Google interprets the robots.txt specification). Answer with 200 and your rules, or 404 when you publish none.

Training and AI-use controls, reported without scoring

The crawler section of the report also shows three tokens read with the same rule: GPTBot, ClaudeBot and Google-Extended. They never change the score, so a training opt-out never lowers readiness.

• GPTBot: OpenAI says disallowing it indicates your content should not be used in training generative AI foundation models.
• ClaudeBot: Anthropic says restricting it signals that your future materials should be excluded from its model training datasets.
• Google-Extended: controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion in Google Search and is not a ranking signal (Google's common crawlers).

How to fix a blocked crawler

Give the search crawlers their own group and keep training controls separate:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

A crawler with its own group ignores the * group entirely, so repeat any Disallow lines it still needs inside the named group. Robots.txt is one layer: a firewall or CDN rule that rejects a crawler's published IP ranges blocks it even when robots.txt allows it. This audit reads robots.txt, so review firewall rules in the same pass.

More check references

• Page access and indexing: Status code, HTTPS, noindex, canonical, viewport, language and HTML weight.
• Discovery files: Whether robots.txt, sitemap.xml and llms.txt answer at the site root.
• Structured data and entity: Valid JSON-LD, a primary entity type, and sameAs identity links.
• Titles, descriptions and share metadata: Title and description length, Open Graph, X card and favicon.
• Headings, content and links: One H1, two H2 sections, 300 words, three internal links and image alt attributes.
• Retrieval chunk size: Whether every heading section fits in one 375-word retrieval chunk.

Related Suede guides

• Which AI Crawlers Should You Allow?: Suede Signal on which AI crawlers feed training or live answers, and how to configure robots.txt to keep answer visibility while limiting model training.
• Search Crawler Access Repair Services: Suede AI reviews crawler rules and HTTP responses, ships agreed repairs and verifies the resulting access.

Every rule on this page is read from the scoring code in lib/engine.mjs. If the code changes, this reference changes with it.