Check reference
Three root files that tell crawlers where to go.
Alongside the homepage, every audit requests three files at the root of the final homepage origin: /robots.txt, /sitemap.xml and /llms.txt. Each request follows up to five redirects and reads at most 256,000 bytes. A file passes when it answers with a 2xx status inside that limit. Together the three checks carry 5 of the 90 points. What robots.txt says is scored separately, on the crawler access page.
robots.txt is available
TechnicalLow severity1 of 90 pointsid: robots
Repair shown in the report: Publish a clear robots.txt at the site root.
The rule. Passes when /robots.txt returns any 2xx status. The observed value is the HTTP status, or HTTP 0 when the request failed or the file exceeded 256,000 bytes.
Why it matters. RFC 9309 places the file at /robots.txt, all lowercase, in the top-level path of the site. A missing file blocks nothing: under the RFC, a 4xx response lets a crawler access any resource on the server (RFC 9309). When /robots.txt returns 404, the audit scores the absence at low severity and keeps the crawler checks open. Any other failure, including a 401, 403, 410, 429 or 5xx, a timeout or a file over 256,000 bytes, leaves crawler access unconfirmed, and all four checks on the crawler access page fail. Return 404 when you publish no rules. Google requires the file to be UTF-8 plain text and ignores anything after 500 KiB (Google, robots.txt specification).
How to fix it. Serve a UTF-8 plain-text file at /robots.txt with a 200 status, keep it well under 256,000 bytes, and add a Sitemap line with the absolute sitemap URL.
XML sitemap is available
TechnicalMedium severity3 of 90 pointsid: sitemap
Repair shown in the report: Publish a sitemap.xml that lists canonical public pages.
The rule. Passes when /sitemap.xml returns a 2xx status within 256,000 bytes. The scanner requests that exact path, so a sitemap published elsewhere and declared only in robots.txt does not satisfy it. The evidence summary separately reports whether the file is a urlset or a sitemap index and how many entries it lists. A sitemap index counts child sitemaps, not their pages, and neither count is scored.
Why it matters. Google recommends posting sitemaps at the site root, because a sitemap there can affect all files on the site. It generally shows canonical URLs in its results, and sitemaps help influence which ones (Google, build and submit a sitemap). The protocol caps one sitemap at 50,000 URLs and 50 MB uncompressed; larger sites use a sitemap index (sitemaps.org protocol).
How to fix it. Publish the sitemap, or a sitemap index, at /sitemap.xml. On a large site, make that file an index that lists child sitemaps; it stays small enough to read in full. List only canonical, indexable URLs, and reference the same address from robots.txt.
llms.txt evidence index is available
AI discoveryLow severity1 of 90 pointsid: llms
Repair shown in the report: Publish an accurate llms.txt as an optional plain-text evidence index.
The rule. Passes when /llms.txt returns a 2xx status and its body contains at least one non-whitespace character. The file's structure is not validated.
Why it matters. The llms.txt proposal by Jeremy Howard, published in September 2024 and now at v2, last modified in August 2026, defines a Markdown file that gives agents brief background and links to detailed pages (The /llms.txt file, v2). Its audience is agents that read your site directly, such as a coding agent fetching documentation or a chat assistant reading pages to answer a question. Google states that you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode (Google, AI features and your website). The audit weights the file low.
How to fix it. Follow the proposal's order: an H1 with the site name, which is the only required section, then a blockquote summary, then H2 sections that list links with short notes.
# Example Co
> One sentence that states what the site covers.
## Key pages
- [Pricing](https://www.example.com/pricing): plans and what each includes
- [Documentation](https://www.example.com/docs): setup and API referenceKeep every link pointed at a live page, and update the file when those pages move.
More check references
- Page access and indexing: Status code, HTTPS, noindex, canonical, viewport, language and HTML weight.
- Crawler access: How robots.txt is read for OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot.
- Structured data and entity: Valid JSON-LD, a primary entity type, and sameAs identity links.
- Titles, descriptions and share metadata: Title and description length, Open Graph, X card and favicon.
- Headings, content and links: One H1, two H2 sections, 300 words, three internal links and image alt attributes.
- Retrieval chunk size: Whether every heading section fits in one 375-word retrieval chunk.
Related Suede guides
- What Is llms.txt and Should Your Site Have One?: Suede Signal on what belongs in the file, how crawlers use it, and how to write one.
- llms.txt Implementation and Maintenance: Suede AI creates the file and keeps its contents aligned with your site.
- Indexing and Sitemap Optimization: align canonical URLs, indexing directives and XML sitemaps with a validated URL inventory.