Free tool · SEO
Robots.txt and XML sitemap checker
Enter a domain to validate robots.txt line by line, see which search and AI crawlers are allowed or blocked, and check that your XML sitemaps are valid, complete and free of errors and redirects.
Robots.txt & Sitemap Checker
What it checks
The two files that guide every crawler
robots.txt tells crawlers where they may go; the XML sitemap tells them which URLs you want found. When the two disagree, or either is broken, crawlers waste time on the wrong URLs or miss the right ones. The checker reads both and compares them.
robots.txt syntax
Each line is parsed the way Google does: unknown directives and likely typos (“Dissallow”), rules outside a user-agent group, paths that do not start with “/”, missing colons, and directives Google ignores such as Crawl-delay and Noindex. Problem lines are highlighted in the file view.
Crawler rules
For Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, GPTBot, ClaudeBot, Google-Extended, CCBot and others: Allowed, Partially restricted, Blocked, or No explicit rule (and what the User-agent: * group means for it).
Important sections
Whether the homepage, the page you entered, the CSS and JavaScript files it loads, and URLs listed in the sitemap are blocked for Googlebot.
Sitemap discovery and validity
Sitemaps declared in robots.txt, then the usual locations; HTTP status, redirects, content type, gzip, XML errors, the sitemaps.org namespace, sitemap indexes and the 50,000-URL limit.
Sitemap contents
URL count, duplicates, URLs on another host, http:// URLs on an HTTPS site, and lastmod dates that are invalid or in the future. Image and video sitemaps are recognized and not counted as duplicates.
Sitemap URL status
Twenty URLs spread across the sitemap are requested to find errors (404, 5xx) and redirects that should not be listed.
Why it matters
One line can hide a whole site
A single “Disallow: /” copied from a staging server blocks every compliant crawler from the entire site. A typo turns an intended rule into a line crawlers ignore. And when robots.txt answers with a server error, Google pauses crawling of the site altogether until it can read the file again.
Sitemaps are a signal of which URLs you consider canonical and worth crawling. Listing URLs that redirect, return errors or are blocked by robots.txt sends mixed signals, and large sites with inaccurate sitemaps see new and updated pages discovered more slowly.
AI crawlers have made robots.txt a business decision as well as a technical one. Providers publish separate user agents for search features, user-requested fetches and model training, so you can allow one and block another. Getting the names right matters: blocking the wrong token can remove a site from AI search results while doing nothing about training.
“No explicit rule” is not a guarantee
It means robots.txt has no group for that crawler, so the crawler follows the User-agent: * group, if any. CDNs, firewalls and bot-protection services can still block it, and robots.txt is a request that well-behaved crawlers follow, not access control.
How to fix problems
Common fixes
- 01Replace blanket blocks with specific paths
Keep crawlers out of internal search results, carts, account pages and faceted filters, not whole sections. Never block CSS and JavaScript that pages need to render.
- 02Separate AI search from AI training
Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot if you want to appear in those search products; add groups for GPTBot, ClaudeBot, Google-Extended or CCBot only if you want to opt out of training.
- 03Declare the sitemap
Add “Sitemap: https://example.com/sitemap.xml” to robots.txt with the full URL.
- 04List only live, canonical URLs
Remove redirected, deleted, noindex and blocked URLs from the sitemap, list each URL once on its canonical host, and set lastmod only when content really changes.
- 05Split large sitemaps
Keep each file under 50,000 URLs and 50 MB uncompressed, and list the files in a sitemap index.
Questions
About robots.txt
and sitemaps
To test a single page path against 14 search, user and training crawlers, use the AI Crawler Checker.
Does robots.txt remove pages from Google?
No. It stops crawling, not indexing. A blocked URL can still be indexed without its content if other pages link to it. To keep a page out of search results, allow crawling and use noindex.
How does Google decide between Allow and Disallow?
The most specific rule, meaning the longest matching path, wins. When an Allow and a Disallow rule are equally long, Allow wins. The checker applies the same logic.
Is a missing robots.txt a problem?
A 404 means crawlers may fetch everything, which is fine for many sites. A 5xx error or timeout is a problem, because Google then stops crawling.
How often do crawlers read my sitemap?
It varies by site. Google reads sitemaps it knows about regularly; declaring the sitemap in robots.txt and submitting it in Search Console and Bing Webmaster Tools lets them find it quickly.
Is anything stored?
Results are cached for 20 minutes, then deleted. Usage is counted anonymously with the host name only.
Need help with what
the tools found?
HUB engineers implement technical SEO, performance, security and AI-readiness fixes on WordPress, Magento, custom PHP and modern JavaScript stacks.