Robots.txt Validator and Analyzer

robots.txt is a small text file with a big job: it tells crawlers which parts of your site they may read. One wrong line (a stray Disallow: / left over from staging, a pattern that blocks your CSS, a rule that shuts out an AI crawler) can remove pages from search or AI answers without any visible error on the site.

What the validator checks

• Syntax: unknown directives, misspellings, rules outside a user-agent group and invalid wildcards.
• Googlebot rules: which paths Googlebot may crawl, worked out the way Google does it (the most specific rule wins, and Allow beats Disallow on a tie).
• Crawl blocks that hurt SEO: blocked CSS, JavaScript or image folders that stop Google from rendering pages properly.
• AI crawlers: rules for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others.
• Sitemaps: whether a sitemap is listed and written correctly.
• Security leaks: rules that advertise private paths (robots.txt is public, so it's no way to hide things).
• Your sitemap pages: when you check by URL, every page in your sitemap is tested against the rules, so you see "submitted but blocked" pages before Search Console does.

Written for people who are new to it

Every line of your file is explained in plain words. Every problem says what it is, why it matters and how to fix it. A corrected file is ready to copy, so you don't have to look up what an error means. The robots.txt, llms.txt and .htaccess tools share one tab bar, because they're the 3 files that decide what crawlers and AI tools can reach.

robots.txt isn't noindex

Blocking a URL in robots.txt stops crawling, not indexing: a blocked page can still appear in results if other sites link to it, just without a description. To keep a page out of search, allow crawling and use a noindex robots meta tag instead.