Robots.txt checker
Fetch a site's robots.txt and check it for rules that block search engines, a missing sitemap reference, and noindex directives on the page itself.
Free, no signup. Checks one page.
What this tool checks
- Whether /robots.txt exists
- Rules that block all crawling (Disallow: /)
- Unusually many Disallow rules
- A Sitemap: line pointing search engines to your XML sitemap
- noindex or nofollow in the page's meta robots tag or X-Robots-Tag header
robots.txt controls crawling, not indexing
robots.txt tells crawlers which URLs they may fetch. It doesn't remove pages from search results: a blocked URL can still be indexed if other sites link to it, just without a description.
To keep a page out of search results, let it be crawled and add a noindex meta tag or X-Robots-Tag header. If robots.txt blocks the page, search engines never see the noindex.
Frequently asked questions
Where does robots.txt go?
At the root of the host, for example https://example.com/robots.txt. Each subdomain needs its own file.
Do I need a robots.txt file?
It's not required, but it's recommended. Without one, crawlers assume everything is allowed, and you lose an easy place to point them to your sitemap.
Why is my page still indexed after I blocked it in robots.txt?
Blocking crawling stops Google reading the page, including any noindex tag on it. Remove the robots.txt rule and add noindex instead, then wait for Google to recrawl.
Should robots.txt list my sitemap?
Yes. A "Sitemap: https://example.com/sitemap.xml" line helps every search engine find it, not just the ones where you submitted it manually.
More free SEO tools
- AI Crawler CheckerFree check: can ChatGPT, Claude, Perplexity and Gemini crawlers reach your site? Tests robots.txt rules per AI bot, firewall and CDN blocks, and llms.txt..
- Meta Tag CheckerCheck any page's title tag, meta description, canonical, Open Graph and Twitter Card tags.
- Schema Markup CheckerSee which schema.org structured data a page has (JSON-LD, Microdata, RDFa), find invalid JSON-LD, and get the markup types you're missing..
- Heading CheckerSee the H1–H6 outline of any page.