Noindex and X-Robots-Tag checker
Find out whether Google and Bing can index a URL. We read the X-Robots-Tag header, every meta robots tag, robots.txt, the canonical and the redirect chain, and tell you exactly which one is keeping the page out of search.
Free, no signup. Checks one page.
What this tool checks
- X-Robots-Tag HTTP headers, including bot-specific values like "googlebot: noindex" and multiple headers
- Meta robots tags: <meta name="robots">, <meta name="googlebot"> and <meta name="bingbot">
- A separate verdict for Googlebot, Bingbot and other search engines
- robots.txt rules for the URL, matched the way Google does (most specific group, longest rule, * and $ wildcards)
- A noindex hidden behind a robots.txt block, which Google can never see
- The canonical, from the link tag or the Link header, and whether it points elsewhere
- HTTP status, every redirect hop, nofollow, nosnippet and expired unavailable_after dates
- Works on PDFs and other files too, which can only be noindexed with the X-Robots-Tag header
Where a noindex can hide
A page can be kept out of search in two places: a meta robots tag in the HTML, or an X-Robots-Tag header the server sends with the response. The header is invisible when you view the page source, so a noindex added by a server rule, a CDN or a staging config is easy to miss for months.
Directives can also target one search engine. "googlebot: noindex" in the header, or <meta name="googlebot" content="noindex">, removes the page from Google while Bing still shows it. When signals conflict, search engines apply the most restrictive one, so a single stray noindex wins over any number of "index" values.
robots.txt works differently: it stops crawling, not indexing. If robots.txt blocks a URL, Google never downloads the page and never sees its noindex, so the bare URL can stay in results. To remove a page, let it be crawled with noindex until it drops out.
Frequently asked questions
How do I check if a page is noindex?
Enter the URL above. The checker reads both the meta robots tags in the HTML and the X-Robots-Tag header, and tells you which one sets noindex and for which search engine. By hand, you can view the page source and search for "noindex", but that misses the header; use your browser's developer tools (Network tab, response headers) or curl -I to see it.
What is the X-Robots-Tag?
An HTTP response header that carries the same directives as the meta robots tag, such as noindex, nofollow and nosnippet. It's set in your server, CDN or application config, and it's the only way to noindex files that have no HTML, such as PDFs, images and feeds.
How do I check the X-Robots-Tag header?
Run the URL through this checker, which lists every X-Robots-Tag header the server sent, including on redirect hops. From a terminal, curl -sI https://example.com/page | grep -i x-robots-tag shows it too.
Why is my page not indexed by Google?
The most common technical causes are a noindex (meta tag or X-Robots-Tag header), a robots.txt block, a canonical pointing to another URL, or an error status code. This checker tests all four. If none apply, the page may be new, thin or duplicated; check the URL Inspection tool in Google Search Console for Google's own answer.
Should I use noindex or robots.txt to remove a page from Google?
noindex. robots.txt only stops crawling, and a blocked URL can stay indexed if other pages link to it. Add noindex and keep the page crawlable until it disappears from results.
Does Google see a noindex added with JavaScript?
Sometimes, but don't rely on it. If the original HTML says noindex, Google may skip rendering entirely, so removing it with JavaScript doesn't work. Adding noindex with JavaScript can work, but it's delayed until rendering. This checker reads the HTML the server sends, without running JavaScript.
Does noindex stop AI crawlers?
Not reliably. noindex keeps a page out of search engine results, including Google's AI Overviews, but AI crawlers like GPTBot and ClaudeBot follow robots.txt rules, not noindex. Use robots.txt to control them; our AI Crawler Checker shows which ones you allow.
More free SEO tools
- Business Idea FinderFind a profitable business idea in your city for free.
- AI Bot Log AnalyzerUpload your server access log and see which AI bots crawl your site: GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and more.
- AI Crawler CheckerFree check: can ChatGPT, Claude, Perplexity and Gemini crawlers reach your site? Tests robots.txt rules per AI bot, firewall and CDN blocks, and llms.txt..
- Redirect CheckerFree redirect checker: follow every hop with status codes and timing, find redirect chains, loops and meta refresh, and test http, https, www and non-www..
- Domain Age CheckerFree domain age checker: see when a domain was registered, when it expires, its registrar and when it first appeared online, from official registry data..
- Twitter Card & Open Graph ValidatorFree Twitter card validator and Open Graph checker: preview how a link looks on X, Facebook and LinkedIn, and check og:image size, twitter:card and missing tags..
- Favicon CheckerFree favicon checker: find every favicon a site declares, check that each loads, test it against Google's favicon rules for search results and preview it..
- Meta Description CheckerCheck any page's meta description and title tag length, plus canonical, Open Graph and Twitter Card tags.
- Robots.txt CheckerFetch and test any site's robots.txt.
- Structured Data CheckerFree structured data checker: see which schema.org markup a page has (JSON-LD, Microdata, RDFa), find invalid JSON-LD, and get the types you're missing..
- H1 & Heading CheckerCheck the H1, H2 and other heading tags on any page.
- AI crawler listUser agents, IP ranges and robots.txt rules for GPTBot, ClaudeBot, PerplexityBot and every other AI bot.