How to check if a page is noindex

A page marked noindex will not appear in search results, however good it is. It is also one of the easiest ways to lose traffic without noticing: a setting is left on after a launch, a plugin adds it, and weeks later someone asks why the page is not ranking.

Where noindex can hide

A noindex instruction can come from three places, and you must check all of them.

1. A meta tag in the HTML

<meta name="robots" content="noindex">

Also watch for noindex, nofollow, and for a tag aimed at one crawler, such as <meta name="googlebot" content="noindex">.

2. An HTTP response header

X-Robots-Tag: noindex

This is invisible in the page source. It is set by the server, a CDN or a plugin, and it is the one people miss most. It is also how PDFs and images are noindexed.

3. JavaScript

If a script adds the meta tag after the page loads, the rendered page has a noindex that the raw HTML does not. Google renders JavaScript, so it can still act on it.

How to check

In the browser. View source and search for robots. Then open DevTools, go to the Network tab, click the page request and read the response headers for X-Robots-Tag.

From a terminal. Read the headers directly:

curl -sI https://example.com/page | grep -i x-robots

Or let a tool check meta tag, header and robots.txt in one go:

curl seoloop.in/indexable/example.com/page

The answer is a plain yes or no plus the reasons. For seoloop.in it returns indexable: yes. A blocked page names the cause, such as a robots meta tag or an X-Robots-Tag header.

In Google Search Console. Paste the URL into the inspection bar. Under Page indexing you will see either "Indexing allowed? Yes" or "No: 'noindex' detected in 'robots' meta tag" (or in the HTTP header). This also shows what Google saw on its last crawl, which may be older than your latest change.

Why pages go noindex by accident

• A staging setting reaches production. Many sites noindex everything before launch and forget to turn it off.
• A CMS option. WordPress has "Discourage search engines from indexing this site" under Settings, Reading. Ticked by mistake, it adds noindex site-wide.
• An SEO plugin. Per-page or per-post-type switches, such as noindex on tags or pages, hide content you wanted indexed.
• A server or CDN rule adding the header to a whole folder.

If a whole site is missing from Google, check this first.

Removing noindex, and the trap

When you remove noindex, Google has to recrawl the page to notice. Request that in Search Console with Request indexing, and make sure the page is not blocked in robots.txt. If it is, Google cannot fetch the page and never sees that noindex was removed. See how to check robots.txt.

Should a page be noindex?

Use it deliberately for pages visitors need but search does not: thank-you pages, internal search results, login pages, thin archives. Everything else should be indexable.

Checklist

• No robots meta noindex in the rendered page
• No X-Robots-Tag: noindex header
• Not blocked in robots.txt
• Search Console says indexing is allowed
• Checked again after every launch or plugin change