Excluded by Noindex Tag: You Told Google Not To

Google Search Console has a report called Page Indexing, and one of its rows tells you a page was kept out of Google because of a noindex tag. That row is different from every other row in the report. The others are all Google making a decision about your page. This one is Google doing exactly what you asked.

A noindex rule is a one-line instruction that tells search engines not to store a page. Here are three URLs that all return 200, all load perfectly, and all sit in this bucket:

https://example.com/staging/new-homepage/
https://example.com/blog/tag/marketing/
https://example.com/checkout/thank-you/

A staging page nobody remembered to unblock, an archive page the theme blocks by default, and a thank-you page somebody blocked deliberately. Only one of those three is a problem.

Most advice for this status treats the whole bucket as a mess to clear out. That instinct is how people end up indexing their own checkout pages. The job here is not emptying the bucket. It is sorting it.

Where you are seeing this

If you landed here from an error string and are not sure which screen it came from, it is this one.

• Open Google Search Console and pick your property.
• In the left sidebar, under Indexing, click Pages.
• Scroll to the table headed “Why pages aren’t indexed”.
• The row you want mentions a noindex tag. Click it to see example URLs.

You can also get here one URL at a time. Paste any address into the search bar at the top of Search Console and the URL Inspection tool reports the same status for that single page.

The report and the API call it two different things

This is worth knowing before you go looking for documentation, because it will waste an afternoon otherwise.

Google’s own Page Indexing report documentation lists this status as URL marked ‘noindex’, and describes it as Google encountering a noindex directive and therefore not indexing the page. Search that help page for the phrase “excluded by” and you get nothing.

Now inspect a real URL through the URL Inspection API. Here is an abbreviated response from a page on this site, pulled while writing this:

{
  "coverageState": "Excluded by ‘noindex’ tag",
  "indexingState": "BLOCKED_BY_META_TAG",
  "robotsTxtState": "ALLOWED",
  "pageFetchState": "SUCCESSFUL",
  "verdict": "NEUTRAL"
}

Same status, different words. The help center says “URL marked ‘noindex’” and the API says “Excluded by ‘noindex’ tag”.

That matters for two practical reasons. If you searched the API’s wording and could not find it in Google’s docs, this is why, and you were not looking in the wrong place. And if you are writing a script that filters inspection results, match the API string, because that is the only one the API will ever return.

Watch the quotes

There is a smaller trap inside that string. Google wraps the word noindex in curly quotes, ‘ and ’, not the straight ' your keyboard types. A filter written with straight quotes matches nothing and reports zero blocked pages, which reads like good news and is not. Match on the word noindex alone, or on the indexingState field, and the problem goes away.

What the status actually means

Google reached your page, read it, found an instruction telling search engines not to store it, and complied.

Google’s documentation on blocking indexing describes noindex as a rule that keeps a page out of search results when Googlebot crawls it and parses the rule. Nothing about that is an error state. The page can return 200, load fast, have perfect internal links and sit in this bucket forever, because the rule outranks all of it.

Two things follow. The rest of your site is unaffected, so this is not a penalty leaking outward. And no amount of content work moves the page, because the block is a directive rather than a judgment. Rewriting a noindexed page is the single most common wasted afternoon on this status.

Meta tag, or HTTP header?

The coverage string tells you a noindex rule exists. It does not tell you where. The indexingState field does, and this is the most useful field in the whole response.

Google’s API reference defines these values:

• INDEXING_ALLOWED: no noindex rule found.
• BLOCKED_BY_META_TAG: noindex detected in the robots meta tag, which lives in your page’s HTML.
• BLOCKED_BY_HTTP_HEADER: noindex detected in an X-Robots-Tag HTTP header, which lives in your server or CDN config.

Those are two completely different investigations. A meta tag is in a template, a layout file or a CMS field. A header is in your web server rules, your hosting platform, or an edge config you may not have written. People lose hours grepping templates for a rule that was never in the HTML.

The two forms Google accepts look like this:

<meta name="robots" content="noindex">

X-Robots-Tag: noindex

The robots meta tag specification covers both, along with the googlebot-only variant that blocks Google while leaving other engines alone.

Five ways an accidental noindex happens

Almost every unintentional one comes from a short list.

1. A staging site went live with its blocks intact. The most expensive version of this. The whole site is in the bucket and traffic never arrives.
2. A CMS privacy toggle. WordPress has a “Discourage search engines from indexing this site” checkbox in Reading settings. It is one click and it noindexes everything.
3. A per-page switch somebody flipped. Squarespace, Shopify, Webflow and most SEO plugins put a hide-from-search control on each page. It is easy to set and invisible afterwards.
4. A template default. Themes routinely noindex tag pages, author archives, paginated results and internal search pages. Nobody chose it. It shipped that way.
5. An edge or hosting rule. Preview deployments, password-protected environments and some CDN configurations add X-Robots-Tag: noindex at the network layer, where no amount of template searching will find it.

The pattern is that four of those five were set once, by someone who is not looking at this report today.

Most of this bucket is supposed to be there

Here is the opinion, and it runs against most of the advice written about this status.

On a healthy site, this row should have plenty of pages in it, and the count going up is not automatically bad news. Noindex is a tool, not a failure. Filtered product views, tag archives, thank-you pages, internal search results, thin location variants and staging URLs all deserve to be there. Leaving them out of the index protects the pages you actually care about.

This site is a live example. It runs a directory of MCP server pages, and 80 of its 109 server pages carry a noindex rule on purpose. They rank in the thirties and above for terms we cannot win, and they were adding thin pages to the index for no return. Blocking them was a deliberate decision, recorded in the code, and it shows up in this exact report as a large number that is entirely correct.

So the question to ask is never “how do I empty this bucket”. It is “is anything in here that I wanted people to find”. Usually the answer is a handful of URLs, and the rest is the system working.

The robots.txt trap

One mistake on this status is worth calling out on its own, because it makes things worse in a way that is genuinely hard to undo.

The instinct, once you decide a page should stay out of Google, is to block it in robots.txt as well. Belt and braces. It does the opposite.

Google’s documentation is unusually blunt about it. For the noindex rule to work, the page must not be blocked by a robots.txt file, and it must be reachable by the crawler. If robots.txt blocks it, the crawler never sees the noindex rule, and the page can still appear in search results.

You have then built something with no exit. The page can rank, you cannot remove it with noindex because the rule is unreachable, and the fix is to unblock the page so Google can crawl it and read the instruction you left. Pick one mechanism. If you want a page out of the index, let Google in to read the rule.

Sorting deliberate from accidental

Three questions, in this order, and they take about a minute per URL.

Did I intend this? Look at the URL, not the page. A path with /tag/, /staging/, /search?, /cart/ or a query string in it is almost always meant to be here. A clean article or product URL is not.

Where is the rule? Check the indexing state. A meta tag on a page whose siblings are indexed points at a per-page setting somebody flipped. A header, or a meta tag across an entire section, points at config.

Would I have missed the traffic? If the page has never had a job in search, leave it. The work is only worth doing for pages you would have wanted people to land on.

Anything that answers no, no and yes is a real problem. Everything else is fine where it is.

Running the check across a whole site

One warning before planning an afternoon around this.

The report caps its examples. Google states the list “is limited to 1,000 items, and isn’t guaranteed to show all URLs in a given status, even when less than 1,000 items”. So the number on the row and the URLs you can actually open are two different things.

No API closes that gap. The Search Console API has four resources, covering search analytics, sitemaps, sites and URL inspection. None of them returns a list of URLs filtered by index status. The per-reason lists exist in the interface and nowhere else.

What you can do is work the other way around. Build your own URL list from a sitemap or a crawl, inspect each one, and keep the results whose indexing state comes back blocked. Per-URL truth is available even when the per-status list is not, and the published quota is 2,000 inspections per property per day.

Doing that as a script means writing and maintaining a script. Doing it in an agent means typing a sentence.

Copy this prompt

Read my sitemap and inspect every URL against Search Console. List
only the URLs blocked by a noindex rule. For each one, show whether
the block came from the meta tag or the HTTP header, and the last
crawl date. Group the meta tag ones and the header ones separately.

Grouping by source is the part that saves the time. The header group is a config problem you fix in one place. The meta tag group is a page-by-page problem, and it is usually much shorter than people expect. This is the same move as running a full site audit that way, pointed at one field.

How this differs from the other not-indexed statuses

Four rows in the same report look similar and have nothing in common underneath. The closest neighbor is crawled, currently not indexed, and mixing the two up is the most common way people fix the wrong thing. The two canonical rows are a separate pair, split by whether you declared one at all, and the harder of them is duplicate, Google chose different canonical than user, where Google read your tag and picked something else. There is one more block you caused yourself, and it is the hardest to find, because blocked due to access forbidden is a refusal that happens in your firewall rather than anywhere you can grep for it.

Only the first row has a switch you control directly. That is why it is the one worth checking first, and the one most likely to be resolved the same day.

Where the data comes from

This diagnosis needs two things. Your live index status, which Google gives you free, and somewhere for the readings to pile up, because a bucket of 200 URLs tells you nothing until you know whether it was 40 last month.

ContextBolt SEO is the tool I build. It puts a read-only Search Console connection and a full SEO toolkit inside the agent you already work in, so inspecting three hundred URLs is a sentence rather than three hundred tabs. The Search Console tools are free and never spend your monthly research credits. It is $35 a month with a 7-day free trial, and there is no dashboard to work in, because every lookup lands on a board that fills itself while you work somewhere else.

The reason that matters on this status specifically is the second reading. Removing a noindex rule does nothing until Google recrawls, and Google warns that a revisit can take months on a page it does not consider important. One reading tells you the rule is gone. A reading a few weeks later tells you Google noticed.

What I would actually do

Open the row and read the URL list before touching anything.

Sort it into two piles on the paths alone. Anything with a tag, filter, staging, cart or search path goes in the leave-alone pile, and that pile is normally most of the list. Do not open those pages. They are doing their job.

Take what is left, which on most sites is a handful of URLs, and inspect each one. Read the indexing state, not just the coverage state. If they all come back as headers, stop and go look at your hosting config, because it is one fix and not five. If they come back as meta tags on otherwise normal pages, somebody flipped a per-page switch and you are looking for a pattern in who published them.

Then remove the rule, leave the page crawlable, and request indexing on the ones that matter.

The thing to resist is the urge to make the number smaller. A noindex rule is the only instruction in this whole report that Google follows without arguing. That is worth using well, not clearing out.