Google Search Console has a report called Page Indexing, and one of its rows says a page was blocked due to access forbidden. It means your server answered Googlebot with an HTTP 403, which is the code for “I know who you are and you cannot have this.”
The confusing part is that the page works. You click the URL, it loads, everything looks right. Here are three addresses that can all sit in this bucket while returning a perfectly good page in your browser:
https://example.com/pricing/
https://example.com/blog/how-we-built-it/
https://example.com/product/blue-widget/Nothing is wrong with those pages. Something in front of them refused one specific visitor, and that visitor was Googlebot. Most guides for this status send you into your site’s code. The rule is almost never there.
- Google says this one is your fault, in writing. Its own documentation calls a 403 to Googlebot an error your server is returning incorrectly.
- The page loading for you proves nothing. A different request was refused, so you cannot test it by visiting the URL.
- Look at the layer in front of your site, not your code. A CDN firewall rule, a bot filter or a rate limit is the usual culprit, and it leaves a log entry naming itself.
- Every 4xx except 429 means “gone” to Google. The page is dropped, not queued, and no amount of content work moves it.
- Diagnosing it means reading what Googlebot got, not what you get. That is a Search Console call, and ContextBolt SEO puts those calls inside the agent you already work in.
Where you are seeing this
If you landed here from an error string and are not sure which screen it came from, it is this one.
- Open Google Search Console and pick your property.
- In the left sidebar, under Indexing, click Pages.
- Scroll to the table headed “Why pages aren’t indexed”.
- The row reads “Blocked due to access forbidden (403)”. Click it for example URLs.
You can also get here one page at a time. Paste any address into the search bar at the top of Search Console and the URL Inspection tool reports the same status for that single URL.
Google’s own wording is unusually blunt
Most statuses in this report describe a judgment Google made. This one describes a mistake it thinks you made, and the help center does not hedge about it.
Google’s Page Indexing report documentation says a 403 means the user agent provided credentials but was not granted access, and then points out that Googlebot never provides credentials, so your server is returning this error incorrectly.
Read that twice. A 403 is the answer to a failed login. Googlebot never attempts one. So whatever your server was replying to, it was not a rejected sign-in, and Google is telling you the response was wrong for the request it made.
The consequence is in a different document. Google’s page on HTTP status codes and network errors says all 4xx errors except 429 are treated the same way, with the crawler telling the next processing system that the content does not exist. Not “try later”. Does not exist.
That is why this status behaves differently from the slow ones. Discovered, currently not indexed is a queue you can move up. A 403 is a deletion.
Why the page loads fine for you
This is where people burn a morning. You open the URL, it renders, and you conclude Search Console is out of date.
It is not. A web request carries more than a URL. It carries an IP address, a user agent string, cookies, a TLS fingerprint and a history of what that IP did in the last few minutes. Anything sitting between the public internet and your server can read all of that and decide per request.
Your request comes from a residential IP, in a real browser, with a session cookie, after you clicked a link on your own site. Googlebot’s request comes from a datacenter IP, with no cookies, no referrer, no JavaScript execution, at whatever rate Google felt like crawling that hour. Those are two completely different visitors, and a security rule is built to tell them apart.
So the page is not broken. Access to it is conditional, and Googlebot fails the condition.
The layer that is usually doing it
A CDN is a network of servers that sits in front of your site, serves cached copies and filters traffic before it ever reaches your host. Cloudflare, Fastly, Akamai and the security add-ons bundled with most managed hosting all work this way.
Four settings in that layer produce a 403 to a crawler, and none of them lives in your codebase.
- A firewall or custom rule. Somebody blocked a country, an ASN, a path or a user agent pattern months ago. Googlebot crawls from Google’s own network, and “block all datacenter traffic” catches it.
- Bot protection. A bot score threshold set aggressively will refuse anything that does not run JavaScript, which is most of a crawl.
- A rate limit. Googlebot ramps its crawl rate up on its own. A limit tuned for human traffic trips the moment it does, and the block persists long enough for a crawl to be recorded as refused.
- A managed ruleset. Vendor-maintained WAF rules update themselves. A rule that starts matching your URL patterns can begin returning 403 without anybody touching a setting.
The fourth one is the reason this status appears on sites where nothing changed. Nothing on your side did.
If you are on Cloudflare, the block leaves a specific fingerprint. Cloudflare’s documentation for Error 1020 says access denied by a firewall rule shows that code in the response body along with a Ray ID, and tells you to search the Security Events log by Ray ID or client IP to find the rule that fired. That log is where this investigation ends, not your templates.
Why you cannot reproduce it with curl
The first instinct is to fake it. Send a request with Googlebot’s user agent string and see what comes back.
curl -A "Googlebot/2.1 (+http://www.google.com/bot.html)" \
-s -o /dev/null -w "%{http_code}\n" https://example.com/pricing/That is one of several Googlebot strings Google publishes, and it is the short one. A user agent rule almost always matches on the word rather than the whole string, so any of them will trip it.
The test is worth thirty seconds and it answers exactly one question. If it returns 403, the block is on the user agent string, and you have found it. If it returns 200, you have learned almost nothing, because the other three causes all key on things you cannot fake from your laptop.
You cannot send from a Google IP. You cannot reproduce Googlebot’s request rate. You cannot match the fingerprint a bot detector is scoring. A 200 here means “not a user agent rule”, and people read it as “not blocked”.
The same gap runs the other way, and it matters for the fix. Google’s guide to verifying Googlebot tells you to run a reverse DNS lookup on the accessing IP and confirm the hostname resolves to googlebot.com, google.com or googleusercontent.com, then run a forward lookup to check it matches. Google also publishes its crawler IP ranges as JSON files you can match against automatically. Both methods exist because the user agent string is public text that anybody can send.
The fix everyone reaches for is the wrong one
Search this status and you will find the same advice on several of the top results. Add a rule allowing any request whose user agent contains “googlebot”, and put it above the rule doing the blocking.
It works. It also hands every scraper on the internet a way past your firewall, and the instructions for using it are the same sentence you just read.
That trade might still be worth making on a brochure site with nothing to protect. It is a bad one on anything with pricing logic, member content, an API or a login. The people most likely to hit this status are the people who turned the security up, which is to say the people with the most to lose from turning it off by name.
Google, to its credit, tells you the right version in the same paragraph where it defines the status. Allow Googlebot requests without authentication after verifying its identity. Verified identity, not a string. Every serious edge platform supports that, either through a built-in verified-bot category or a rule matching Google’s published IP ranges.
Do the ten minutes. The shortcut is a door you will forget you opened.
The one fetch that comes from the real Googlebot
Once you have changed a rule, you need to know whether it worked. There is exactly one check that answers that honestly, and it is not a browser.
Search Console’s URL Inspection tool has a Test Live URL button. Google’s documentation for the tool is explicit that this is a live test and the tool fetches and examines the URL in real time. The request comes from Google’s infrastructure, with Googlebot’s identity, right now. If it comes back with a successful page fetch, your rule change reached the thing that was blocking.
Two details save time here.
Read the Page fetch line, not the verdict. The verdict summarizes indexability. Page fetch answers the only question you have, which is whether Google could get the file at all.
The indexed result and the live result will disagree, and that is correct. The indexed side reflects the last real crawl, which happened while the block was still in place. It stays wrong until Google recrawls. Request indexing once the live test passes, then leave it alone.
Checking it across a whole site
One URL is a diagnosis. The question that actually matters is how many pages are in this state, and a firewall rule rarely hits only one.
The report itself will not tell you. Google caps the example list at 1,000 items and says it is not guaranteed to show every URL in a status even below that number. So the count on the row and the URLs you can open are two different things.
The URL Inspection API closes half of that gap. You can inspect any URL in your property and read its state, at a published quota of 2,000 queries per property per day. Its pageFetchState field is the one to filter on, and the values are named after the statuses. ACCESS_FORBIDDEN is this one. ACCESS_DENIED is the 401 case. BLOCKED_4XX is everything else in the family.
The half it does not close is worth knowing before you plan around it. The API’s own reference says only the status of the version in the Google index is available and you cannot test the indexability of a live URL. So the API sweeps, and the live test proves. Use both, for different jobs.
Doing the sweep as a script means writing and maintaining a script. Doing it in an agent means typing a sentence.
Copy this prompt
Read my sitemap and inspect every URL against Search Console. List
only the URLs whose page fetch failed, and group them by the failure
reason. For each group, show how many URLs and the last crawl date
of the oldest one.Grouping is the part that does the work. Forty URLs sharing one failure reason and one path prefix is a single firewall rule, and you will see the shape of it in the list before you open a single log. This is the same move as running a full site audit that way, pointed at one field.
How this differs from the other blocked statuses
Four statuses in this report all mean Google did not get the page, and they have completely different fixes. Getting them mixed up is how people go looking for a robots.txt rule that was never there. The closest neighbor is excluded by noindex tag, which is also a block you set, just in a place you can actually grep for.
| Status | What the server did | Where to look |
|---|---|---|
| Blocked due to access forbidden (403) | Refused the request outright | CDN, firewall and bot rules |
| Blocked due to unauthorized request (401) | Demanded credentials Googlebot has none of | HTTP auth, staging password, membership gate |
| Blocked due to other 4xx issue | Some other client error | Inspect one URL to get the real code |
| Blocked by robots.txt | Answered normally, but was never asked | One file at your domain root |
The bottom row is the useful contrast. A robots.txt block is a request Google chose not to make. A 403 is a request Google made and your stack refused. Only one of those leaves an entry in your edge logs, and it is this one.
The other-4xx row is the one that changes method. On blocked due to other 4xx issue the block is rarely identity-based, so the curl test that lies to you here usually tells the truth there. That status hides the code instead of hiding the cause.
Do not use a 403 to slow crawling down
A small number of people arrive at this status on purpose, having read that returning errors to a crawler reduces its load on the server.
Google’s status code documentation closes that off directly. Do not use 401 and 403 status codes to limit crawl rate, because 4xx status codes other than 429 have no effect on it.
So the trade is worse than it looks. You do not get less crawling. You get the same crawling, with your pages dropped out of the index while it happens. If crawl load is a genuine problem, 429 is the code that means “too many requests” and the only one in the family Google treats as a rate signal.
Where the data comes from
Diagnosing this needs two readings, not one. What Googlebot got on its last real crawl, and what it gets now that you have changed a rule. Google gives you both for free. The friction was never the data.
ContextBolt SEO is the tool I build. It puts a read-only Search Console connection and a full SEO toolkit inside the agent you already work in, so inspecting three hundred URLs is a sentence rather than three hundred tabs. The Search Console tools are free and never spend your monthly research credits. It is $35 a month with a 7-day free trial, and there is no dashboard to work in, because every lookup lands on a board that fills itself while you work somewhere else.
On this status specifically, the value is the sweep. One URL tells you a rule exists. Three hundred tell you which paths it covers, which is the thing that turns a log-diving session into a five-minute fix.
What I would actually do
Start with one URL from the row and inspect it. Note the last crawl date, because that timestamp is what you are about to search your logs for.
Open your CDN or firewall event log and filter to that window. You are looking for a denied request to that path from a datacenter IP. When you find it, the log names the rule, and that is the whole diagnosis. Nine times out of ten you will recognize the rule and remember why it exists.
Then fix it properly. Add a skip or allow for verified crawlers, above the rule that fired, matched on reverse DNS or Google’s published IP ranges. Not the user agent string.
Run the live test. If the page fetch succeeds, request indexing and move on. If it still fails, you found a second rule, which is common on stacks where hosting security and a CDN are both switched on and neither team knows about the other.
The thing to resist is turning security off to make the number go down. This status is a misrouted refusal, not a reason to open the front door.