This status confuses people more than almost any other, because on the face of it, it is a contradiction. You blocked the URL. Google indexed it anyway. Did it ignore your instruction? No — it followed it precisely, and the outcome you got is exactly what that instruction produces. The confusion comes from a widespread misunderstanding about what robots.txt actually controls, and clearing that up makes the fix obvious.
Crawling and Indexing Are Different Things
This is the whole explanation, so it is worth stating plainly.
- Crawling is Google fetching a URL and reading its content.
- Indexing is Google adding a URL to its searchable database.
robots.txt controls crawling only. It says “do not fetch this.” It says nothing about whether the URL can appear in search results.
Google can index a URL it has never fetched. If it finds enough evidence that the URL exists and is worth listing — usually from links pointing at it — it can add the URL to the index using only that external information. What it produces is a search result with the URL, often a title inferred from anchor text, and a snippet reading something like “No information is available for this page.”
So the status means: Google found links to this URL, decided it was worth indexing, and could not fetch it to learn more because you blocked it.
The Trap: Blocking Prevents Google Reading Your noindex
Here is the specific mistake that causes most cases of this status, and it is worth understanding because the instinct it comes from is entirely reasonable.
Someone wants a page kept out of search results. They add a noindex meta tag. Sensible. Then, to be extra safe, they also disallow the URL in robots.txt.
The result is the opposite of what they wanted. Googlebot can no longer fetch the page. It never sees the noindex tag. The tag is inside a document Google has been forbidden from reading. Meanwhile, links to the URL still tell Google it exists — so it gets indexed.
Blocking and noindexing are mutually exclusive. If you want a page out of the index, Google has to be able to crawl it to see that instruction.
When This Status Is Fine
Before fixing anything, decide whether it matters. Plenty of entries here are harmless:
- Admin and system paths —
/wp-admin/,/cart/,/checkout/. Even if the bare URL appears in results, nobody searches for it and no confidential content is exposed, because Google never fetched the page. - Parameter and filter URLs blocked deliberately to save crawl budget.
- URLs with no meaningful inbound links, which are unlikely to surface for any real query.
The test: would it embarrass you or mislead someone if this URL appeared in search results? If not, leave it. Chasing this report to zero is not a goal.
When It Genuinely Matters
Act when:
- Private or sensitive content is exposed — staging environments, internal documents, customer-specific pages. Note that robots.txt is a public file that anyone can read, so listing sensitive paths in it actively advertises them.
- A content page you want indexed is blocked — an overly broad disallow rule catching real pages. This is the inverse problem and much more damaging.
- Blocked URLs are appearing for brand searches, producing ugly no-snippet results next to your legitimate listings.
- CSS and JavaScript are blocked, which prevents Google rendering pages properly and can affect how it assesses them.
How to Fix It Properly
The correct approach depends on what you are actually trying to achieve.
To Keep a Page Out of Search Results
- Remove the disallow rule from robots.txt so Googlebot can fetch the page.
- Add a noindex directive — either the meta tag:
or the HTTP header, which is the only option for non-HTML files like PDFs:<meta name="robots" content="noindex">X-Robots-Tag: noindex - Wait for a recrawl. Google needs to fetch the page to see the directive. You can speed this up with URL Inspection → Request Indexing, which is slightly counterintuitive but correct — you are asking Google to re-read the page so it discovers the noindex.
- Once it has dropped out, you may re-block it in robots.txt to save crawl budget, if you want. Not before.
To Genuinely Restrict Access
If the content is actually private, neither robots.txt nor noindex is the right tool. Both are public instructions that assume good-faith clients. Use authentication — password protection, IP restriction, or a login requirement. A URL behind auth returns 401 or 403 and cannot be indexed meaningfully.
Staging sites should always be behind HTTP authentication rather than a robots.txt disallow.
To Save Crawl Budget Without Indexing Risk
Blocking in robots.txt is correct here, and this status appearing is an acceptable side effect. To reduce it, cut the number of internal links pointing at those URLs — if your own site does not link to a filtered variant, Google is far less likely to discover and index it.
To Remove Something Urgently
Search Console’s Removals tool hides a URL from results within about a day. It is temporary — roughly six months — so use it to buy time while implementing a real fix, not as the fix itself.
Checking Your robots.txt
Overly broad rules are common and quietly expensive. A few things to verify:
Test individual URLs. Search Console’s robots.txt report shows the current fetched file and lets you test specific paths against it.
Watch for accidental breadth. Disallow: /blog blocks /blog, /blog/post-name, and /blogging-tips — anything starting with that string. If you meant the directory, use Disallow: /blog/.
Never block CSS or JS. Google needs these to render pages. Rules like Disallow: /wp-includes/ or Disallow: /assets/ can prevent rendering and materially hurt how your pages are assessed.
Check for a leftover global block. Disallow: / on a live site blocks everything. It gets left behind after launches more often than anyone would like, and it is worth checking first whenever traffic has collapsed inexplicably.
Remember it is public. Anyone can read yoursite.com/robots.txt. Do not use it as a list of things you would prefer people did not find.
Our guide to robots.txt covers syntax, the sitemap directive, and the rules worth having in more depth.
A Quick Decision Table
| Goal | Correct tool |
|---|---|
| Keep out of search results | noindex — and allow crawling |
| Keep genuinely private | Authentication (401/403) |
| Save crawl budget | robots.txt disallow |
| Remove urgently | Removals tool, then a real fix |
| Consolidate duplicates | Canonical tag — and allow crawling |
| Page permanently gone | 410 status |
The pattern across most of these: anything that requires Google to read an instruction requires Google to be allowed to fetch the page. Canonical tags have the same constraint, which is why blocking and canonicalising also do not combine — we cover that in “alternate page with proper canonical tag”.
Not a Contradiction, Just a Misread
“Indexed, though blocked by robots.txt” is Google reporting a consequence of your own configuration, not defying it. Blocking prevents reading, not listing. Once that distinction is clear, the whole status stops being mysterious.
Most entries under it are harmless system URLs that need no action. The ones worth your attention are pages you wanted indexed and accidentally blocked, and pages you wanted hidden and accidentally advertised. Both are quick to fix once you know which tool does which job.
If your indexing reports are full of statuses you cannot interpret, or you have inherited a robots.txt file nobody understands, our SEO services include a full crawl and indexation audit. Reach out to the team at blogthememachine.com, and subscribe to our newsletter below for more technical guides. Our Google Search Console tutorial is a good next read if these reports are new to you.
