It is a plain text file, usually under twenty lines, sitting at the root of your domain — and it is capable of removing your entire site from Google in a single line. robots.txt is the highest-consequence-per-character file most sites have, which makes it worth understanding properly rather than copying from a tutorial and hoping. Here is what it does, what it does not do, and the specific mistakes that cause real damage.
What robots.txt Is For
robots.txt tells crawlers which parts of your site they may fetch. It lives at exactly one location — https://yoursite.com/robots.txt — and nowhere else. A file at /blog/robots.txt is ignored entirely.
Two limits define what it can achieve:
It controls crawling, not indexing. Blocking a URL does not keep it out of search results. Google can index a blocked URL based on links alone, which produces the “Indexed, though blocked by robots.txt” status — covered here in full.
It is advisory. Reputable crawlers honour it. Malicious scrapers ignore it completely. It is not a security measure, and the file is publicly readable by anyone.
What it is genuinely good for is crawl budget management — keeping crawlers away from URLs that waste their time so they spend it on pages that matter.
The Syntax
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap-index.xmlUser-agent names the crawler the following rules apply to. * means all crawlers. Specific names include Googlebot, Bingbot, and Googlebot-Image.
Disallow blocks a path. Disallow: / blocks everything. An empty Disallow: blocks nothing.
Allow creates an exception inside a disallowed path. Google supports this; not every crawler does.
Sitemap points to your XML sitemap. It is independent of user-agent groups and conventionally goes at the end.
Three behaviours worth knowing:
- Paths are prefix matches.
Disallow: /blogblocks/blog,/blog/post, and/blogging-tips. For a directory, write/blog/. - Rules are case-sensitive.
/Admin/and/admin/are different. - The most specific matching rule wins, not the first one listed.
Two wildcards are supported by Google:
Disallow: /*? # any URL containing a query string
Disallow: /*.pdf$ # any URL ending in .pdf* matches any sequence; $ anchors to the end of the URL.
Adding Your Sitemap
Sitemap: https://yoursite.com/sitemap-index.xmlUse the full absolute URL including protocol. Relative paths do not work. You can list several lines if you have multiple sitemaps.
This is worth doing even though you have submitted the sitemap in Search Console, for two reasons: other search engines find it without you registering anywhere, and it survives changes of ownership of your Search Console account.
It does not replace submitting in Search Console, because only Search Console gives you the processing report showing how many URLs were read and indexed.
What to Block
Genuinely worth blocking:
- Admin and login paths —
/wp-admin/,/administrator/ - Internal search results pages — these generate unlimited low-value URLs
- Cart, checkout, and account pages
- Filter and sort parameter URLs on large catalogues
- Duplicate print or export views
- Server-generated temporary and staging directories
Do not block:
- CSS and JavaScript. Google needs these to render your pages. Blocking
/wp-includes/,/assets/, or/static/can prevent proper rendering and materially change how your pages are assessed. This is one of the most damaging common mistakes. - Images you want in Google Images.
- Anything you want a
noindexhonoured on. Google must fetch the page to see the tag. - Anything you want a canonical tag honoured on, for the same reason — see “alternate page with proper canonical tag”.
- Genuinely sensitive paths. Listing them advertises them. Use authentication.
A Sensible WordPress Default
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Sitemap: https://yoursite.com/sitemap_index.xmlThat is close to all most WordPress sites need. Note there is no block on /wp-content/ or /wp-includes/ — older tutorials recommend those and they break rendering.
The Mistakes That Cause Real Damage
Disallow: / left on a live site. Blocks everything. Usually a staging configuration that shipped to production. If organic traffic has collapsed with no other explanation, check this first — it takes ten seconds and it is the single most common catastrophic cause.
Blocking render-critical assets. Covered above. Test with URL Inspection’s rendered view; if the screenshot looks broken, you are blocking something Google needs.
Assuming it hides content. It does not. It is a public file requesting cooperation.
Blocking a page that has a noindex tag. Guarantees the noindex is never read. The two do not combine.
Trailing-slash confusion. Disallow: /blog and Disallow: /blog/ behave differently. Be deliberate.
Blocking URLs that have backlinks. Those links point at a URL Google cannot fetch, so their value is largely wasted. If a URL has earned links, let it be crawled.
Editing without testing. Search Console’s robots.txt report shows the file Google last fetched and lets you test specific paths. Use it after every change.
Verifying It Works
- Visit
yoursite.com/robots.txtin a browser. Confirm it returns 200 and shows what you expect. A 404 is fine — it means no restrictions — but a 500 error can cause Google to pause crawling your site entirely. - Check the Search Console robots.txt report. It shows the fetched version, when it was last read, and any parse errors. Google caches the file for around 24 hours, so changes are not instant.
- Test specific URLs, especially ones near a rule boundary.
- Use URL Inspection on a real page and check the rendered screenshot for missing styles.
- Watch the Page Indexing report for a week after any change.
Our Google Search Console tutorial covers navigating these reports if they are unfamiliar.
robots.txt and AI Crawlers
A live question in 2026: whether to block AI training crawlers. These identify themselves separately from search crawlers, and you can block them independently:
User-agent: GPTBot
Disallow: /Whether to do this is a business decision, not an SEO one. Blocking AI crawlers does not affect your Google rankings — but it may reduce the chance of your content being cited in AI answers, which is an increasingly meaningful referral source. Our take on how that landscape is shifting is in is SEO dead.
Whatever you decide, keep it deliberate. Blocking a crawler because a template included the rule is not a decision.
Small File, Large Blast Radius
robots.txt does one narrow job: telling well-behaved crawlers where not to go. It is not a security tool, not an indexing control, and not a way to hide anything. Nearly every problem people have with it comes from expecting it to do one of those three things.
Keep it short. Block genuine crawl waste. Never block assets Google needs to render your pages. Include your sitemap. And check it exists and returns 200 after every deploy — because the failure mode is not subtle, and it is entirely preventable.
If you want a technical audit covering crawl budget, indexation, and the rest of your site’s foundations, our SEO services start there. Get in touch with the team at blogthememachine.com, and subscribe to our newsletter below for more technical SEO guides. For the wider picture, technical SEO basics covers how crawling fits alongside indexing, speed, and site structure.
