Robots.txt is a plain text file that sits in your site’s root directory and tells search engine crawlers which parts of your site they should and shouldn’t request. It’s the first thing a crawler checks before it starts working through your pages. The syntax is simple – user-agent declarations, allow and disallow rules, and an optional sitemap reference. It doesn’t remove pages from search results, it doesn’t protect private content, and it doesn’t guarantee anything. But set up properly, it gives you meaningful control over how crawlers spend their time on your site.
Most sites have a robots.txt file already. Content management systems generate one by default, and in plenty of cases it works fine without anyone touching it. The problems start when someone edits it without fully understanding the syntax, when rules conflict with each other, or when the file quietly blocks pages that should be indexed. At Gorilla Marketing, we see robots.txt misconfiguration in a good proportion of technical SEO audits – sometimes minor, sometimes silently damaging. This guide covers what the file actually does, how to write it correctly, and the mistakes that trip people up most often.
How Does Robots.txt Work?
When a search engine crawler arrives at your domain, its first request is to `https://yourdomain.com/robots.txt`. If it finds a file there, it reads the rules before crawling anything else. If there’s no file, or the server returns a 404, the crawler assumes everything is fair game and starts crawling freely.
The file is a UTF-8 encoded text file with a straightforward syntax. Here’s a minimal example:
“`
User-agent: *
Disallow: /admin/
Disallow: /internal-search/
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
“`
That tells every crawler (the `*` wildcard means “all user-agents”) to stay out of `/admin/` and `/internal-search/`, but everything else is allowed. The `Sitemap` directive at the bottom points crawlers to your XML sitemap so they can discover your pages more efficiently.
A few things to note about how this works in practice. Crawlers read the file top-to-bottom, but they match the most specific rule, not the first one they encounter. Google’s crawler specifically looks for the longest matching path when multiple rules could apply. Also, robots.txt is fetched and cached – Google caches the file for up to 24 hours, so changes you make won’t take effect immediately.
Where Does Robots.txt Live?
The file must sit in the root directory of your domain. That means `https://yourdomain.com/robots.txt` – not in a subdirectory, not renamed, not behind a redirect. If you’re running subdomains, each one needs its own robots.txt file. The rules at `https://yourdomain.com/robots.txt` don’t apply to `https://blog.yourdomain.com/` – that subdomain needs its own file at `https://blog.yourdomain.com/robots.txt`.
This catches people out during migrations or when consolidating properties. Each origin (protocol + domain) operates independently.
What’s the Syntax?
Robots.txt syntax looks simple. It mostly is. But the details matter, because one misplaced rule can block your entire site from being crawled.
User-agent
Every block of rules starts with a `User-agent` line identifying which crawler the rules apply to.
“`
User-agent: Googlebot
Disallow: /private/
“`
That only applies to Googlebot. To target every crawler, use the wildcard:
“`
User-agent: *
Disallow: /private/
“`
You can have multiple blocks in one file, each targeting different crawlers. If a crawler finds a block that matches its specific user-agent name and a block targeting `*`, it follows the specific block and ignores the wildcard one entirely. It doesn’t combine them.
Disallow
`Disallow` tells a crawler not to request URLs matching a given path.
“`
Disallow: /staging/
“`
This blocks any URL that starts with `/staging/` – so `/staging/page-1/`, `/staging/test/`, and everything else under that directory. An empty `Disallow` line (or `Disallow:` with no path) means “disallow nothing” – i.e., everything is allowed.
Allow
`Allow` is used to override a broader `Disallow` rule for specific paths.
“`
User-agent: *
Disallow: /resources/
Allow: /resources/guides/
“`
This blocks everything under `/resources/` except pages under `/resources/guides/`. The `Allow` directive creates an exception within a broader restriction.
Wildcards and Pattern Matching
Google and Bing support two wildcard characters that aren’t in the original specification but are widely implemented:
`*` matches any sequence of characters
`$` matches the end of a URL
Practical examples:
“`
Disallow: /*.pdf$
“`
Blocks any URL ending in `.pdf`. The `*` matches everything before it, and the `$` anchors the match to the end of the URL.
“`
Disallow: /*?
“`
Blocks any URL containing a query string. Useful for preventing crawlers from wasting time on parameterised URLs from faceted navigation or internal search.
“`
Disallow: /products/*.json
“`
Blocks JSON files under `/products/` while leaving the HTML product pages accessible.
These wildcards are powerful but easy to get wrong. A misplaced `*` can block far more than you intended. Test every wildcard rule before deploying it.
Comments
You can add comments to your robots.txt file using the `#` character:
“`
# Block staging area
User-agent: *
Disallow: /staging/
# Let Google see the public guides section
Allow: /resources/guides/
“`
Comments don’t affect how crawlers read the file, but they make maintenance much easier – especially when multiple people edit the file over time.
Sitemap Directive
The `Sitemap` line tells crawlers where to find your XML sitemap. It’s not technically part of the crawl rules, but it’s a standard inclusion:
“`
Sitemap: https://yourdomain.com/sitemap.xml
“`
You can include multiple sitemap references. This directive works regardless of which user-agent block it sits in – crawlers treat it as standalone. Worth keeping even if you’ve submitted your sitemap through Search Console, because Bing, Yandex, and other crawlers pick it up from here. For more on setup and best practices, we’ve covered XML sitemaps in a separate guide.
Crawling vs Indexing: Why This Distinction Matters
This is the single biggest source of confusion with robots.txt, and it causes real problems.
Robots.txt controls crawling – whether a search engine bot requests a URL. It does not control indexing – whether a page appears in search results.
If you block a page in robots.txt, Google won’t crawl it. But if other sites link to that page, Google can still index the URL based on external signals. You’ll see the page in search results with “No information is available for this page” because Google knows it exists but can’t access it to generate a snippet.
This is why robots.txt is not a tool for removing pages from search. If you want a page excluded from search results entirely, you need a `noindex` meta tag or an `X-Robots-Tag` HTTP header. And here’s the catch-22: for Google to see a `noindex` directive, it needs to crawl the page. If you’ve blocked the page in robots.txt, Google can’t crawl it, so it never sees the `noindex` tag.
The practical rule: never use robots.txt to block pages that have a `noindex` tag. The two directives work against each other.
What’s the Difference Between Robots.txt, Meta Robots, and X-Robots-Tag?
Three mechanisms, three different jobs. Understanding when to use each one saves you from conflicting directives that create hard-to-diagnose problems.
| Mechanism | Level | Controls | How it works |
|---|---|---|---|
| Robots.txt | Site/directory | Crawling | Prevents crawlers from requesting specific URLs. File-level control. Doesn’t affect indexing directly. |
| Meta robots tag | Page | Indexing and following | HTML tag in the page’s ``. Directs search engines on whether to index the page and whether to follow its links. Requires crawling to work. |
| X-Robots-Tag | Page/resource | Indexing and following | HTTP header. Works like meta robots but can be applied to non-HTML resources (PDFs, images, JSON files). Requires crawling to work. |
Robots.txt is best for broad crawl management – keeping crawlers out of directories that serve no search purpose.
Meta robots (``) is what you use when a page should be crawled but not indexed. Thin pages, paginated archives, filtered views – `noindex` handles these.
X-Robots-Tag (`X-Robots-Tag: noindex`) does the same job as meta robots but through an HTTP header. Particularly useful for non-HTML files like PDFs, where you can’t place a meta tag.
The key point: these mechanisms can contradict each other. A robots.txt `Disallow` that prevents Google from seeing a page’s `noindex` tag is the most common conflict.
What About Crawl Budget?
Crawl budget is the number of pages a search engine will crawl on your site within a given period. For small-to-medium sites, it’s rarely a concern. But for larger sites with tens of thousands of pages, robots.txt is one of the main tools for managing it. Block paths that have no search value, and you free up budget for pages that matter.
Common candidates for blocking:
Internal search results (`/search/`, `/?s=`) – these generate near-infinite URL combinations and almost never deserve indexing
Faceted navigation parameters – size, colour, price range filters that create thousands of URL variants
Admin and system paths (`/wp-admin/`, `/cart/`, `/checkout/`) – no search value
Development or staging paths – should be blocked from crawling and ideally from public access entirely
API endpoints – if they’re accessible via the same domain
Print-friendly or alternate format pages – duplicate content with no search purpose
The goal isn’t to block as much as possible. It’s to block what’s genuinely wasting crawler attention so the pages that drive traffic get crawled more frequently.
Should You Block CSS and JavaScript?
No. Years ago, it was common practice to block CSS and JavaScript files in robots.txt. Some old tutorials still recommend it. Don’t follow that advice.
Google renders pages. It executes JavaScript, it applies CSS, and it uses the rendered result to understand your content. If you block your CSS and JavaScript files, Google can’t render your pages properly. It sees unstyled, potentially broken HTML, and it can’t assess the page the way a user would see it. This can directly affect how Google evaluates and ranks your content.
Google has been explicit about this. Their documentation warns that blocking CSS and JS resources can result in suboptimal rankings. Google Search Console will flag blocked resources as issues if they affect rendering.
If your robots.txt contains lines like `Disallow: /wp-includes/` or `Disallow: .css$` or `Disallow: .js$`, remove them. The only exception would be JavaScript files that trigger infinite crawl traps, but that’s a niche edge case, not a general rule.
How Do You Block AI Crawlers?
Since 2023, AI companies have been deploying crawlers to scrape web content for training data. Robots.txt is the primary mechanism site owners have for controlling access. Here are the main AI bot user-agents to be aware of:
| User-agent | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data for GPT models |
| ChatGPT-User | OpenAI | Real-time browsing for ChatGPT |
| Google-Extended | Training data for Gemini/Bard (doesn’t affect Search) | |
| ClaudeBot | Anthropic | Training data for Claude models |
| anthropic-ai | Anthropic | Older Anthropic crawler identifier |
| Bytespider | ByteDance | Training data for TikTok’s AI models |
| CCBot | Common Crawl | Open dataset used by many AI companies |
| PerplexityBot | Perplexity | Content retrieval for AI search |
| Applebot-Extended | Apple | Training data for Apple Intelligence features |
| Meta-ExternalAgent | Meta | Training data for Llama models |
| cohere-ai | Cohere | Training data for Cohere models |
To block all of these:
“`
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: cohere-ai
Disallow: /
“`
A few things to understand here. Robots.txt relies on voluntary compliance – there’s no enforcement mechanism, and a crawler that ignores the file will scrape your content regardless. Most legitimate AI companies respect it because the reputational cost of ignoring it is high, but it’s not a guarantee.
Be careful about what you block. `Google-Extended` only controls training data collection – blocking it doesn’t affect regular Googlebot or your search rankings. But accidentally blocking `Googlebot` itself will remove your pages from Google Search entirely.
The list changes regularly. Check user-agent documentation from each company periodically, or follow technical SEO sources that track new additions.
Does Robots.txt Affect Link Equity?
Yes, and not in the way most people assume.
If you block a page in robots.txt, crawlers can’t access it. But external links pointing to a blocked URL still pass equity to it – Google just can’t see the page to follow its outbound links.
This creates a dead end. Equity flows in but can’t flow out. If that blocked page links to important sections of your site, you’ve cut off the flow of equity to those sections.
The common misconception is that blocking a page “saves” equity for other pages. It doesn’t. Robots.txt controls crawling, not how PageRank is distributed. If you want to prevent equity flowing to specific pages, that’s a job for `nofollow` attributes – not robots.txt.
What Is RFC 9309?
In September 2022, the Robots Exclusion Protocol was formally standardised as RFC 9309. Before this, robots.txt operated on an informal convention dating back to 1994. Everyone agreed on the basics, but there were gaps – how should crawlers handle malformed files? What’s the maximum file size? What happens on a 5xx error?
RFC 9309 answered these with a formal specification:
File size limit: 500 kibibytes (512,000 bytes, roughly 512KB). Content beyond this limit may be ignored by compliant crawlers. Google has stated it enforces a 500KB limit.
Server errors (5xx): When a robots.txt request returns a 5xx status code, compliant crawlers should assume full disallow – meaning they won’t crawl anything on the site until the robots.txt becomes accessible again. This is a big deal. If your server intermittently returns 5xx errors for the robots.txt URL, crawling can stop entirely during those periods.
404 responses: A 404 on robots.txt means no restrictions – crawlers treat the entire site as open. This is the safe default and how most implementations have always handled it.
Caching: The standard allows for caching, and Google caches robots.txt for up to 24 hours. If you make changes, they won’t propagate to all crawlers instantly.
The formalisation also clarified that `Allow` is a legitimate directive and specified how conflicting rules should be resolved: the most specific match wins, measured by path length.
For most site owners, RFC 9309 doesn’t change daily practice. But understanding file size limits and server error behaviour helps when diagnosing crawl issues that don’t have an obvious explanation.
Is Robots.txt a Security Mechanism?
No. Robots.txt is publicly accessible – anyone can type `yourdomain.com/robots.txt` into a browser and see exactly what you’re blocking. If you’re using it to hide an admin panel or staging environment, you’re effectively publishing a map of those locations.
Malicious actors specifically check robots.txt for interesting paths. A `Disallow: /admin-secret-portal/` entry is an invitation, not a barrier.
For actual security, use proper authentication, server-level access restrictions, or keep sensitive environments on separate, non-public domains. Block staging environments in robots.txt to prevent search engines from indexing them, but don’t rely on it as your security layer.
How Do You Test Your Robots.txt?
Before deploying changes, test them. Google Search Console provides a robots.txt tester (under Settings for your property) where you can enter URLs and check whether your rules allow or block them.
Beyond Search Console, validate manually:
Fetch the file directly – visit `yourdomain.com/robots.txt` in a browser and read it line by line
Check status code – the file should return a 200 status code. A 404 means crawlers see no restrictions. A 5xx means crawlers may assume everything is blocked (per RFC 9309)
Review after CMS updates – some CMS updates or plugin changes can modify your robots.txt without warning. Check after any major update
For sites with complex rules, build a test matrix of URLs that should be allowed and URLs that should be blocked. Run through it after every change.
What Are the Most Common Robots.txt Mistakes?
Certain mistakes appear over and over. Here are the ones that cause the most damage.
Blocking the entire site
“`
User-agent: *
Disallow: /
“`
This single rule blocks every crawler from every page. It’s the nuclear option, and it’s sometimes left in place after a staging site goes live. Always check robots.txt immediately after launching a site. If your traffic drops to zero after a migration, this is the first thing to check. For more on avoiding migration pitfalls, we’ll be covering site migration in detail in an upcoming piece.
Blocking CSS and JavaScript
Already covered above, but it appears on this list because it’s so common. Legacy rules from years ago that nobody has reviewed. Remove them.
Conflicting rules between robots.txt and noindex
Blocking a page in robots.txt while also relying on a `noindex` tag to keep it out of search. Google can’t see the `noindex` because it can’t crawl the page. The result is often the page appearing in search results with a “no information available” snippet – the opposite of what was intended.
Not having a robots.txt at all
While a missing file isn’t a disaster – crawlers simply crawl everything – it means you have no mechanism for guiding crawl behaviour. Even a minimal robots.txt with just a `Sitemap` directive is better than nothing.
Forgetting about subdomain separation
Rules at `example.com/robots.txt` don’t apply to `blog.example.com` or `shop.example.com`. Each subdomain needs its own file. This is especially common after adding a subdomain for a blog or shop – the main domain’s robots.txt does nothing for the new subdomain.
Using robots.txt to handle duplicate content
If you have duplicate content issues, robots.txt isn’t the right tool. Canonical tags, 301 redirects, or `noindex` directives are all better options depending on the situation. Blocking duplicate pages in robots.txt still allows Google to index them based on external signals – it just prevents Google from seeing the content and any corrective tags you’ve placed on the page. We’ve covered duplicate content strategies in a separate article.
Incorrect wildcard patterns
A rule like `Disallow: /*print` blocks every URL containing the word “print” – including `/blueprint-software/`, `/sprint-training/`, and anything else with those characters. Wildcard rules need careful testing to avoid catching URLs you didn’t intend to block.
Trailing whitespace and invisible characters
Robots.txt parsers can be sensitive to invisible characters – trailing spaces, BOM (byte order mark) characters, or wrong line endings. If your rules aren’t being respected and the syntax looks correct, open the file in a plain text editor that shows whitespace characters. Files edited in rich text editors or word processors sometimes pick up hidden formatting.
What Should a Good Robots.txt Look Like?
There’s no universal template, because every site has different requirements. But here’s a solid starting point for a typical business website:
“`
# Main crawl rules
User-agent: *
Disallow: /wp-admin/
Disallow: /internal-search/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Allow: /wp-admin/admin-ajax.php
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
# Sitemap location
Sitemap: https://yourdomain.com/sitemap.xml
“`
The `Allow` line for `admin-ajax.php` is there because WordPress uses that endpoint for front-end AJAX calls – blocking the entire `/wp-admin/` directory would break dynamic functionality if Google tries to render pages that depend on it. AI crawler blocks are separated into their own blocks for clarity, and the sitemap reference sits at the bottom where it’s easy to find.
Adapt this to your setup. E-commerce sites will need rules for faceted navigation parameters. Large publishers might need rules for paginated archives. Multi-language sites need to consider path structures for each language version.
Crawl-delay: Does It Still Matter?
The `Crawl-delay` directive tells crawlers to wait a specified number of seconds between requests.
“`
User-agent: *
Crawl-delay: 10
“`
Google ignores this entirely – it uses its own adaptive crawl rate system and provides controls through Search Console. Bing does respect `Crawl-delay`, as do some smaller crawlers. If server load from bots is a genuine problem, set a crawl-delay for the bots that honour it, but don’t treat it as a universal fix.
Keeping Your Robots.txt Maintained
Robots.txt isn’t a set-and-forget file. Sites change, CMS platforms get updated, new bots appear. A file that was correct two years ago might be blocking content that matters now.
Build a review into your regular maintenance:
After any migration or platform change – the new platform may generate a different default robots.txt, or the old one may reference paths that no longer exist
When adding new site sections – if a new subdirectory is added, consider whether it needs crawl rules
Quarterly AI bot review – new AI crawlers appear regularly; update your blocking rules to match
When crawl budget issues appear – if Google Search Console shows a drop in crawl rate or pages aren’t being indexed as quickly, check whether robots.txt rules are part of the problem
If you’re running an SEO content programme and publishing regularly, it’s worth confirming that new content paths aren’t inadvertently caught by existing disallow rules. A broad rule like `Disallow: /blog/draft/` is sensible, but `Disallow: /blog/d` would also block `/blog/digital-marketing/` and anything else starting with that path.
Robots.txt is one of those files that’s easy to overlook because it sits quietly in the background. But it’s one of the first things search engines look at, and getting it wrong has real consequences – from wasted crawl budget to entire sections of your site disappearing from search results. Get it right, keep it maintained, and it does its job without any drama.
