You've launched a new site, opened an FTP client, and found an empty root directory staring back at you. The question is simple but consequential: should you generate a robots.txt file, and could one careless rule keep search engines away from your pages?
A well-built robots.txt file gives cooperating crawlers a clear access policy. It can reduce unnecessary fetching, identify XML sitemaps, and express preferences for classic search bots and newer AI crawlers. It can't protect private information, enforce access against hostile scrapers, or guarantee that a blocked URL disappears from search results.
Table of Contents
What Robots TXT Actually Does
robots.txt is a public plain-text file that sits at the root of a host and tells compliant automated crawlers which URL paths they may request. Google describes it as a mechanism for controlling crawler access, with the important requirement that the file be placed at the root of the host it governs. The formal protocol is documented in RFC 9309, which standardized the long-used Robots Exclusion Protocol in September 2022.
The file began as a practical safeguard against unnecessary server load. Martijn Koster proposed the protocol in 1994, and early search engines widely adopted it by June of that year. Modern generators are therefore encoding rules from a protocol that operated informally for 28 years before formal standardization, a history that explains both its usefulness and its limits.
A robots.txt file controls crawling, not guaranteed indexing. If a crawler can't fetch a page, it may still learn the URL from links or other signals. Use authentication for confidential material, and use a noindex directive where you need search engines to process a page and then exclude it from search results. Robots.txt itself isn't a security mechanism, a privacy layer, or a removal request.
Practical rule: Treat robots.txt as a sign at the entrance, not a locked door.
The file now matters beyond traditional search. Googlebot and Bingbot interpret it, while AI-related crawlers may use it to understand a publisher's access preferences. Independent research has found that AI search crawlers rarely check robots.txt and that stricter directives can reduce compliance, so the file should be one part of a broader crawler policy. For a useful overview of how crawling fits into technical SEO, see crawling in SEO.
Creating and Placing the File
You don't need a specialist application to generate robots.txt. A plain-text editor such as Notepad, TextEdit in plain-text mode, or VS Code is enough.
Start with a clean file
Create a new document and save it with this exact filename:
- Filename:
robots.txt
- Casing: lowercase
- Encoding: UTF-8 plain text
- Format: no rich-text styling, hidden formatting, or extra extension
Don't save it as robots.txt.txt, Robots.txt, or a word-processing document. A byte order mark or unexpected control character can create confusing parser behavior, especially when a hosting platform rewrites the file during deployment.
For an open site with no blocked paths, this minimal file is valid:
User-agent: *
Disallow:
The asterisk applies the group to crawlers that don't have a more specific group. An empty Disallow value blocks nothing, which is different from Disallow: /, where the slash blocks every path.
Put it at the correct root
Upload the file to the root of the relevant host so it resolves at:
`
Google states that a site can have only one robots.txt file for a host and that it must live at the host root. Rules are also specific to the protocol, host, and port, so a file on doesn't automatically control or another subdomain. This is one of the easiest mistakes to make during a multi-domain launch.
Upload through your FTP client, your hosting control panel's file manager, or the deployment process that publishes your site. If a CMS generates robots.txt dynamically, edit the platform's SEO settings instead of uploading a competing static file that the application may overwrite.
Check the live response
Open the final URL in a browser before testing individual rules. Confirm that you see plain text, the expected contents, and the production hostname. A staging copy, cached response, or redirect to an unexpected host can make a correct file appear broken.
Keep the file concise. Google recommends keeping robots.txt within the parser's supported size, and very large rule sets are difficult to audit even when they load successfully. For most small and medium sites, a short policy with clear groups is easier to maintain than a catalogue of speculative exclusions.
Core Directives Explained
Robots.txt syntax looks simple because it is simple. The hard part is deciding whether a rule expresses a real crawling need or merely adds noise.
User-agent selects the crawler group. User-agent: * targets all crawlers without a more specific group, while User-agent: Googlebot targets Google's main search crawler. A specific group should reflect an intentional policy. Listing many names without checking server logs often creates false confidence because a typo in a token fails to match the intended bot.
Disallow excludes paths from crawling, not necessarily from discovery or indexing. Disallow: /private/ targets that path prefix, while Disallow: / covers the whole host. An empty value disallows nothing. Allow can create an exception inside a broader blocked path, but crawler support and pattern interpretation deserve testing. Where rules overlap, the most specific matching rule generally determines the outcome, so broad blocks and narrow exceptions need careful review.
Sitemap identifies an XML sitemap using a fully qualified absolute URL. Google allows multiple Sitemap directives, which helps sites with several sitemap indexes or distinct content collections. The directive doesn't grant access to blocked URLs, but it gives crawlers a reliable discovery location.
Crawl-delay asks a crawler to pause between requests. Some crawlers, including Bing and Yandex, may honor it, while Google doesn't use it. It can also be misconfigured, slowing useful crawling without solving the actual server-load problem. For serious traffic control, server-side rate limiting is more dependable.
Host appeared in older Yandex-oriented configurations, but it isn't part of the standardized protocol and should generally be treated as deprecated noise.
Robots.txt directives at a glance
| Directive |
Purpose |
Honored By |
Common Pitfall |
User-agent |
Selects the crawler group |
Crawlers that implement robots.txt |
Misspelled or outdated bot token |
Disallow |
Excludes matching paths from crawling |
Compliant crawlers |
Using / unintentionally and blocking the whole host |
Allow |
Creates an exception or permits a path |
Crawlers supporting the relevant matching behavior |
Assuming every crawler resolves conflicts identically |
Sitemap |
Declares an XML sitemap URL |
Crawlers that read sitemap directives |
Using a relative URL or pointing to a dead sitemap |
Crawl-delay |
Requests slower request pacing |
Some crawlers, including Bing and Yandex |
Expecting Google or every bot to honor it |
Host |
Legacy preferred-host signal |
Limited legacy implementations |
Treating a deprecated directive as a standard control |
The practical test is intent. If you can't explain what a directive protects, improves, or makes discoverable, remove it.
Controlling AI and Search Crawlers
Many site owners assume that a Disallow rule is a universal command. It isn't. Robots.txt works as a signaling layer for crawlers that choose to comply, not as an enforcement wall for every program that requests a page.
Classic search crawlers such as Googlebot and Bingbot generally use robots.txt to decide whether they may fetch a path. AI-related user agents can include GPTBot, ClaudeBot, PerplexityBot, CCBot, Applebot-Extended, and Google-Extended. Their purposes aren't identical. A training crawler, a retrieval crawler, a link-preview agent, and a search crawler can represent different business and visibility choices.
Review server logs before writing a blocklist. Look for the actual user-agent strings reaching your site, compare them with the provider's documentation, and decide whether blocking a crawler could remove your content from a training dataset, an answer engine, or a search feature. An LLM audit for content strategists can help teams examine how their content appears across AI systems before they make access decisions.
A deliberately narrow block might look like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
This expresses a preference to block those named agents while leaving the general policy open. It doesn't authenticate the caller, validate the claimed identity, or stop an uncooperative scraper from ignoring the file.
AI and search crawlers, robots.txt compliance
| Crawler |
User-agent token |
Respects robots.txt |
Notes |
| Google Search |
Googlebot |
Generally yes |
Keep separate from AI-related Google controls |
| Bing Search |
Bingbot |
Generally yes |
May support crawl pacing directives |
| OpenAI crawler |
GPTBot |
Behavior varies |
Verify the token against current logs and provider guidance |
| Anthropic crawler |
ClaudeBot |
Behavior varies |
A block is a preference, not technical enforcement |
| Perplexity crawler |
PerplexityBot |
Behavior varies |
Consider its role in answer-engine visibility |
| Common Crawl |
CCBot |
Behavior varies |
Treat it separately from commercial search crawling |
| Google AI-related agent |
Google-Extended |
Behavior varies |
Don't confuse it with Googlebot search crawling |
The empirical evidence is uncomfortable but useful. A large study found that bots become less likely to comply with stricter directives, while AI search crawlers often don't check robots.txt at all. Use the file to communicate policy to well-behaved systems, then add rate limiting, bot verification, access controls, and applicable text-and-data-mining reservations where your legal and technical setup supports them.
Testing and Validating the Live File
A robots.txt file can be syntactically neat and operationally wrong. Test the production response, the exact paths that matter, and the crawler identities your policy targets.
Start by opening /robots.txt on the live hostname in an incognito browser window. Check that the server returns the version you edited, not a staging file or an old cached response. Confirm the sitemap URL resolves separately, and inspect redirects because a policy on one host doesn't automatically govern another.
Use several validation views
- Check the intended paths. Test a page that should remain crawlable, a folder that should be blocked, and any exception created with
Allow. Pay special attention to a leading slash and to rules that could match more URLs than intended.
- Use Google Search Console. Google's documentation and testing guidance provide a way to inspect whether Googlebot can access a URL under the live policy. The Google Webmaster Console guide is useful when you need to connect robots checks with broader indexing diagnostics.
- Cross-check with another parser. A technicalSEO.com checker or Screaming Frog's robots.txt tool can expose malformed patterns, unsupported directives, or assumptions that one tool doesn't flag. Treat third-party results as a second opinion, not as proof that every crawler will behave identically.
- Review access logs. After deployment, inspect requests from the user agents you care about. A blocked path should stop receiving compliant crawler requests, while an allowed path should remain fetchable. Logs also reveal bots that claim a familiar name but don't behave like the genuine service.
Retest after a CMS update, plugin change, hosting migration, CDN change, or redesign. Many platforms generate robots.txt dynamically, so a manual file can be replaced without an obvious deployment error. A traffic or indexing problem that appears immediately after a release deserves a live-file check before deeper diagnosis.
A robots.txt test isn't complete until the file you tested matches the file crawlers actually receive.
Maintaining Robots TXT Over Time
Treat robots.txt as production infrastructure, not a one-time SEO chore. The file sits at a sensitive junction between crawl access, sitemap discovery, content policy, and deployment logic. It needs an owner who knows why each rule exists.
Use a launch checklist
Before a new site or redesign goes live, verify the following:
- Reachability: The production host returns the intended plain-text file at
/robots.txt.
- Sitemap discovery: Every
Sitemap directive uses a fully qualified URL and points to a live XML sitemap.
- Critical access: Important templates, product pages, articles, JavaScript, and CSS aren't caught by a broad path rule.
- Development isolation: Staging, preview, internal search, confirmation, and administrative paths aren't exposed through an accidental open policy.
- Rule behavior: A representative allowed URL and every important blocked path have been tested.
- Host coverage: The canonical protocol and hostname serve the policy that search engines are expected to use.
Google's guidance confirms that a host has one robots.txt file and that multiple sitemap directives are supported. That makes hostname discipline more important than adding elaborate syntax. A beautifully formatted file on the wrong host still controls nothing useful.
Review it when the business changes
Three events should trigger an immediate review:
- URL structures change: A redesign, migration, or CMS switch can make old path rules irrelevant and cause new rules to match valuable content.
- New site areas launch: Internal search, faceted navigation, thank-you pages, staging routes, and customer-only areas may need separate treatment.
- Crawler policy changes: AI providers introduce new agents and alter how their systems use fetched content. Recheck logs and provider documentation before adding or removing tokens.
A routine audit still matters when nothing obvious changed. CDN settings, plugins, server templates, and deployment scripts can rewrite the response. Add the check to your broader technical SEO checklist, and make validation part of release review rather than an emergency response.
Store the file in version control when your platform permits it. Require code review for changes, record the business reason for every override, and keep a rollback path for an accidental Disallow: /. AutoSEO can fit into this operational workflow as a platform that audits technical SEO issues, manages robots.txt and related checker tools, publishes changes across supported CMS integrations, and monitors search and AI visibility.
Keep an audit trail that answers four questions: who changed the policy, what changed, why it changed, and how the live result was tested. That discipline prevents the most expensive robots.txt mistakes, especially rules added during a rushed launch and forgotten after the original problem disappears.
If you're launching or revising a site, use AutoSEO to check robots.txt access alongside sitemap, indexing, and broader technical SEO issues. Review the generated policy, validate it on the live host, and connect the result to an ongoing monitoring workflow before your next deployment.
