robots.txt
What is robots.txt?
robots.txt is a plain-text file at your domain root that gives crawling instructions to bots via Allow/Disallow rules per user-agent. It controls crawling, not indexing — a disallowed page can still be indexed if linked elsewhere, so use noindex to keep a page out of results.
In the AI era, robots.txt also governs access for AI crawlers (e.g. GPTBot, Google-Extended, PerplexityBot). Blocking them keeps your content out of AI training and, in some cases, out of AI answers — a deliberate choice with real trade-offs for visibility.
Because a broad Disallow can accidentally hide your whole site, robots.txt changes should be reviewed carefully; it's also where you declare your sitemap location.
Why it matters
A single wrong line can deindex your site or block AI crawlers you actually want citing you.
robots.txt — FAQ
Does robots.txt stop a page from being indexed?
No — it controls crawling, not indexing. A blocked page can still appear in results if other pages link to it. Use a noindex meta tag or header to keep a page out of the index.
Should I block AI crawlers in robots.txt?
It's a trade-off. Blocking bots like GPTBot or Google-Extended keeps your content out of AI training/answers; allowing them can earn AI citations. Decide based on whether AI visibility helps your goals.
Related terms
Crawl Budget
Crawl budget is the number of pages a search engine will crawl on your site in a given period, set by crawl rate and crawl demand.
Canonical URL
A canonical URL is the version of a page you tell search engines to treat as the master when duplicate or similar URLs exist.
llms.txt
llms.txt is a proposed plain-text file at your site root that gives AI models a curated, structured map of your most important content.
Answer Engine Optimization (AEO)
Answer Engine Optimization is the practice of structuring content so AI answer engines like ChatGPT, Perplexity, and Google's AI Overviews cite it directly.
Put SEO on autopilot
AutoSEO writes, optimizes, and publishes answer-first content that ranks in Google and gets cited by AI engines — automatically.
$1 trial for 3 days · 30-day money-back guarantee · See pricing or how we compare.