How to Block Ahrefs, Moz and Majestic Crawlers (and AI Bots)

To block Ahrefs, Moz and Majestic, add one robots.txt group per crawler at the root of your domain and disallow everything. The three user-agent tokens you need are AhrefsBot, rogerbot and dotbot for Moz, and MJ12bot for Majestic. All three vendors document that their crawlers obey robots.txt, so the directives below are honoured within a crawl cycle or two.

Before you paste them, be clear about what blocking actually achieves — because the most common reason people give for doing it is the one thing it does not accomplish.

The copy-ready robots.txt block

Put this in a plain text file at the root of your site, reachable at yourdomain.com/robots.txt. Each crawler gets its own group, separated by a blank line.

• User-agent: AhrefsBot
• Disallow: /

• User-agent: AhrefsSiteAudit
• Disallow: /

• User-agent: rogerbot
• Disallow: /

• User-agent: dotbot
• Disallow: /

• User-agent: MJ12bot
• Disallow: /

• User-agent: SemrushBot
• Disallow: /

Moz needs two entries because it runs two crawlers: rogerbot is the Moz Pro campaign crawler that audits sites inside the product, and dotbot is the crawler that builds the Link Explorer web index. Blocking one and not the other is the usual mistake.

Ahrefs also runs two: AhrefsBot builds its public index, and AhrefsSiteAudit is the crawler its Site Audit tool uses on your own site. Block AhrefsSiteAudit only if you never intend to run Ahrefs' free Webmaster Tools audits on your own domain — otherwise you will have disabled a tool you might want.

Semrush runs several named bots including a separate site-audit agent, so if a full block is the goal, check their current bot documentation rather than assuming one token covers everything.

Other crawlers worth adding

If your goal is reducing bot load rather than hiding from one competitor, these are the SEO and data crawlers that most commonly show up in access logs.

• User-agent: BLEXBot
• Disallow: /

• User-agent: DataForSeoBot
• Disallow: /

• User-agent: Barkrowler
• Disallow: /

• User-agent: SeekportBot
• Disallow: /

Add them one at a time and watch your logs. Blocking a crawler you have never actually been visited by adds maintenance with no benefit.

What blocking these crawlers does and does not do

This is the part that changes the decision for most people.

• It does hide your site's pages and outbound links from those tools. Competitors will not be able to see your internal link structure, your content footprint, or where you link out.
• It does not remove your backlinks from their indexes. The links pointing at you live on other people's websites. Those crawlers still visit those sites, still see the link, and still attribute it to your domain. If your goal is stopping a competitor from seeing who links to you, blocking your own server does not achieve it.
• It does reduce server load and bandwidth. This is the honest, boring, best reason to block. Aggressive crawlers on a large site can consume a meaningful share of origin requests.
• It does not affect Google or Bing. Googlebot and Bingbot are separate user-agents and are unaffected by any of the rules above. Your organic rankings do not change because you blocked AhrefsBot.
• It does not hide anything. Your robots.txt is a public file. Anyone can read it and see exactly which crawlers you decided to block, which is itself a signal some people would rather not send.
• It does break your own tooling. If you block a vendor's crawler and later subscribe to that vendor, their site audit will not work on your domain until you unblock it.

Blocking AI crawlers: GPTBot, ClaudeBot, PerplexityBot

The same mechanism controls AI training and answer-engine crawlers, and this is where most of the current interest actually sits. These are the tokens the major providers document.

• User-agent: GPTBot
• Disallow: /

• User-agent: OAI-SearchBot
• Disallow: /

• User-agent: ChatGPT-User
• Disallow: /

• User-agent: ClaudeBot
• Disallow: /

• User-agent: Claude-User
• Disallow: /

• User-agent: PerplexityBot
• Disallow: /

• User-agent: CCBot
• Disallow: /

• User-agent: Google-Extended
• Disallow: /

• User-agent: Applebot-Extended
• Disallow: /

• User-agent: Bytespider
• Disallow: /

• User-agent: Amazonbot
• Disallow: /

• User-agent: meta-externalagent
• Disallow: /

Three distinctions matter more than the list itself.

Training crawlers versus answer crawlers are not the same thing. GPTBot and CCBot gather content for model training. OAI-SearchBot and PerplexityBot build the indexes those assistants search when answering a live question. Claude-User and ChatGPT-User fetch a page because a user asked about it right now. Blocking the training bots protects your content from being learned. Blocking the answer bots removes you from the results those products cite — a very different outcome.

Google-Extended is not a Search opt-out. Google documents it as controlling whether your content helps improve its generative products, and states that it does not affect inclusion or ranking in Google Search. Because AI Overviews are part of Search, opting out of Google-Extended is not a way to stay out of them. Applebot-Extended works similarly for Apple: it governs training use, not Apple's search crawler. Check both vendors' current documentation before relying on either.

Blocking answer engines has a real cost. AI assistants are becoming a discovery channel in their own right. If a model cannot fetch your page, it cannot cite you, and it will cite a competitor who left the door open. Blocking training crawlers while allowing the retrieval and search bots is the position most publishers land on, and it is a defensible middle ground.

If you would rather be visible to AI assistants than hidden from them, that is a different project. An llms.txt file gives assistants a curated map of your most useful pages — you can generate one with our free llms.txt generator, and check how machine-readable your site currently is with the agent-ready score checker. Note that llms.txt is an emerging convention rather than a standard any provider has committed to honouring, so treat it as low-cost insurance, not a ranking lever.

Rules that trip people up

Six technical details that cause most failed blocks.

1. robots.txt only works at the root. It must live at yourdomain.com/robots.txt. A file in a subdirectory is ignored, and subdomains need their own file.
2. Each crawler obeys only the most specific group that matches it. A bot with its own named group ignores the User-agent: * group entirely — including any Disallow rules you put there. Do not assume the wildcard group is a safety net.
3. User-agent tokens are matched case-insensitively; paths are case-sensitive. Disallow: /Admin does not block /admin.
4. Disallow is not noindex. A blocked URL can still appear in search results as a bare link if other pages link to it, because the crawler is forbidden from fetching the page and therefore cannot see a noindex tag. If you need a page out of an index, allow the crawl and use a noindex meta tag or an HTTP header.
5. Crawl-delay is honoured by some crawlers and not others. AhrefsBot, MJ12bot and Bingbot support it; Googlebot does not. If load is the problem, a crawl-delay line is often better than a full block, because you keep the tool's usefulness while capping the cost.
6. robots.txt is a request, not a wall. Reputable vendors comply. Scrapers, spoofed user-agents and content-theft bots do not. Enforcement requires blocking at the server, CDN or WAF layer — by verified user-agent, IP range, or a bot-management ruleset — and every major vendor publishes the IP ranges or reverse-DNS pattern needed to verify their crawler is genuine.

After editing, validate the file before you trust it. Our free robots.txt checker parses your live file and shows what each rule actually does, and the robots.txt generator builds a syntactically valid file if you would rather not hand-write one.

Should you block them at all?

For most sites, no. The load these crawlers generate is negligible below a few hundred thousand pages, the competitive secrecy benefit is largely illusory because your backlinks are visible from the other side regardless, and the public robots.txt advertises the decision.

Blocking makes sense when crawler traffic is measurably hurting origin performance, when you run a large site where bot requests carry real infrastructure cost, or when you have a specific policy reason — a private knowledge base, licensed content, or paywalled material that should not be fed into a training corpus.

Fixation on hiding from competitor tools has the same shape as fixation on third-party authority scores: energy spent on a proxy rather than on the thing itself. We wrote about that pattern in Ahrefs DR vs Moz DA, and if you are weighing which of these tools to actually pay for, Moz vs Semrush vs Ahrefs compares them directly.

Where AutoSEO fits

AutoSEO sits on the opposite side of this question from crawler blocking: our job is making your pages more findable, by both search engines and AI assistants. That means researching what to publish, generating articles with the on-page and structured-data work already done, publishing to thirteen CMS platforms, and tracking both classic rankings and whether AI assistants cite you. Current plans are on the pricing page.

Frequently Asked Questions

How do I block Ahrefs, Moz and Majestic in robots.txt?

Add a separate group for each crawler with Disallow: / underneath. The tokens are AhrefsBot for Ahrefs, rogerbot and dotbot for Moz, and MJ12bot for Majestic. Save the file at the root of your domain as robots.txt. All three vendors state their crawlers obey robots.txt, so the block takes effect on their next crawl cycle.

Does blocking AhrefsBot hide my backlinks from competitors?

No. Your backlinks sit on other people's websites, and Ahrefs discovers them by crawling those sites, not yours. Blocking AhrefsBot stops it seeing your pages and your outbound links; it does not remove the inbound links pointing at your domain from anyone's index.

Will blocking SEO crawlers hurt my Google rankings?

No. Googlebot and Bingbot are separate crawlers with separate user-agent tokens and are unaffected by rules naming AhrefsBot, rogerbot, dotbot or MJ12bot. The one genuine risk is a syntax error that accidentally disallows everything for every crawler, so validate the file after editing.

How do I block AI crawlers like GPTBot and ClaudeBot?

Use the same robots.txt syntax with the AI vendors' documented tokens — GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI, ClaudeBot and Claude-User for Anthropic, PerplexityBot for Perplexity, CCBot for Common Crawl, plus Google-Extended and Applebot-Extended for the two big platform opt-outs. Decide separately about training crawlers and answer crawlers: blocking the latter removes you from AI answers that would otherwise cite you.

Is robots.txt enough to actually stop a crawler?

Only for crawlers that choose to comply. Robots.txt is a published request, and reputable vendors honour it, but scrapers and spoofed user-agents simply ignore it. Real enforcement needs server, CDN or WAF-level blocking, ideally with crawler verification by IP range or reverse DNS so you do not block a legitimate search engine by mistake.

Should I use Disallow or noindex to keep pages out of search?

Use noindex when you want a page kept out of an index, and allow the crawl so the directive can be read. A Disallow rule prevents fetching, which means a crawler never sees your noindex tag and can still list the bare URL if other pages link to it. The two directives solve different problems and are frequently swapped by mistake.