Free tool · No signup
Googlebot checker
Evaluate supplied robots.txt rules for a path and selected crawler tokens. This checks text policy, not actual crawler access, indexing or AI citations.
Quick answer
What does this crawler checker test?
This tool evaluates rules in fetched or pasted robots.txt text for a selected path and crawler token. It reports matching groups and the winning Allow or Disallow rule. It does not send requests as those crawlers, verify their identity, test indexing, or measure AI citations.
Interpret a rule result narrowly
An allowed result means the supplied text has no matching restriction for that crawler and path. Authentication, network failures, firewalls and other controls can still prevent retrieval.
A blocked result describes a matching robots restriction. Review why the rule exists before changing it; private account flows and intentional training restrictions may be appropriate.
Keep search and training roles separate
Crawler tokens can serve different purposes. The tool labels model-training crawlers separately from search indexing. Google-Extended is a Gemini training and grounding control, not a Google Search crawler.
Do not remove intentional restrictions solely to turn every result green. Choose policies according to the specific services and uses you want to support.
Use complete, relevant source text
Fetch mode requests the HTTPS origin’s robots.txt through the shared page fetcher. Failed or truncated responses are not evaluated. Paste mode evaluates the text you supply, which may differ from the current published file.
Matching groups with equal best specificity are merged. Rule matching normalizes encoded octets, supports wildcard and end-anchor patterns, and gives Allow precedence on equal literal specificity. These text checks do not certify every crawler’s implementation.
Where this data comes from
The tool parses user-agent groups, Allow and Disallow rules, sitemap declarations and selected other directives. It selects the most specific matching user-agent groups, merges equal-specificity groups, and evaluates the requested path using normalized UTF-8/unreserved octets. Longer literal rule matches win, with Allow winning ties. Fetch mode does not authenticate as a real crawler. No live indexing or citation check is performed.
Frequently asked questions
Does an allowed result prove Googlebot can fetch my page?
No. It describes the supplied robots rules. Network, authentication and other access controls are not tested.
Does an allowed result guarantee indexing?
No. This tool does not inspect indexing status or predict search inclusion.
Should every AI crawler be allowed?
Choose policies for the particular services and uses you intend to support. Search and training controls have different purposes, and intentional restrictions can be appropriate.
How are repeated user-agent groups handled?
Groups with equal best specificity are merged before evaluating their rules. More specific user-agent groups take precedence over wildcard groups.
What happens when the file is unavailable or truncated?
The tool shows an error rather than evaluating incomplete rules. You can retry or paste complete text.
Does this tool measure AI citations?
No. It checks selected robots rules for a path and crawler token. It does not observe or predict citations.
Related free tools
Review discovery settings alongside useful content
AutoSEO offers drafting and publishing workflows for supported connections. Review content and crawler policies before publishing changes. The plan is $39/mo per website, charged today; cancel anytime, with a 30-day money-back guarantee. Rankings and AI citations are not guaranteed.
Get started for $39/mo