Skip to content

← All tools

Free webmaster tool

robots.txt generator and tester

Pick a preset or write your own rules, copy the file, then test whether a given crawler may fetch a given URL. The tester follows Google's rules: the longest matching rule wins, Allow beats Disallow on a tie, and * and $ work as wildcards.

Start from a preset

Hides the admin area, login page and internal search, but keeps admin-ajax.php reachable because themes use it.

Rules

Group 1

* means every crawler. Examples: Googlebot, Bingbot, GPTBot.

Start with /. Use * for any characters and $ for the end of the address, e.g. /*.pdf$

Bing and some others obey it; Google ignores it.

Your robots.txt

Save it as robots.txt in the root of your site, so it opens at https://yoursite/robots.txt.

Test a URL

Check whether a crawler may fetch an address, using Google's rules: the longest matching rule wins, Allow beats Disallow on a tie, * matches any text and $ means the end.

File to test

Blocked

The most specific matching rule is a Disallow.

Matched rule: Disallow: /wp-admin/ (line 3)

Group used: User-agent *

How it works

  • •Choose a preset (allow all, block all, WordPress, Next.js or block AI-training crawlers) and edit the groups. Each group names crawlers (User-agent) and the paths they may or may not fetch.
  • •Add your sitemap address, then copy or download robots.txt and put it in your site root.
  • •Use the tester: enter an address and a crawler name and see whether it is allowed, which rule decided it and on which line. You can also paste any other robots.txt to test it.

What it does and what it cannot do

  • •robots.txt is a request, not a lock. Well-behaved crawlers obey it, but bad bots ignore it, and it never protects private pages. Use passwords for that.
  • •Blocking a page from crawling does not remove it from Google: a blocked URL can still be indexed if other sites link to it. To keep a page out of results, allow crawling and use noindex.
  • •The AI list names crawlers as their operators document them today; new ones appear, and not every AI company honours the file. Review the list from time to time.
  • •Crawl-delay is obeyed by some crawlers such as Bing but ignored by Google. The Host line is written only if you add it by hand.
  • •The tester follows Google's documented behaviour. Other engines may differ in small details, such as how they handle wildcards.

Questions

Where does robots.txt go?
In the root of your domain, so it opens at https://yoursite.com/robots.txt. Each subdomain needs its own file. A robots.txt in a sub-folder is not read.
How do I block AI crawlers like GPTBot and ClaudeBot?
Use the “Block AI-training crawlers” preset. It adds a group listing each crawler name with Disallow: / and a second group that leaves everyone else alone. Blocking Google-Extended and Applebot-Extended opts out of AI training without affecting search ranking.
What does Disallow: / mean, and Disallow: with nothing?
Disallow: / blocks the whole site for those crawlers. A Disallow line with nothing after it blocks nothing, which is how you say “everything is allowed”.
Which rule wins when Allow and Disallow both match?
The one with the longer pattern. If both are the same length, Allow wins. For example Disallow: /private/ together with Allow: /private/press.html lets only that one file through.
How do the * and $ wildcards work?
* matches any run of characters, and $ at the end means the address must end there. Disallow: /*.pdf$ blocks every address ending in .pdf, but not /file.pdf?download=1.
Should WordPress sites block /wp-admin/?
Yes, but keep /wp-admin/admin-ajax.php allowed, since many themes and plugins call it from the public site. The WordPress preset does exactly this.

Related tools