Enter the required information
Use the fields in the tool above and keep units and formats consistent.
Build a valid robots.txt file with Disallow, Allow and Sitemap lines, plus rules for AI crawlers. Learn why blocking a page does not keep it out of Google.
Crawlers happily wander into /wp-admin/, filtered search pages and cart URLs that add nothing to search results. A robots.txt file at the root of your domain tells bots which paths to leave alone. Search engines check it before crawling anything else. A robots.txt generator builds the file with correct syntax. That matters, because one misplaced slash can block a whole site.
The file must open at https://yourdomain.com/robots.txt, in plain UTF-8 text. Each subdomain needs its own copy. Google reads only the first 500 KiB and ignores anything beyond that.
| Directive | Purpose | Google support |
|---|---|---|
| User-agent | Names the crawler a group of rules applies to; * means all | Supported |
| Disallow | Blocks crawling of a path | Supported |
| Allow | Reopens a more specific path inside a blocked one | Supported |
| Sitemap | Points to your XML sitemap by full URL | Supported |
| Crawl-delay | Asks bots to pause between requests | Ignored by Google, honoured by Bing |
| Noindex | An old unofficial rule | Not supported since 2019 |
Paths are case-sensitive, so Disallow: /Blog/ will not block /blog/. The * wildcard matches any run of characters. A $ marks the end of a URL, as in Disallow: /*.pdf$.
User-agent: *Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.phpDisallow: /?s=Disallow: /cart/Sitemap: https://example.com/sitemap_index.xmlThis keeps bots out of the admin area, internal search results and the cart. It leaves the AJAX endpoint open, since some themes call it to render content. Never block your CSS or JavaScript folders. Google needs those files to see pages the way visitors do.
Many publishers now add groups for AI bots. GPTBot is OpenAI's training crawler, while OAI-SearchBot powers its search features. Google-Extended controls Gemini training use without touching Google Search. CCBot belongs to Common Crawl. Compliance is voluntary, so treat these lines as a request, not a lock.
This is the most misunderstood point about the file. Disallow stops crawling, not indexing. If other sites link to a blocked URL, Google can still list it, usually without a description. Search Console reports these as "Indexed, though blocked by robots.txt".
To keep a page out of results, leave it crawlable and add a noindex robots meta tag or X-Robots-Tag header. Google has to crawl the page to see that instruction. Blocking the same URL in robots.txt hides the noindex and defeats the purpose.
The file is also public, and anyone can read it. Never list secret folders there, and protect private areas with a password. Whatever the ToolsMate robots.txt generator gives you, check the live file in the robots.txt report in Google Search Console.
Use the fields in the tool above and keep units and formats consistent.
Review the result shown in the output area.
Confirm that the result matches your situation before using it for a decision.
A standard crawler rule set and sitemap line are generated.This describes the core logic used by the tool. Check any assumptions that may affect your result.
Use accurate inputs and keep units consistent.
Review the assumptions before using the result for a decision.
Compare important results with an independent or official source.
Not reliably. It stops Googlebot from crawling the page, but the URL can still be indexed if other pages link to it. To remove a page from results, use a noindex tag and keep the page crawlable.
Place it in the root directory so it loads at yourdomain.com/robots.txt. A copy inside a subfolder is ignored. On WordPress, SEO plugins such as Yoast and Rank Math let you edit robots.txt from the dashboard without FTP access.
It blocks compliant crawlers from every page on the site. That is useful on a staging server but disastrous on a live one. Plenty of traffic drops after a relaunch trace back to this single line carried over from development.
It depends on your goals. Blocking training crawlers keeps content out of future model training, if the bot complies. OpenAI uses GPTBot for training and OAI-SearchBot for search features, so you can block one and allow the other.
No. An empty file, or a Disallow line left blank, allows everything. A missing file returning a 404 is treated the same way. Trouble starts only when the server returns a 5xx error, which can make Google pause crawling.
Google generally caches robots.txt for up to 24 hours, so most changes apply within a day. After a serious mistake, request a recrawl from the robots.txt report in Google Search Console.
Yes. Create a separate User-agent group for each crawler with its own Allow and Disallow lines. A crawler obeys only the most specific group matching its name. It ignores the general asterisk group once a named group applies.
Explore other useful tools across the site.