Robots.txt Generator
Build a ready-to-publish robots.txt file, with optional AI-crawler blocking and sitemap linking.
How the Robots.txt Generator Works
A robots.txt file lives at the root of your domain (yourdomain.com/robots.txt) and tells search engine crawlers which parts of your site they are allowed to visit. It is a set of instructions, not a security wall, anything you truly need private should be password-protected instead.
How this tool builds your file
Choose whether to allow crawling everywhere, or list specific folders to block, optionally block common AI training crawlers, and add your sitemap URL. The tool assembles a ready-to-use robots.txt file you can copy and publish.
Good practices for robots.txt
- Most small business sites are fine allowing everything, blocking pages by accident can remove them from Google entirely.
- Always include a Sitemap line so crawlers can find your XML sitemap easily.
- Blocking AI crawlers only stops that specific bot from reading your site, it does not affect your Google or Bing rankings.
- After publishing, test it with Google Search Console’s robots.txt report to confirm it is being read correctly.
Using robots.txt safely
Robots.txt is powerful because a single wrong line can block your whole site. The line Disallow: / under User-agent: * tells every crawler to stay away from everything. Only use it on sites that should not appear in search, such as a development copy. A typical WordPress file blocks the admin area, allows the admin-ajax file that some features need and points to the sitemap.
What robots.txt does not do
- It does not remove a page from Google. A blocked page can still appear in results if other sites link to it. To keep a page out of search, use a noindex tag and leave the page crawlable so the tag can be read.
- It does not protect private content. Use passwords for that.
- It does not stop bad bots. It is a polite request that legitimate crawlers follow.
- Blocking CSS or JavaScript files can stop Google from rendering your pages properly.
After publishing, check the file in Google Search Console and visit yourdomain.com/robots.txt in a browser to make sure it shows what you expect.
Frequently asked questions
Where does the file go?
In the root of your domain, so it opens at yourdomain.com/robots.txt.
Does blocking AI crawlers affect my rankings?
No. It only stops those bots from reading your site and does not change Google or Bing results.
Do I need a robots.txt file?
Most small sites work fine with a simple one that allows everything and lists the sitemap.
Next steps: build a sitemap with the XML Sitemap Generator, read our XML sitemap guide and see the llms.txt Generator.