Skip to content
Get a Free Audit

Robots.txt generator with AI crawler control

Choose which search engines and AI bots can use your website: allow AI search, block AI training, or block all AI. Get a correct robots.txt, test any link against any bot, check an existing file and see which bots visit you. Free, on your device.

This tool needs JavaScript. Please turn it on, or open the page in a current browser.

How to use this tool

  1. Pick your policy Choose a preset such as “Allow AI search, block AI training”, and your website platform.
  2. Add paths and sitemap List pages to hide (cart, admin, search results) and your sitemap link.
  3. Review each bot Bots are grouped by what they do. Tap any bot to allow or block it.
  4. Copy or download Copy the file or download robots.txt and upload it to the top folder of your site.
  5. Test and check Test links against any bot, check an existing file for mistakes, or read your server log to see which bots visit.

Before you upload

  • Never block Googlebot or Bingbot unless you want to leave search.
  • Do not block CSS and JavaScript folders; Google needs them to see your pages.
  • robots.txt must be at https://your-site/robots.txt. Subdomains need their own file.

What robots.txt can and cannot do

  • It asks bots not to crawl. It does not remove pages from Google; use a noindex tag for that.
  • Google-Extended and Applebot-Extended only control AI training. You stay in Google and Apple search.
  • Bots that hide their name, and some user-triggered AI fetchers, may not follow robots.txt. Use your host or CDN bot settings for those.

Keep it current

  • New AI bots appear often. Check this list every few months.
  • Google may cache robots.txt for up to 24 hours, so changes are not instant.
  • Keep the file under 500 KiB; Google ignores anything after that.

Questions about this tool

Will blocking GPTBot remove me from ChatGPT search?

No. GPTBot is for training. ChatGPT search uses OAI-SearchBot, which you can allow separately.

Does blocking Google-Extended hurt my Google ranking?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.

Can robots.txt stop all AI bots?

No. It is a request. Most named bots follow it, but some user-triggered fetchers say they may not, and unnamed bots cannot be targeted.

Is my log file uploaded?

No. The log analyser reads the file in your browser only.

How fast do changes work?

Google usually re-reads robots.txt within 24 hours. Meta and Amazon say about 24 hours; DuckDuckGo says 72 hours.