Will blocking AI training crawlers remove me from AI search?

Not with the crawlers on this page. Operators run separate crawlers for search and for training: OpenAI's OAI-SearchBot and GPTBot, for example, or Anthropic's Claude-SearchBot and ClaudeBot. Blocking the training ones leaves the search ones free to find and cite you.

Does Google-Extended affect my Google rankings?

No. Google says Google-Extended only controls whether your pages can be used for its Gemini models. It doesn't change how your site appears in Google Search.

Do all crawlers obey robots.txt?

Reputable ones do, but robots.txt is a request, not a lock. Some operators also say their user-triggered fetchers, such as ChatGPT-User and Perplexity-User, may not follow it, because a person asked for that page.

Where does the file go?

At the root of your domain, so it loads at yourproduct.com/robots.txt. In Next.js you can use the robots.ts version instead, saved as app/robots.ts. Each subdomain needs its own file.

Should I block my admin and API pages?

It's a good habit, since there's nothing useful there for search. But robots.txt isn't security: anyone can read it, so it lists your private paths. Protect them with authentication too.

Free Maker Tools

Robots.txt Generator for the AI era

Create a robots.txt that keeps search engines and AI search tools able to find your product, blocks the pages that don't matter, and lets you decide which AI crawlers can train on your site.

Free, no sign-up. Everything runs in your browser; nothing you type is sent to us.

Your site

Website

Sitemap URL

Leave it empty if you don't have one yet.

Paths no crawler should visit

One per line. Blocking a page here doesn't remove it from search if other sites link to it. To keep a page out, use a noindex tag instead, and don't block it here, or crawlers can't see the tag.

Hide the whole site from every crawler. Only for staging or private sites: your live site would disappear from search.

AI crawlers

AI search and assistants (they read pages to answer and cite)

OAI-SearchBot · OpenAI · allowedFinds pages to show and link to in ChatGPT search.
ChatGPT-User · OpenAI · allowedVisits a page when a ChatGPT user asks for it.
Claude-SearchBot · Anthropic · allowedReads pages to improve Claude's search results.
Claude-User · Anthropic · allowedFetches a page when a Claude user asks a question that needs it.
PerplexityBot · Perplexity · allowedIndexes pages to show and link to in Perplexity's answers.
Perplexity-User · Perplexity · allowedVisits a page when a Perplexity user asks a question.

AI training (they collect pages for models)

GPTBot · OpenAI · allowedCollects pages that may be used to train OpenAI's models.
ClaudeBot · Anthropic · allowedCollects pages that may be used to train Anthropic's models.
Google-Extended · Google · allowedNot a crawler: tells Google whether pages can be used for its Gemini models. It doesn't affect Google Search.
Applebot-Extended · Apple · allowedNot a crawler: tells Apple whether pages can be used to train its AI models.
CCBot · Common Crawl · allowedBuilds the open Common Crawl dataset, which many AI models are trained on.
meta-externalagent · Meta · allowedCollects pages for training Meta's AI models.

# robots.txt for yourproduct.com
# Made with https://hauntboard.com/tools/robots-txt-generator

User-agent: *
Disallow: /admin
Disallow: /api

Sitemap: https://yourproduct.com/sitemap.xml

Put this file at the root of your site, so it loads at /robots.txt.

What robots.txt Does

robots.txt is a small text file at the root of your site that tells crawlers which pages they may visit. Search engines read it before they crawl, and so do the crawlers behind AI search tools and AI model training.

It doesn't hide pages from people, and it doesn't remove pages from search on its own. Its job is to steer crawlers: away from admin areas and APIs, towards the pages you want found, and, if you choose, away from AI training altogether.

Search Crawlers vs Training Crawlers

The big AI companies now run separate crawlers for different jobs. Search crawlers, like OAI-SearchBot, Claude-SearchBot and PerplexityBot, read pages so their tools can answer questions and link to sources. Training crawlers, like GPTBot, ClaudeBot and CCBot, collect pages that may be used to train models.

For a new product, AI search is a real source of visitors, so most makers allow the search crawlers. Whether to allow training is a values question, and it's yours: the “AI search yes, training no” preset does exactly that. Our guide to getting found in ChatGPT and AI search covers the rest of the picture.

How the Rules Are Read

A crawler follows the group that names it. If a crawler has its own group, it ignores the “*” group. That's why this tool only gives blocked AI crawlers a group of their own; allowed ones follow your general rules, including the paths you block for everyone.
Disallow: / blocks a crawler from the whole site. Allow: / allows everything.
The Sitemap line tells every crawler where your sitemap is, which helps new pages get found sooner.

After You Publish It

Open yourproduct.com/robots.txt in a browser to check it's live.
Check it in Google Search Console's robots.txt report, and make sure your important pages aren't blocked.
Run through our technical SEO launch checklist for the rest.

Common questions

Will blocking AI training crawlers remove me from AI search?: Not with the crawlers on this page. Operators run separate crawlers for search and for training: OpenAI's OAI-SearchBot and GPTBot, for example, or Anthropic's Claude-SearchBot and ClaudeBot. Blocking the training ones leaves the search ones free to find and cite you.
Does Google-Extended affect my Google rankings?: No. Google says Google-Extended only controls whether your pages can be used for its Gemini models. It doesn't change how your site appears in Google Search.
Do all crawlers obey robots.txt?: Reputable ones do, but robots.txt is a request, not a lock. Some operators also say their user-triggered fetchers, such as ChatGPT-User and Perplexity-User, may not follow it, because a person asked for that page.
Where does the file go?: At the root of your domain, so it loads at yourproduct.com/robots.txt. In Next.js you can use the robots.ts version instead, saved as app/robots.ts. Each subdomain needs its own file.
Should I block my admin and API pages?: It's a good habit, since there's nothing useful there for search. But robots.txt isn't security: anyone can read it, so it lists your private paths. Protect them with authentication too.

More free tools

All free tools

Ready to launch? Put it on Hauntboard.

A person reviews every launch. It gets a week on the board, a permanent page Google and AI search can find, and a public count of every visit we send you.

Launch for free