100% free tool

AI crawler robots.txt generator

Settings that let AI visit your site, one visitor at a time. Choose which AI crawlers may read your pages and copy the robots.txt lines that say so.

AI search covers the visitors that read a page when a buyer asks a question. Training covers the ones that collect pages for future models.

Finds pages for ChatGPT search.

May visit a page when someone asks ChatGPT a question.

Reads pages while Claude answers someone.

Navigates the web to improve the quality of Claude's search results.

Surfaces and links websites in Perplexity's search results.

May visit a page to help answer a question, and link to it in the answer.

Controls whether Gemini may use your pages to ground its answers and to train future models.

Collects pages that may be used to train OpenAI's models.

Collects web content that could contribute to training Anthropic's models.

Apple's setting for using your pages to train its AI models.

Collects pages into a public web archive that AI models are trained on.

Adds a Sitemap line, a map of your pages for every crawler that reads the file.

Instant · nothing stored · free, no signup

How it works

Three steps, no account

  1. Pick a starting point

    Allow AI search and block training, allow every AI visitor, or block them all. The first lets assistants read you when a buyer asks and keeps you out of model training.

  2. Change any one crawler

    11 AI visitors from OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl, the same ones the free AI check reads your robots.txt for.

  3. Copy the lines

    One group per crawler, the sitemap line if you gave one, and a table of what blocking each one does, in its maker's words.

Frequently asked questions

Which AI crawlers should I allow?

The ones that fetch pages for answers: OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User and Google-Extended. Rankli's free AI check fails a site that blocks any of them and names which one.

What is the difference between GPTBot and OAI-SearchBot?

OpenAI's crawler page says OAI-SearchBot finds pages for ChatGPT search and GPTBot collects pages that may be used for training. They are set separately, so you can allow one and block the other. Rankli's free AI check reports each one by name.

Does blocking Google-Extended hurt my Google ranking?

Google says no: Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It does decide whether Gemini may use your pages for its answers and its training. Rankli asks Gemini your customers' questions every week and shows whether it names you.

Will blocking training crawlers keep me out of ChatGPT search?

No. OpenAI names OAI-SearchBot for ChatGPT search, and it is set apart from GPTBot. Blocking training means future models learn about you only from other people's pages. Rankli publishes answers on your own site, so the pages AI reads about you are yours.

Where do these lines go?

In /robots.txt at the root of your site, above any "User-agent: *" group. Then run Rankli's free AI check, which reads the file and names every AI visitor it still blocks.

More free tools

Get the checklist for this tool by email

The rules behind the AI Crawler robots.txt Generator, written out as a checklist you can work through on your next draft. The tool above stays free and complete either way.

Ready to get started?

These tools do one step each. Rankli does the whole loop: it reads your site, plans a month of articles, writes them in your measured brand voice, checks each draft against fifteen on-page rules and publishes it to your CMS.

Start free trial

7 days free, then your plan. Cancel before day 8 and nothing is charged.