· Notes for owners
Check which AI crawlers your site lets in before AI answers skip you
Cloudflare introduced a system to evaluate how well Verified bots follow site rules. Here is how to make sure your robots.txt and CDN let the right crawlers in.
On October 16, 2026, Cloudflare introduced a system to evaluate whether Verified bots honor site preferences, with initial signals of whether a bot respects instructions and whether it responds to direct feedback. It also plans to share these scores publicly on Radar.
What Cloudflare's bot scores measure
Cloudflare puts the idea in one line:
A driver’s license tells you who someone is and that they passed a driving test, not whether they’ll drive responsibly in the future.
A bot can be known and still ignore your preferences, overwhelm your server or evade controls. Cloudflare adds that this is often a misconfiguration or a bug rather than malice. So it is building an explainable trust band from what a bot actually does after your site tells it something. The first two signals are whether the bot respects instructions and whether it responds to direct feedback.
This matters to you because a score like that measures a bot against what your site says. If your robots.txt is stale, or says something you never meant, the measurement means little. The same file also decides whether the crawlers behind AI answers can read your pages at all. Google lists crawling allowed in robots.txt, and by any CDN or hosting infrastructure, among the fundamentals for appearing in its AI features (Google AI features).
There are two layers to keep straight. Cloudflare's documentation says robots.txt expresses preferences and does not prevent access at a technical level (Cloudflare robots.txt setting). Your CDN's bot rules do the enforcing. Your site can say one thing in robots.txt while the CDN does another.
What to do, in order
Hand this list to whoever touches your website and CDN settings.
- Read your live robots.txt. Open yourdomain.com/robots.txt in a browser and save a copy. Ask who last edited it and why.
- Split search from training. Where a platform documents them, search, user-fetch and training are separate tokens. Anthropic names ClaudeBot (training), Claude-User (user requests) and Claude-SearchBot (search indexing), and says disabling either of the last two may reduce visibility (Anthropic crawler article). Google-Extended is a standalone product token for Gemini training and grounding (Google crawlers). Decide for each one: allow or block, on purpose.
- Check the CDN, not just the file. Ask whoever manages it whether any bot rule or challenge could be refusing verified crawlers you want. Cloudflare notes that a zone can show a Disallow while its enforcement rules do not block the crawler. The reverse can also happen: the file says welcome and a rule says no.
- If you are on Cloudflare, make the two agree. Its managed robots.txt serves search=yes and ai-train=no. It also says a blank ai-input signal means no stated preference, so set that one deliberately (Cloudflare robots.txt setting). Cloudflare's post introducing Bot Preference Sync is titled "Say it once" (Bot Preference Sync).
- Test the right way. Ask for evidence of what the real, verified crawlers received from your site, not a screenshot of your own browser.
- Write down the decision. One short page: which crawlers you allow, which you block, and who can change it. When Cloudflare's scores for Verified bots go public, you will want that page to read them against.
The traps
One rule for everything. Blocking training tokens and search tokens in the same rule can remove your site from ChatGPT search, Perplexity or Claude answers. Training and search are different jobs. Treat them separately.
Trusting the file while the CDN says no. robots.txt states a preference and the firewall enforces. If a bot rule or challenge turns the verified crawler away, the permission in your file does nothing.
Testing with a spoofed user agent. Someone sets their browser to claim to be a crawler, loads the page from a desktop connection, sees it work, and calls it done. Verified-bot rules key on published IP ranges, so that probe proves nothing either way.
What to expect after you change it
- Cloudflare says it will share these scores publicly on Radar and expand how it measures.
- OpenAI publishes crawler pages for OAI-SearchBot, GPTBot and ChatGPT-User, and Perplexity publishes one for PerplexityBot and Perplexity-User. Read each page for its current rule before you decide.
- After any robots.txt change, check again later instead of assuming the change worked.
Where this leaves you
The news is about bots being held to account. Your part is smaller and more useful: say clearly what you want, make sure your CDN agrees, and check that the crawlers you want are getting in. None of that is hard, but it does need an owner. Being that owner is our work: we map how your business runs, build the software that fits it, and run it for you. If you would like someone to look at your crawler settings alongside the rest of your setup, book a discovery call.