Cloudflare Now Lets You Choose Which AI Bots Can Access Your Website

Until recently, website owners had a binary choice: let AI bots crawl your content, or block all of them at once. Cloudflare just changed that. On July 1, 2026, the company launched new Cloudflare AI bot management controls that let you decide separately which type of AI crawler gets access to your site — and starting September 15, 2026, new defaults will automatically protect ad-monetized pages without you lifting a finger.

The Old Approach Was Too Blunt

Last year, Cloudflare introduced a one-click “Block AI Bots” setting. It was a step forward, but it treated every AI crawler the same — whether it was indexing your site for search results, scraping your content to train a language model, or acting as an automated agent completing a task on someone’s behalf. Those are three very different behaviors, and the same block policy made no distinction between them. Website owners were stuck choosing between full exposure and full lockdown.

The problem with full lockdown is that blocking all crawlers also hurts your discoverability. If you block search-related crawlers along with training scrapers, you risk disappearing from AI-powered search results entirely. That’s a trade-off most small business owners and web developers don’t want to make.

Three Bot Categories, Three Separate Controls

Cloudflare now classifies AI traffic into three distinct categories, each with its own toggle in your security settings. Here’s what each one actually means:

Search bots crawl your content to index it so it can appear in search results later. These are the crawlers that drive organic discovery. Blocking them means fewer people find you through AI search tools and traditional search engines alike. For most sites, you want these allowed.

Agent bots are different. They act in real time on behalf of a person — things like ChatGPT’s browsing feature or AI assistants that fetch pages to answer a question right now. As you can read in our post on how AI agents are changing how small businesses operate, these automated tools are becoming a normal part of how people interact with the web. Whether you allow them depends on whether you want your content surfaced in AI assistant responses.

Training bots take your content to train or fine-tune AI models. Your content gets permanently absorbed into the model. You get nothing back — no traffic, no referrals, no credit. This is the category most website owners have the strongest objection to, and now you can block it specifically without affecting the other two.

According to Cloudflare’s official announcement, all three controls are available to every customer, including those on the free tier — not just enterprise accounts.

What Happens on September 15, 2026

Cloudflare is also changing its defaults for new domains. Starting September 15, 2026, any new domain added to Cloudflare will automatically have Training and Agent bots blocked on pages that display ads. Search bots remain allowed by default.

The reasoning is straightforward: if a page shows ads, the intended visitor is a human. Training bots and Agent bots aren’t human visitors and don’t see ads, so there’s no business reason to let them through on those pages.

There’s one important catch: if you use the Training crawler block and your site also uses Cloudflare’s managed robots.txt, multi-purpose crawlers like Googlebot, Applebot, and BingBot will be blocked too. That’s because those crawlers are classified as doing both Search and Training. Cloudflare now enforces rules based on all of a crawler’s behaviors, not just one. If you block Training and Googlebot does Training, it gets blocked — even if you also have Search allowed. Cloudflare will notify existing customers ahead of September 15 so you can adjust settings before anything changes on your site.

Why Website Owners Should Pay Attention to This

Your website’s crawlability directly affects how it performs in both traditional and AI-powered search. The way your site is built and configured has a direct impact on which bots reach it and how — as we covered in our post on why your website’s build decides your local SEO ranking. The Cloudflare update adds a new layer to that equation: you now need to think not just about whether bots can crawl your site, but which types you actually want to let in.

For most small business owners, the practical move is to allow Search bots (so you stay discoverable), block Training bots (so your content isn’t used to train models without compensation), and make a deliberate call on Agent bots based on whether you want your content surfaced in AI assistant responses. Cloudflare makes this a few toggles in your security dashboard — no technical expertise required.

Cloudflare also launched a new “BotBase” directory for enterprise customers that gives you a searchable view of every known bot and how it’s classified. And they’re testing a new robots.txt signal called use=reference that signals crawlers may index and excerpt your content, but not reproduce it in full. These are early signals of where content governance on the web is heading.

Frequently Asked Questions

Will blocking Training crawlers affect my Google Search rankings?

It can. Googlebot is classified as both a Search and Training crawler. If you block Training crawlers in Cloudflare’s new system, Googlebot gets blocked too, even though you might want it for search indexing. To avoid this, you need to check your specific Cloudflare settings carefully before September 15, 2026, or opt out of the new defaults if your configuration requires it.

Do these new Cloudflare AI bot management controls cost anything?

No. Cloudflare made all three AI traffic controls — Search, Agent, and Training — available to all customers including those on the free plan. You can configure them now in your zone security settings at dash.cloudflare.com. The BotBase directory and content-use rules are enterprise-tier features, but the core controls are free.

What does “Training” actually mean when it comes to crawlers?

A Training crawler takes your website’s content and uses it to improve or build an AI model. Your words, your product descriptions, your articles — they get absorbed into the model’s training data. The key distinction from Search bots is that nothing comes back to you. There’s no referral traffic, no link, no attribution. You contribute to the model without any compensation or credit.

Cloudflare AI Bot Management Gives You Control You Didn’t Have Before

The shift from “block all bots” or “allow all bots” to granular category controls is a meaningful step. For website owners, it means you can stop treating AI crawlers as a monolith and start making deliberate choices about how your content gets used. Search stays open. Training gets blocked. Agent access becomes a strategic decision. That’s a more sensible default than anything the web has had before. Log into your Cloudflare dashboard and set your terms now, before the September 15 defaults kick in.