Cloudflare Blocks AI Crawlers: What It Means for Backlinks | Rixot

On September 15, 2026, Cloudflare flipped a default that touches a fifth of the web, and it quietly changed what a backlink can do for you. Pages protected by Cloudflare now block "mixed-use" AI crawlers by default on ad-supported sites, while classic search crawlers like Googlebot keep walking through. For anyone buying links, that creates a split that did not exist before: a link can still pass full value for Google search and pass no value toward AI-search citations, purely because of the donor's bot settings. Here is what changed, why it matters for your link profile, and the one check to run on a donor before you pay.

What Cloudflare actually changed

Cloudflare's position is that creators, not AI companies, should decide who gets their content. As its pay-per-crawl announcement put it: "If a creator wants to block all AI crawlers from their content, they should be able to do so. If a creator wants to allow some or all AI crawlers full access to their content for free, they should be able to do that, too." The September 15 change turned that principle into a default.

The mechanics matter. On ad-supported pages, Cloudflare now blocks crawlers in the training and agent categories by default, and blocks "mixed-use" crawlers that do not cleanly separate what they are doing. Pure search crawlers stay allowed, which is why Googlebot is unaffected. Publishers can also opt into Pay Per Crawl, setting each crawler to one of three states - "Allow" (free), "Charge" (require payment) or "Block" - with Cloudflare acting as merchant of record. The newer Pay Per Use model, announced July 1, 2026, pays publishers on the output event, when their content surfaces in an AI-generated answer, rather than on every crawl.

The crucial scope detail: this default is applied to new domains and free plans. An established site already on Cloudflare does not flip on September 15. But that skew matters for link buyers, because a large share of cheap, newly spun-up donor sites are exactly the new-domain, free-plan properties this hits hardest.

Why this matters for your backlinks

A backlink only does work if the page it sits on can be read by the thing you want to influence. For classic SEO, that reader is Googlebot - and Googlebot still gets through Cloudflare's new default. For AI search, the readers are a different set of bots entirely, and those are the ones now being turned away. The result is a two-channel split in a single link:

Crawler	Purpose	Cloudflare default (ad pages)	Effect on your link
Googlebot	Classic search	Allowed	Full value for Google rankings
GPTBot (OpenAI)	Training / agent	Blocked	No path into ChatGPT answers
ClaudeBot (Anthropic)	Training / agent	Blocked	No path into Claude answers
PerplexityBot	Agent / answer engine	Blocked	No path into Perplexity citations

Read that as a warning for the AI-visibility goal so many link buyers now have. We covered in our piece on where AI-cited links come from that a growing share of link budgets is aimed at getting mentioned by ChatGPT and Perplexity, not just ranking on page one. If the donor page is behind Cloudflare's new default, that money buys nothing toward AI citations - the AI crawlers never see the link. For pure Google SEO, the link is fine. For AI search, it may be invisible.

The donor you bought may already be invisible to AI

This is not hypothetical. A cheap link placed on a brand-new site on a Cloudflare free plan is, by this default, likely blocking every AI crawler right now. You would never know from a DR score or a traffic chart. Checking is simple, and it is the check almost nobody runs:

• Fetch the page as an AI crawler. Request the donor URL with a GPTBot or ClaudeBot user-agent and see whether you get the content or a block. A 403 or a challenge page means the link is invisible to that engine.
• Read the robots.txt. Explicit Disallow rules for GPTBot, ClaudeBot, Google-Extended or PerplexityBot are a direct signal the site has chosen to keep AI out.
• Check whether it is a new domain on a free plan. Those are the properties Cloudflare's default targets. A freshly registered donor is the highest-risk case.
• Confirm the page actually renders for a bot. A link inside content that only a logged-in human sees passes nothing to any crawler, AI or otherwise.

What to check before buying a link for AI visibility

If your goal includes being cited by AI engines, the donor's bot policy is now a first-class buying criterion, sitting right next to relevance and real traffic:

• AI-crawler access. The donor should let GPTBot, ClaudeBot and PerplexityBot fetch its content. This is the new, decisive signal, and the one the market is not checking yet.
• Googlebot access. Still allowed under the default, but confirm it anyway - a misconfigured site can block everything.
• Real traffic and relevance. Unchanged. An accessible page with no readers and no topical fit is still a weak link.
• Established, not freshly spun up. The new-domain, free-plan donors most likely to block AI crawlers are also the lowest-quality placements for every other reason.

The through-line is that AI accessibility and link quality point the same way. The donors that block AI crawlers by default are disproportionately the cheap, new, free-plan sites you should already be avoiding.

Where this leaves link buyers

Cloudflare just made a donor's bot policy part of a link's value. A placement on a real, established site that lets AI crawlers in can feed both Google rankings and AI citations. A placement on a new, ad-stuffed, free-plan site may feed neither. That is exactly the filter behind how Rixot sources placements: real-traffic, established donors that search and AI crawlers can both actually reach, rather than disposable domains that look fine in a metrics tool and are walled off from the engines that matter. As AI search keeps eating clicks, "can the AI even see this link" stops being a technicality and becomes the question.

FAQ

Does Cloudflare block Googlebot?

No. Cloudflare's September 15, 2026 default blocks mixed-use and training/agent AI crawlers on ad-supported pages, but single-purpose search crawlers like Googlebot remain allowed. Classic Google SEO is unaffected by the default.

Which AI crawlers does the Cloudflare default block?

Crawlers in the training and agent categories, and "mixed-use" crawlers that do not separate their functions - in practice GPTBot, ClaudeBot, PerplexityBot and similar. The block applies by default on ad-supported pages for new domains and free plans.

How does this affect my backlinks?

If the donor page hosting your link blocks AI crawlers, that link cannot contribute to citations in ChatGPT, Claude or Perplexity, even though it still passes value for Google search. For AI-visibility goals, an AI-blocked donor is effectively dead.

How do I check if a donor blocks AI crawlers?

Fetch the page using a GPTBot or ClaudeBot user-agent and see whether you get the content or a 403/challenge, and read the site's robots.txt for Disallow rules against GPTBot, ClaudeBot, Google-Extended or PerplexityBot. New domains on Cloudflare free plans are the highest-risk.

What is Cloudflare Pay Per Crawl?

It lets publishers set each AI crawler to Allow (free), Charge (require payment) or Block, with Cloudflare as merchant of record. The related Pay Per Use model, announced July 1, 2026, pays publishers when their content surfaces in an AI-generated answer.

Sources

• Cloudflare - "Introducing pay per crawl," blog.cloudflare.com
• TechCrunch - "Cloudflare's new policy pushes AI companies to pay for publishers' content" (July 1, 2026), techcrunch.com
• The Next Web - "Cloudflare to block AI crawlers, pay publishers," thenextweb.com