Cloudflare Content Signals: What Google’s Crawlers Follow
A website owner can add a line to robots.txt expressing a preference about AI training. Whether a particular service acts on that preference depends on what it supports.
That’s the practical issue behind SEO Sherpa’s article about Cloudflare Content Signals. It highlights Google’s John Mueller saying he was unaware of crawlers or language models using the directive.
For a business managing its website, the story turns into a specific check: which service are we trying to influence, what control does it document, and what evidence shows our configuration is working?
Current documentation also adds an important qualification to the broad headline. Google doesn’t recognize the directive as a crawl rule, while Cloudflare documents support within its own crawling product. Those details deserve separate attention.
What Google recognizes in robots.txt
Google’s robots.txt documentation lists the fields it supports: user-agent, allow, disallow, and sitemap. Unsupported fields don’t become instructions merely because they appear in the file. Google’s own wording is that it ignores invalid lines.
That means a Content-Signal line doesn’t change Google’s crawl permissions. Google continues to interpret the recognized rules around it.
This distinction helps when investigating a warning. An unsupported line and a rule that prevents crawling are different findings. Inspect the actual rules and the response received by the crawler before deciding what needs to change.
What Cloudflare’s signals are intended to communicate
Cloudflare’s Content Signals Policy defines preferences for how retrieved content may be used:
| Signal | Meaning under Cloudflare’s policy |
|---|---|
search | Search indexing and results containing links and excerpts; the definition excludes AI-generated search summaries |
ai-input | Supplying content to AI models for activities such as retrieval and grounding |
ai-train | Training or fine-tuning AI models |
An abbreviated example from the policy looks like this:
User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /The signal expresses permission for search use and a restriction on training use under that policy. Omitting ai-input leaves that use neither granted nor restricted; the policy’s comment block says an omitted signal withholds both. Cloudflare also publishes explanatory policy comments alongside the machine-readable values, and its own served robots.txt shows what a finished file looks like, with the policy text, signal values, and ordinary crawl rules together.
Cloudflare describes the signals as preferences about how content is used after it has been accessed, not as access controls. For the example above, Google’s parser ignores the Content-Signal line. Its interpretation of the recognized crawl rules remains the relevant question for Googlebot.
The comment’s date and scope matter
In the original Reddit discussion, Mueller qualified his assessment with “AFAIK,” meaning “as far as I know.” His practical point was that crawlers use the directives they support and ignore the rest.
The comment was already covered by Search Engine Roundtable on July 6, 2026. SEO Sherpa’s October article revisits that discussion.
Preserve that context when assessing current support. A comment about known adoption at one point in time needs checking against the documentation available when you make a configuration decision.
Current documentation provides a concrete example: Cloudflare’s Crawl API reference documents a crawlPurposes option for respecting Content-Signal directives in robots.txt, with search, ai-input, and ai-train as supported values, all enabled by default. A separate contentUse option, set to reference or full, respects the policy’s use directive; a URL is disallowed when the publisher’s declared level is lower than the requested intent.
That establishes documented support in a particular product. Each other service still needs its own confirmation. For Google, its published parser behavior is the reference to follow.
Choose a control that matches the intended outcome
Start by writing down what you want to happen to a particular set of pages.
A shop may want public product descriptions discoverable in search while restricting particular training uses. A publisher may have public summaries and subscriber-only material. These situations give the website team different requirements to implement and verify.
For Google, its crawler documentation distinguishes Googlebot from the Google-Extended product token. Googlebot preferences affect Google Search. Google-Extended governs whether crawled content may be used to train future generations of Gemini models and for grounding in Gemini Apps and the Vertex AI API for Gemini; Google says it doesn’t impact a site’s inclusion in Google Search and is not used as a ranking signal.
Read the documented scope before changing either control. A preference about training should be evaluated against the service’s training controls. A decision about Search access should be evaluated against its Search controls.
If the goal is to reject requests from selected crawlers, Cloudflare’s managed robots.txt documentation points to AI Crawl Control for crawler-specific blocking. The same documentation explains that robots.txt compliance is voluntary.
Record the intended outcome beside the selected setting. That gives the next person maintaining the website a reason for the configuration and a clear question to verify.
Inspect the file visitors actually receive
A website’s deployed response deserves attention alongside the file in its codebase.
Cloudflare documents that Bot Preference Sync can prepend managed directives and the Content Signals Policy to your robots.txt when the setting is enabled and at least one of your Search, Agent, or Training policies is set to block. The managed content appears within a few seconds of saving, and Cloudflare says reloading the file verifies it, with no cache purge required. Cloudflare’s managed robots.txt guidance.
Save a copy of the publicly served robots.txt, its HTTP status, and the date checked. Then compare that response with the site’s own configuration and the active Cloudflare settings.
This can reveal a straightforward maintenance problem: the application contains the rules the team expects, while an additional service modifies what is delivered. Once you know which system contributes each section, you can identify who owns the change.
Declarations are preferences, not enforcement
The Content-Signal line states a preference, and the evidence says to treat it exactly that way. Google’s parser ignores the line, and as of Mueller’s July comment no crawler or language model had confirmed using it. Support exists where a product built it in: Cloudflare’s crawl endpoint reads the signals when its crawlPurposes option says to, with all three purposes on by default. Where preferences need teeth, Cloudflare points to AI Crawl Control, because robots.txt compliance is voluntary and a Disallow binds only crawlers that choose to obey it.
The evidence hierarchy matters as much as the controls. A request log shows that a crawler reached your infrastructure and how it was handled. A skipped crawl in the Crawl API shows the signal working in one product. Establishing a downstream use, such as model training, requires different evidence entirely. Keep each conclusion tied to what the observation actually shows.
The resulting division of labor is clean. Declare preferences for services that document support, use hard controls for the rest, and verify what your domain actually serves before trusting either. The file crawlers read is the one on the wire, not the one in the repository.