News & analysis
Cloudflare-OpenAI Pilot Expands What ChatGPT Search Can Find
Cloudflare and OpenAI say a research pilot helped OpenAI's crawler discover and index hundreds of millions of new web pages, using signals from about 1% of Cloudflare's network.
By Roomer Editorial · · 3 min read
What did Cloudflare and OpenAI announce?
On Oct. 15, Cloudflare and OpenAI published a joint blog post reporting early results from a research pilot. OpenAI's crawler discovered and indexed hundreds of millions of new web pages using data from a participating cohort covering roughly 1% of Cloudflare's total network.
Cloudflare first announced the pilot on July 8, and the two companies shared these early results on Oct. 15. The post does not say how often the newly found pages get cited in ChatGPT or how much traffic they've sent to the sites that own them.
What kind of pages is OpenAI finding?
The blog gives three examples: a washing machine's user guide, current passport requirements and fees, and a restaurant's menu and hours. The post says these pages "can be valuable even if few other sites link to them," which is a different discovery method than counting backlinks.
Cloudflare's signals also tell OpenAI's crawler which pages have actually changed, so it can skip pages that haven't been updated instead of re-crawling them on a schedule. The companies are separately testing signals that flag harmful pages, such as a fake login form planted on one page of an otherwise legitimate site, so the crawler can avoid that single page rather than the whole domain.
Does being indexed mean getting cited?
No. The post is explicit that OpenAI independently evaluates and ranks every page it indexes. Appearing in the index through this pilot carries no guarantee of a citation in a ChatGPT answer.
The post also warns that finding new pages can temporarily increase crawl volume on a site while those pages get evaluated, though it gives no figures for how much.
Can site owners control any of this?
Yes, through robots.txt and Cloudflare's AI Crawl Control, which is now available on every plan and shows which AI services are hitting a site, with an option to allow or block each one.
OpenAI runs two separate bots for two separate jobs. GPTBot collects content that may be used for model training. OAI-SearchBot crawls for ChatGPT's search results. Blocking OAI-SearchBot removes a site from ChatGPT's search results, though the site may still show up as a navigational link. Allowing OAI-SearchBot while blocking GPTBot keeps a site searchable in ChatGPT without its content going toward training — OpenAI lists this exact setup as an example configuration on its own crawler documentation page.
A signal about a page, per the post, "doesn't grant permission to access it." Cloudflare says the metadata is being used to improve search and discovery, not to feed model training directly.
Separately, tracking which ChatGPT visits reach a site has been unreliable. Some ChatGPT links lacked UTM tags or referrer data as of early October, though one analyst said a fix began rolling out on Oct. 15.
Cloudflare says the next phase of the pilot will focus on improving how pages get selected, testing faster change-detection signals, and measuring what happens after new content is found. Cloudflare's July 1 post said broad access to this capability is planned sometime in 2026, but the October update gave no specific date.
Roomer Editorial's take
This matters most for the kind of page that normally never gets found: a specific restaurant's hours, a walking tour's current price, the opening times for a museum you reviewed two years ago. Those pages rarely earn backlinks, and this pilot says that's no longer a barrier to getting indexed.
Roomer Editorial's view is that indexing is not the thing to get excited about here. OpenAI's own post says there's no guarantee of citation once a page is found, and it gives no data on how much traffic indexed pages actually received. Treat this as a change to how content gets discovered, not a new source of visitors.
The robots.txt split is the part worth acting on now. A creator with a destination guide site can allow OAI-SearchBot so their hotel reviews and itinerary pages can surface in ChatGPT search, while blocking GPTBot if they'd rather their writing not become training data. That's a real, available choice — check AI Crawl Control or your robots.txt file to see which bots are currently allowed on your site, and decide deliberately rather than by default.
Source: Search Engine Journal
