AI Agents Are Now a Second Audience on Your Site. How Publishers See Agent Traffic and Get Paid for It in 2026
Publisher in a Box editorial teamUpdated
Key takeaways
Cloudflare says more than half of internet traffic was not human this year, and daily AI agent requests on its network grew more than 1,700% in a year.
An agent reads a page for a person who may never visit, so on an ad supported page it costs bandwidth without producing an impression.
ChatGPT-User and Google-Agent are the user triggered agents to look for in your logs, and both may ignore robots.txt.
Search, training and agent access are now separate settings, and keeping search open keeps you citable.
Pay Per Crawl, Pay Per Use and per request charging reach smaller publishers, and all of them are early.
Table of Contents
On October 27, WIRED reported that some news outlets are now writing for a reader that is not a person. Axios has started building feeds for AI agents, and other publishers are shaping articles so that an agent can rebuild them for the human who sent it. A month earlier, Cloudflare, which sits in front of more than 20% of the web, said that for the first time this year more than half of internet traffic was not human, and that daily requests from AI agents on its network had grown by more than 1,700% in a year.
For a Digital Publisher whose site earns from ads, that raises a practical question about how much of your traffic is now software reading on someone's behalf, and whether any of it is worth money. This guide covers what an agent is, how to find agents in your logs, the controls that separate being found from being used, and the payment routes that exist today. It sits inside GEO, often referred to as AI search optimization, because the same agents that read your pages are the ones deciding which source to cite. Every figure comes from the company that published it, and we mark where their words stop and our reading begins.
What an AI agent is, and why it is not the same as a crawler
Cloudflare's September 30 post, "The Internet has a second audience," draws the line clearly. A training crawler collects your pages to build a model. An agent is "software fetching pages on a person's behalf, often because the human asked a chatbot something." Cloudflare calls agents the fastest growing part of automated traffic and says that "an agent comes back each time someone asks about that content, so this traffic grows with how many questions people ask, not how much you publish."
That changes who is on the other end of the request. An agent usually has a person behind it with a job to get done, which is why Cloudflare warns that if you turn an agent away, "you may be turning away the person who sent it." The catch for an ad supported site is that the agent reads the page without seeing the ads, and as Cloudflare puts it, "an ad only pays when a person sees it." So an agent visit costs bandwidth and may lead to a citation, but on its own it produces no impression.
1,700%
Growth in daily requests from AI agents on Cloudflare's network over the past year
Source: Cloudflare, The Internet has a second audience, September 30, 2026
How much of the web is now automated
At the end of 2024 Cloudflare's network handled an average of 63 million HTTP requests a second, and by its September 30 post that had almost doubled to 115 million. The purpose behind crawler visits shifted at the same time. In spring 2025, 22% of the crawler requests Cloudflare saw were for AI training, according to the crawlers' own stated purpose, and by June 2026 the figure was 52%, so more than half of crawler activity now collects content to build models rather than to send readers back.
Share of crawler requests made for AI training on Cloudflare's network
percent of crawler requests
Source: Cloudflare, The Internet has a second audience, September 30, 2026
Those are network figures, not a measurement of your site, which is why the first step is to look at your own logs.
How to find AI agents in your own traffic
The major platforms publish the names their agents use, so you can search your server or CDN logs for them.
ChatGPT-User. OpenAI's crawler documentation says ChatGPT may visit a page with this user agent "when users ask ChatGPT or a CustomGPT a question." OpenAI adds that it "is not used for crawling the web in an automatic fashion," that "robots.txt rules may not apply" because a user started the action, and that it "is not used to determine whether content may appear in Search."
OAI-SearchBot and GPTBot. These are OpenAI's automatic crawlers. OAI-SearchBot surfaces sites in ChatGPT search and GPTBot crawls content that may be used for training, and OpenAI says each setting is independent of the other.
Google-Agent. Google's list of user triggered fetchers, last updated August 19, 2026, says Google-Agent "is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request." Google notes that because these fetches are requested by a user, user triggered fetchers "generally ignore robots.txt rules."
Names can be faked, so the stronger check is identity. Cloudflare says that with Web Bot Auth, operators including OpenAI, Google and AWS cryptographically sign their agents' requests, and both OpenAI and Google publish the IP ranges their agents use.
Being found and being used are now separate settings
In July, Cloudflare replaced its single "block AI bots" switch with separate Search, Agent and Training controls, available on every plan including Free. Fewer than 1% of sites on Cloudflare block search crawlers while 17% block training, because site owners, in Cloudflare's words, "want to be found without being exploited." On September 15 it added Disallow AI Training, which keeps a site indexed for search while telling mixed use crawlers not to train on it, and Cloudflare says Apple, Google and Microsoft have committed to honor it.
An agent visit is a reader you will never meet. Decide which pages it may read for free, which pages it should pay for, and which pages only a person should see.
Here is how we would map those settings for a typical ad supported publisher. This is our reading, built on what each company documents.
1. Keep search crawlers open. OpenAI says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. That is the GEO door, so we would leave it open. 2. Decide on training separately. Disallowing GPTBot, or using Cloudflare's training controls, keeps your archive out of model training without touching search. 3. Treat agent access by page type. Cloudflare's recommended setup for ad supported sites is to disallow training and block agents on pages that carry ads. Because OpenAI says ChatGPT-User is not used for search inclusion, blocking it on ad pages should not remove you from ChatGPT search, though the agent cannot read that page for its user. User triggered agents may not follow robots.txt, so this belongs in your CDN or firewall.
How publishers get paid for agent and crawler traffic
Cloudflare counts more than 50 publisher and AI deals since 2023, nearly all bespoke deals between large publishers and large AI companies. These are the routes that reach smaller sites, and every one of them is new.
Pay Per Crawl. Cloudflare's documentation describes a feature where a site sets a price, and each AI crawler request either carries payment intent for normal access or receives an "HTTP 402 Payment Required" response with pricing. Cloudflare acts as the merchant of record. In its July 1 post, Cloudflare called crawling "a crude measure of value," because "a single page might be crawled once and then cited in thousands of answers, or crawled over and over and never used at all."
Pay Per Use. So Cloudflare is reshaping Pay Per Crawl into Pay Per Use, which pays when content is used rather than fetched. Each buyer is a verified crawler that defines what counts as use and what it will pay, and publishers can opt in and opt out. The partners named in July were Ceramic.ai, which pays opted in publishers when their content appears in its search results, and You.com, where agents can pay for a specific piece of premium content. Publishers see what was used, when and what they earned, and Cloudflare itself calls this an experiment.
Charging agents per request. Cloudflare's Monetization Gateway, in closed beta for eligible U.S. Cloudflare customers, lets a site put a price on a page, dataset or tool. When a rule matches, Cloudflare returns an HTTP 402 response using the open x402 protocol and the agent pays the seller directly. Its example is a sports statistics site charging a fraction of a cent each time an agent asks who leads the league in assists.
A fourth route is selling to the person who owns the agent. WIRED reports that Axios launched a feed this fall giving the AI agents of Axios Pro Deals subscribers access to its merger and acquisition content, which suits paid content more than an open ad supported site.
What this is worth today, and what it is not
None of these routes replaces ad revenue yet. University of Copenhagen professor Rasmus Kleis Nielsen told WIRED the market for publishing to agents is "emerging" but "extremely uneven," and Cloudflare launched both payment products as betas because "pricing and discovery aren't solved yet." We would treat agent payments as an experimental line until real payouts show up in your own account.
What does pay off now is being the page an agent picks. Axios chief operating officer Allison Murphy told WIRED that the outlet's short, structured style has made it a favorite for AI to pull from. "Time is money for crawlers," she said. "They don't want to have to go three quarters down the page to find the main point." That is the GEO case in one sentence, and it matches what we cover in how agentic AI search picks which publishers to cite. A clear answer near the top makes a page easier for an agent to use, and a citation is how a person ends up on your site where the ads can pay.
1. Pull a week of logs and count requests from ChatGPT-User, OAI-SearchBot, GPTBot and Google-Agent. 2. Sort your pages by job, separating answer pages agents want from ad pages that only pay when a person reads them. 3. Set search, training and agent rules separately, in robots.txt for automatic crawlers and in your CDN for user triggered agents. 4. On Cloudflare, read the Pay Per Use offer and opt in only where the terms are clear, since you can opt out later. 5. Move the main point of your top answer pages into the first paragraph, which serves agents and human readers at once.
If you want your site set up to be found and cited by AI search and the agents that run on it, including ChatGPT, Gemini, Perplexity and Google's AI answers, the GEO Authority System is how Publisher in a Box builds that visibility for publishers, from the crawler and agent settings to the answer pages AI engines cite.
Frequently asked questions
What is AI agent traffic?
It is software visiting your site on a person's behalf, often because someone asked a chatbot a question. Cloudflare says agents sit between humans and traditional bots, with a person usually behind them, but they do not respond to ads.
How do I see AI agents in my analytics?
Look in your server or CDN logs for user agents such as ChatGPT-User and Google-Agent. On Cloudflare, AI Crawl Control also shows who is crawling and what they take.
Will blocking AI agents remove me from ChatGPT search?
OpenAI says ChatGPT-User is not used to decide whether content appears in search. OAI-SearchBot controls that, and sites that opt out of it are not shown in ChatGPT search answers, though they can still appear as navigational links.
Can a small publisher get paid by AI companies?
Programs such as Cloudflare's Pay Per Crawl and Pay Per Use are built for sites that will never sign a direct licensing deal. Cloudflare calls Pay Per Use an experiment, so treat the income as uncertain until it reaches your account.
Does robots.txt stop AI agents?
Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Google says its user triggered fetchers generally ignore robots.txt, so agent rules belong in your CDN or firewall.
Sources
Cloudflare, The Internet has a second audience, September 30, 2026, blog.cloudflare.com
The team behind 300M+ managed followers. We help publishers scale traffic, revenue, and audience across Facebook, Google Discover, and syndication networks.
