Publishers at a Threshold

AI Traffic Is Not All Bad News

Over the past several months, in conversations with publishers, ad ops teams, and platform partners, a familiar concern keeps surfacing. Traffic patterns are shifting. Bots are rendering pages more convincingly. Ads are firing in sessions that never convert. Buyers are asking harder questions about quality. CPMs feel more fragile.

It’s not surprising that much of this gets grouped under a single heading: AI traffic.

Many predicted that AI systems would disrupt distribution, but what we’re seeing now is something more specific. The issue isn’t simply that automated systems are accessing content. The issue is that our monetization infrastructure was built for a web where most traffic was human, and most uncertainty was rare.

That assumption no longer holds.

Automation on the web is not new. Crawlers, monitoring systems, preview tools, QA environments - these have always been part of the ecosystem. What has changed is both the sophistication and the economic surface area of automation. AI agents now render full pages, execute scripts, and interact with content in ways that look increasingly similar to human browsing. When they pass through ad stacks, impressions are generated. When those impressions do not lead to performance outcomes, buyers respond the only way markets know how: by pricing in uncertainty.

The buy side is already sophisticated about invalid traffic, filtering clear fraud and pricing impressions individually based on predicted outcomes. At the same time, buyers - from performance marketers to brand advertisers - still evaluate inventory at the domain or placement level, whether through learning models or allocation decisions. When signals are blended, outcomes are blended. That makes it increasingly paramount for publishers to apply smarter pre-auction routing, separating high-confidence human attention before it mixes with everything else.

There’s a parallel observation on the publisher side. Most yield calculations still divide revenue by total impressions served. If those impressions include a mix of high-confidence human attention plus automation that was never likely to perform, the resulting net CPM reflects a blended denominator. That can obscure what human inventory is actually worth. As traffic composition changes, clearer actor segmentation doesn’t just help buyers price more precisely - it helps publishers understand their own economics more clearly as well.

This traffic is not necessarily fraud. It is, more precisely, non-performing traffic entering premium monetization paths.

And that is a classification problem.

Two Immediate Harms - Both Addressable

In our discussions, two patterns come up repeatedly. Neither is abstract or hand wavy. Both are solvable with better infrastructure.

Agentic Access Without Structure

AI agents are increasingly accessing content directly - sometimes for indexing, sometimes for summarization, sometimes to answer user queries in real time. Some of this activity is constructive. Some of it may be extractive. Most of it is simply opaque.

For publishers, the available choices often feel binary: block aggressively and risk losing legitimate or future value, or allow broadly and accept that usage may be unmeasured and uncompensated.

Neither posture is especially satisfying.

A more mature approach begins with visibility. If automated access can be measured, classified, and understood in terms of actor type and intent, then policy becomes deliberate rather than reactive. Some agents may be blocked. Others may be rate-limited. Others still may be licensed or billed. At minimum, the activity is no longer invisible.

The point is not to assume that all automation is hostile. It is to acknowledge that automation is heterogeneous, and should be treated accordingly.

Automation in the Auction

The second issue is closer to home for revenue teams. When automated traffic renders ads but does not produce meaningful engagement or conversion, it affects how inventory is perceived. Buyers do not typically have the granularity to parse every source of uncertainty. They respond to performance signals.

If uncertainty increases, bids adjust downward.

Many teams we speak with are reluctant to simply block ambiguous traffic, and for good reason. False positives are costly. Traffic that is uncertain is not necessarily invalid. At the same time, allowing all traffic to enter premium demand paths can erode trust over time.

This is where classification becomes economically meaningful. When impressions can be tagged server-side with a validated actor type and a confidence tier before they enter the auction, publishers gain optionality. High-confidence human traffic can be routed to premium demand. Lower-confidence or automated traffic can be downgraded, redirected, or monetized differently.

The framing shifts from “block or allow” to “how should this participate?”

Block when certain. Route when uncertain.

That simple shift can protect yield without discarding value.

A Broader Way of Thinking About Actors

There is a temptation in the current moment to treat the debate as moral or existential: humans versus bots, creators versus AI. That framing is understandable, but it may not be the most productive.

AI agents are not going away. They are increasingly acting on behalf of users - researching, summarizing, recommending, and soon transacting. In many cases, they are intermediaries rather than adversaries.

If we step back, every request to a publisher’s site has three characteristics:

An actor.

An intent.

An economic implication.

The infrastructure of the web has historically treated most actors as interchangeable until proven otherwise. That worked when ambiguity was low. It works less well now.

A more durable approach is to treat content access as a threshold. Actors approach - human readers, AI agents, monitoring systems, crawlers. The publisher defines how each class of actor crosses that threshold. Some subscribe. Some license. Some are rate-limited. Some are blocked. Some are routed to different monetization paths.

This is less about defending against AI and more about acknowledging that the web now includes multiple classes of participants. The goal is not to eliminate automation, but to align it economically.

AI traffic is not all bad news.

When we hear that “AI traffic is hurting publishers,” the instinct is to view automation as the problem itself. But traffic, in isolation, is neutral. The harm arises when we cannot distinguish, classify, and price it appropriately.

If we treat all automated traffic as malicious, we risk closing off future revenue models and productive relationships. If we treat all automated traffic as equivalent to human attention, we risk continued yield erosion.

The opportunity lies in actor-aware economics - in systems that can discern who or what is requesting access, how confident we are in that classification, and how that traffic should be monetized.

We are at a threshold. The web is no longer composed of a single class of visitor, and monetization models built for a mostly human audience are straining under that assumption. Publishers that adapt - by classifying actors, routing traffic intentionally, and aligning policy with economics - can cross into a more holistic model of monetization, one that treats human and AI traffic according to their distinct value and opportunity.

The shift is already underway. The advantage will accrue to those who treat it as infrastructure, not as noise.

Learn more at https://paywalls.net/vai

AI Is Taking What It’s Allowed - and That’s the Problem

As AI systems move from passive indexing to active retrieval, synthesis, and task execution, the line between being crawled and being used has largely disappeared. Content is being consumed programmatically, not navigated. And we believe that programmatic consumption should have programmatic compensation.

Awareness of this shift is not evenly distributed, and neither are the control levers publishers actually use. While CDN- and server-level configurations provide the strongest and most definitive control, they are rarely the primary mechanism for expressing policy. Instead, most publishers rely on the venerable robots.txt file. It is simple, widely understood, and legible to both search engines and AI operators.

Robots.txt has taken on a role it was never designed for: it has become a visible expression of AI access policy. Unfortunately, robots.txt cannot express distinctions like usage class, scope, timing, or compensation, but it has one critical advantage: it functions as a common language. In the absence of more expressive, machine-readable alternatives (which are under development by standards bodies), it has become the place where intent is declared—imperfectly, but publicly.

We’ve performed an automated analysis of the robots.txt file from a number of sites in several verticals and our analysis finds that access policies align less with technical constraints and more with how each sector currently makes money, with inconsistencies that appear partly driven by formal partnerships and partly by simple neglect. One caveat - this analysis does not take into consideration access control technologies like CDN or Web Application Firewalls (WAF).

Some sectors draw firm boundaries, signaling that AI systems represent uncompensated extraction. Others selectively permit access where it supports distribution or commercial partnerships. Still others remain largely silent, effectively allowing broad use while assuming the consequences can be addressed later.

None of this appears coordinated. Much of it is likely transitional. At scale, defaults become expectations. What is allowed today—explicitly or by omission—shapes who gains leverage tomorrow.

The result is that robots.txt is now encoding real positions on control, visibility, and future negotiation power, whether publishers intend it to or not.

Three Verticals, Three Distinct Postures

Looking across news, travel, and real estate, differences in AI access policy are not driven by technical constraints. They reflect how each sector currently captures value—and how clearly that intent is translated into enforceable policy.

News industry policy norms

News publishers have the most restrictive posture, and that intent is now more clearly encoded in robots.txt. Traditional search indexing remains broadly allowed for Google and Bing, but AI crawlers and large-scale collectors (such as CommonCrawl, Perplexity, and Claude-SearchBot) are often blocked. Most major publishers now explicitly deny AI training access for OpenAI, Anthropic, and Google-Extended, with only a few remaining open mostly by omission rather than choice. Larger publishers are also increasingly blocking interactive agents like ChatGPT-User and Claude-User. There are still gaps—some agents are not named—but the direction is clear: publishers are trying to limit unpaid AI reuse, even if robots.txt is a blunt and imperfect tool for doing so.

Inference - AI Model Training Access Policies

Agentic - AI Agent Access Policies

Travel industry policy norms

Travel platforms show a more selective, business-driven posture, and the split in the sector is now clearer. About half the market (especially Expedia-owned brands) openly allows AI indexing, training, and agent access, treating AI traffic as another distribution channel that can drive bookings. The other half (TripAdvisor, Airbnb, Kayak) has moved toward partial or full restrictions, especially around training and agent use. Access decisions tend to follow where each company sees upside or risk in its marketplace. Robots.txt files reflect this uneven maturity: some platforms define agents carefully, while others rely on partial rules or silence that unintentionally allows access. The goal is not controlling AI for its own sake, but protecting or expanding commercial leverage.

Inference - AI Model Training Access Policies

Agentic - AI Agent Access Policies

Real Estate industry policy norms

Real estate platforms remain broadly open by default, with little sign that AI access is being treated as a real policy issue. Search indexing is fully open across major players, and none of the reviewed sites define explicit rules for AI training or agent access. In practice, silence means permission for GPTBot, ClaudeBot, and similar agents. Several robots.txt files appear inconsistent or poorly maintained, reinforcing the idea that programmatic reuse has not yet registered as either a serious risk or a clear opportunity. Unlike media and parts of travel, the sector is still at a pre-policy stage when it comes to AI.

Inference - AI Model Training Access Policies

Agentic - AI Agent Access Policies

Summary

Across all three verticals, the same pattern holds, but the contrast is sharper. Where value feels fragile and tightly tied to proprietary content (news), access is being locked down. Where value scales with distribution and transactions (travel), access is selectively opened or restricted based on business alignment. And where AI has not yet changed the economics in an obvious way (real estate), open defaults persist mostly out of inertia.

These postures directly shape who gets to consume content programmatically, on what terms, and who ends up with leverage. Decisions being made today—sometimes deliberately, sometimes by neglect—are setting expectations for what AI companies assume they can take for free, what platforms may later try to claw back, and where future negotiations will realistically start.

The takeaway - what each vertical is doing today reflects how it believes value flows now, not how it necessarily wants it to flow next.

The Club Good Protocol

Bringing a Balanced Economic Model to the Web

The economics of the open web have been distorted by the rise of AI agents and crawlers. Content, which was once monetized by turning limited space and reader attention into advertising value, is now treated as if it were infinite and free. The surface where attention gathers has shifted away from the pages publishers control to the interfaces of platforms and agents, leaving the underlying content disconnected from the presentation that captures value. As a result, publishers are stripped of pricing power. They risk becoming mere data vendors rather than experience creators, losing their hold on customer relationships in the process. To rebalance the system, they need mechanisms to define and sell inventory that agents recognize as scarce and valuable. The key is to realize that scarcity does not reside in the bits themselves, but in time, context, and proven utility.

To restore a fair marketplace to the open Web we can apply fundamental principles of economics to select a model that works for these new realities. The model that works best is to treat Web resources as a Club Good (https://en.wikipedia.org/wiki/Club_good). This model supports a “non-rivalrous” environment - consuming the goods and services doesn’t prevent others from consuming the same resource (as opposed to advertising where only one ad can chosen for an impression). A Club Good is also characterized by being excludable - the ability to selectively choose who participates. The Web is naturally non-rivalrous however, excludability on the Web still requires some innovation.

In order for this Club Good approach to be as open and scalable as the Web itself, we propose a small extension to the HTTP that we call The Club Good Protocol. This protocol extension is the minimum work necessary to establish the Web as a Club Good and is independent of business model, pricing model or payment method.

This article describes one approach at using The Club Good Protocol to help publishers rebuild their content based revenue streams by using another fundamental economic tool - auctions. This technical white paper provides specific details on how The Club Good Protocol is implemented as an extension to the HTTP and how it leverages the 402 Payment Required response header to make it all work.

There are three primary forms of inventory that publishers can control: base access, quality and first look (or freshness).

The first is base access. This is the everyday availability of baseline content: the ski report that says “40-inch base,” the game score that reads “3–1 in the second period,” the election result that tallies the first precincts. Publishers know this material is valuable, even if it feels like a commodity. By setting a floor price, publishers prevent their work from being treated as worthless. An AI agent can still request as much of this information as it likes, but every call carries a minimum cost. At scale, this changes the calculus: ten thousand low-value requests a day no longer round to zero but accumulate into a predictable revenue stream. Base access does not attempt to extract monopoly rents; it simply establishes that the floor is not zero.

The second form of inventory is quality, which captures the distinctive, nuanced, and contextual content that agents find improves user satisfaction. Not all snow reports are equal. One may simply note the base depth, while another observes that the morning will be powder, the afternoon will crust over, and the east slopes will be best after lunch. For some users, the first fact is enough. For others, the richness of the second report leads to better decisions, higher trust, and more engagement. Agents know this, because they measure their own performance in terms of accuracy, retention, dwell time, and user satisfaction. The challenge is that publishers cannot always know in advance which pieces of content agents will rate as high-utility. The solution is to allow broad sampling at a baseline price, but then introduce floors that rise as demand repeats - essentially dynamic reserve pricing. As a simplified example, a publisher might start by charging $1 CPM. After a few hundred hits, if demand is sustained, the floor automatically rises to $2 CPM, and so on, with a ceiling determined by optimal yield. Each response can signal the next minimum bid, so agents are on notice that continued use requires paying more. This mechanism lets the market discover the true value of quality in real time. It is not a static two-tier representation of the content; it is a sliding equilibrium where the floor ratchets upward as agents prove their willingness to pay.

The third form of inventory is first look, which monetizes freshness. Timeliness is one of the few real scarcities in digital content: there is only one “first five minutes” after the Iowa caucus polls close, the first report of morning snow conditions, or the initial posting of a new house for sale. In these moments, being first carries disproportionate value for agents whose users demand immediacy and who wish to outcompete their rivals.

One way to capture this scarcity is for publishers to sell options through daily auctions, organized by subject domain. Each day, agents bid for tomorrow’s coverage in categories like politics, sports, real estate, or entertainment. The auction allocates exclusivity windows in ranked order: the top bidder wins the first slot, the runner-up takes the next window, and so on. When the event occurs, those winning bidders exercise their rights to the time slots they secured. After the exclusivity periods expire, the content returns to baseline access for all.

For example, on caucus eve, a political news publisher might run an auction for “tomorrow’s results feed.” The top bidder secures the first five-minute window, the second bidder the following five minutes, and the third bidder the five-minute slot after that. Similarly, a real estate listing site could hold a daily auction for “tomorrow’s new property postings,” assigning early-access windows in sequence before listings enter the general feed for AI consumption.

This mechanism blends predictability with competition. Publishers earn guaranteed revenue each day through option sales, while still capturing dynamic upside when multiple agents value the same coverage. AI operators gain confidence that their customers will have the first and best access to quality information—whether that’s a quick take in the moment or a thoughtful summary soon after—which strengthens user trust and loyalty.

Taken together, these three controls form a new set of publisher inventory. Base access ensures that no request is ever priced at zero. Quality allows initial discovery but then captures upside as demand and utility prove themselves. First look transforms freshness into a tradable commodity through options and windowed auctions. In combination, they offer publishers a way to reassert pricing power in an economy that too often assumes their supply is infinite. The scarcity is there—it lives in time, attention, and measurable utility. These inventory controls make it explicit, enforceable, and monetizable.

Join the party. There will be dancing. It will be fun.

Signals / Sept 23, 2025

This week was all about the Benjamins. Four announcements from Microsoft, Cloudflare, Coinbase, Akamai, TollBit, and Google point in the same direction: licensing and payments are being wired directly into the web. What looked speculative a year ago is now being embedded in marketplaces, CDNs, and HTTP itself. For publishers, the takeaway is clear — the excuses for “free use” are evaporating. The plumbing for paid access is being standardized.

Signals

• Microsoft pilots a Publisher Content MarketplaceIn a scoop, Axios reports that Microsoft is developing a marketplace for publishers to license content into Copilot and other AI products, with usage-linked payouts rather than flat fees (Axios). This signals a pivot from bespoke deals toward a more scalable, two-sided platform. The takeaway: If a tech giant like Microsoft builds this, “AI licensing” becomes less about one-off contracts and more about ongoing revenue streams.
• Cloudflare + Coinbase launch the x402 FoundationThe two companies announced a foundation to promote payments via HTTP 402 Payment Required, designed for machine-to-machine and AI agent transactions (Cloudflare). The move pushes payments down to the protocol layer. Translation: If x402 gains adoption, “pay per crawl” could become a baseline assumption for the web and not simply an add-on.
• TollBit partners with AkamaiTollBit is integrating its tolling system with Akamai’s bot detection and edge enforcement, aiming to let publishers monetize AI crawlers instead of just blocking them (TollBit). Embedding monetization at the CDN level means enforcement can scale across thousands of sites. Bottom line: When the edge turns traffic into receipts, “no free crawls” stops being rhetoric and becomes default practice.
• Google unveils AP2 (Agent Payments Protocol)Google launched AP2 as an open standard for AI agents to make purchases with signed mandates verifying user intent and supporting multiple payment methods (Google Cloud; TechCrunch). While framed for commerce, and only riding on top of HTTP and not native to HTTP, this is a fundamental capability of the growing agentic Web. What’s next: If AP2 achieves traction, publishers could bill AI usage directly, with compliance and auditability built in. 

Why this matters

• Marketplaces, protocols, and CDNs are converging. Microsoft, Cloudflare, Coinbase, Google, TollBit, and Akamai are all embedding licensing and payment into their products. This reduces the friction for publishers to charge AI clients.
• Execution now outpaces excuses. A year ago, AI firms could argue that payments were technically impractical. With HTTP 402 standardization and edge partnerships, the tools exist. The debate is no longer if publishers can be paid, but under what terms.
• The leverage is shifting. Publishers that adopt these standards early will set expectations for usage reporting, receipts, and attribution. Those that wait risk being priced or written out of the emerging protocols.

Signals / Sept 19, 2025

Movement towards a transparent, usage-driven licensing economy

A few new items late this week showed how quickly licensing is maturing into a structured marketplace. Meta is re-engaging with major publishers, signaling that licensing is becoming a strategic necessity. Reddit is pressing for dynamic, usage-based pricing, a model that reflects real value exchange. Bloomberg is holding firm on training rights, showing how premium publishers can shape the boundaries of this new market. And Cloudflare is reframing infrastructure itself as a licensing layer, emphasizing that the pipes of the internet can meter and enforce fair access.

The common thread is momentum toward a more transparent, usage-driven licensing economy. Publishers are setting the terms for tomorrow’s licensing framework, building systems that will endure beyond any single contract.

Signals

• Meta re-opens the checkbook Meta has entered talks with Axel Springer, Fox, and News Corp. about licensing news for AI products, reversing its retreat from news deals in 2022 Reuters. The takeaway: Meta is becoming a partner in licensing again because competition now depends on reliable, high-quality content access.
• Reddit pushes for dynamic pricing Reddit is renegotiating with Google and exploring talks with OpenAI, seeking usage-linked terms instead of a flat ~$60M/year Bloomberg. Translation: Licensing is becoming metered and dynamic because platforms recognize that value is created per interaction, not just per year.
• Bloomberg holds the line on training At Digiday’s summit, Bloomberg Media’s CEO explained that they are not licensing for training while “commercial models evolve” Digiday. Our take: Premium publishers are becoming standard-setters because they can wait for clearer separation of training and grounding rights.
• Cloudflare reframes the gate Cloudflare CEO Matthew Prince told WIRED that “robots should pay a ton,” tying this view to Cloudflare’s anti-scraping capabilities WIRED. This means: Infrastructure providers are becoming licensing enablers because they can meter, verify, and enforce access at scale.

Why This Matters

Licensing is evolving into a market built on transparency and measurable usage. Three shifts are defining this new stage:

• Usage-based economics: Deals are becoming tied to interactions and outcomes because that reflects real value creation.
• Litigation as leverage: Lawsuits are becoming inflection points because they expose the risks of unlicensed scraping.
• Standards as infrastructure: Web protocol extensions for pricing such as from Cloudflare and new proposals like RSL are becoming practical pathways because they let publishers automate fair terms at scale.

The good news is that publishers are not just reacting. They are shaping what a functional licensing economy looks like: usage-based, enforceable, and interoperable. For AI companies, the opportunity is to embrace this structure and gain trusted access.

Next week will show whether Meta advances its talks into agreements and whether Reddit’s model reveals specifics. Each step adds clarity to the future design of AI–content partnerships.

Signals / Sept 15, 2025

Lawsuits, Standards, and Leverage: The AI Licensing Wave Rolls On

This week underscored a turning point: the fight over AI’s use of publisher content is moving on two fronts at once — through the courts and through standards. Lawsuits from Penske Media, Britannica, and others are quantifying real traffic and revenue losses. Meanwhile, the launch of a Web based licensing proposal backed by publishers and platforms signals an industry trying to build the rules instead of waiting for courts to impose them. For publishers, the message is clear: leverage is rising, but only if you act while momentum builds.

Signals

• Rolling Stone’s parent takes on Google Penske Media (Rolling Stone, Variety, others) sued Google over AI Overviews, citing ~20% traffic loss and lost ad/affiliate revenue. Our take: this lawsuit moves the debate from theoretical harms to measurable damages — a key step in reshaping negotiations. The Verge
• Britannica vs. Perplexity: credibility on trial Encyclopedia Britannica and Merriam-Webster accused Perplexity of scraping and summarizing their work, even attributing hallucinations to them. This means the case isn’t just about access, but about reputational risk when AI answers misuse content. Reuters
• Anthropic’s $1.5B deal faces judicial skepticism The judge in Bartz v. Anthropic raised concerns about transparency and undisclosed claims in the proposed settlement. The takeaway: courts are signaling they won’t rubber-stamp huge payouts without clarity — raising the bar for future AI litigation. AP News
• Really Simple Licensing (RSL) goes live A new Web licensing proposal lets publishers declare AI use terms (training, inference, royalties) via robots.txt and metadata. Backed by Reddit, Yahoo, Medium, wikiHow, and Fastly. Translation: the industry is moving from theory to tooling — making licensing machine-readable and enforceable. The Verge
• Germany’s Bookwire builds licensing playbook Bookwire is guiding publishers through catalog assessment, restrictions, and deal structures to prep for AI licensing. What’s next: intermediaries may become essential — helping smaller publishers negotiate and operationalize licensing at scale. Publishers Weekly

Why This Matters

The ecosystem is tilting toward structured licensing — through both legal precedent and proactive standards.

• Legal leverage is real. Traffic losses and misattributed content create quantifiable harms courts can’t ignore.
• Standards are emerging. RSL and similar initiatives move publishers from reactive lawsuits to proactive contracts.
• Judicial scrutiny is rising. Settlements won’t sail through without transparency, raising stakes for AI firms.
• Infrastructure is key. Publishers who invest now in enforcement and licensing tools will be positioned to capture value; those who wait risk losing both revenue and leverage.

The battle over AI content use is no longer hypothetical — it’s moving into enforceable standards and high-stakes courtrooms.

From Catalogs to Kitchens

The Cupcake Phenomenon

We’ve noticed a shift in user behavior of consumers in the past year, driven by AI apps and agents, that both improves people’s experience on the Web while also driving down user engagement on web sites - cutting ad revenue. This has led to much angst in the industry & if it continues then sometime later this year it may lead to a Complete Utter Freakout. The cause? It's what we call the “Cupcake Phenomenon” and I explain it further in this article, describing where it is coming from, what is going to happen and what publishers (and eventually advertisers) should do about it. Let’s get cooking!

When you’ve used an AI app and asked for more than a quick fact, have you been surprised at how “fully baked” the response is? It may take half a minute, but what comes back often feels remarkably pleasant — comprehensive, even holistic. That unexpectedly good experience is forming new habits for consumers. Instead of stitching together fragments from different sites, people are starting to expect outcomes that meet their needs in one place. For publishers, that shift is destabilizing. The web used to reward verticals: narrow domains, optimized for search and ad yield. Now those structures feel misaligned with how people actually want to consume and use information. AI has shown that “horizontal integration” is possible, and once people experience it, they won’t go back. People are moving on. The only real choice for publishers and platforms is whether to follow them there or be left behind.

AI is shifting the baseline of what people expect when they turn to the web. People are learning that they can ask for outcomes — a party plan that lays out the menu with a shopping list, curates a playlist, suggests games, and picks a movie for the kids; or an upcoming ski trip that combines the weather forecast, ski conditions, lift ticket prices, gear rental options, condo accommodations, and travel times into a single plan. Instead of pulling together the raw ingredients, people are starting to expect something closer to a finished meal - like getting a cupcake instead of a cup of flour.

Each of these experiences would once have meant hunting across half a dozen sites. The shift is showing up in the data: Business Insider saw a 55% drop in organic search visits between 2022 and 2025, while traffic to the news category overall fell by more than 600 million visits in a year [WSJ]. User behavior tells the same story. Pew researchers found that when Google displayed an AI-generated summary, people clicked through to search-result links only 8% of the time, compared to 15% when no summary appeared. In more than a quarter of those sessions, the AI summary ended the search entirely [Pew]. 

That small shift of going from hunting to crafting is powerful. It turns content consumption into content usage. That creates agency, and once people experience that, it rewires habits. Simply browsing feels clumsy by comparison.

For two decades the web rewarded verticals. Publishers built around narrow subjects like sports, recipes, or gadgets because discovery systems favored specialization. Google’s ranking systems prioritize topic authority, rewarding deep coverage in areas like health or finance [Google, Search Engine Land]. Advertisers reinforced the pattern, paying higher CPMs for targeted audiences — healthcare content, for instance, often commands $15–$35 compared to $4–$12 for travel [Newor Media]. Verticals were easier to package for advertisers, easier to scale operationally, and easier to tune for SEO. The problem is that this structure was designed for distribution mechanics, not for user needs.

The difference is like flipping through the pages of a catalog — pots and pans in one section, ingredients in another, blenders, recipes, plates, flatware scattered throughout — versus walking into a well-stocked kitchen where everything you need is already at hand. A catalog forces you to assemble, compare, and decide. A kitchen is ready for use, built around the outcome you actually want.

For publishers and the platforms around them, this shift is unsettling. The vertical model was built for catalogs: each site neatly contained, monetized, and optimized in isolation. That structure doesn’t translate well when people expect kitchens, integrated experiences where the boundaries between subjects disappear. Advertising systems, identity frameworks, and subscription products that assumed vertical silos now look misaligned with user behavior. The result is that publishers risk becoming invisible. AI intermediaries can pull their content into horizontal experiences without attribution or revenue flowing back. Unless publishers reorganize how they present and package value, the outcome is that the user relationship and the economics that come with it will move elsewhere.

AI has shown people prefer outcomes, not fragments. This moment forces a choice. Publishers and the platforms that serve them can keep defending vertical silos, hoping that search and social traffic will return. Or they can adapt to the horizontal reality that AI has made obvious. The opportunity is to build or join networks that feel like kitchens, places where people can find what they need without friction. That might mean organizing content and value across subjects, federated subscriptions and payments, identity systems that work across domains, or ad monetization that recognizes a user’s broader journey instead of a single site visit. 

The point is simple: the web isn’t broken, but the business models built for it are. Those who reorganize around how people actually use information will lead. Those who don’t will be absorbed into someone else’s feed.

Signals / Aug 25, 2025

From Flat Fees to Meters: AI Content Deals Get Real

This week, the signal is clear: the economics of AI content licensing are tilting from flat fees to usage-based models. Meta is showing it will pay for quality, Perplexity is trying to prove revenue sharing can work, and Amazon is closing the gates. Together, these moves show where leverage is shifting — toward whoever controls the pipes, the data, and the display.

Signals

Meta rents quality instead of faking itMeta licensed Midjourney’s image/video models to power creative tools across Instagram, WhatsApp, and Facebook. Our take: even giants will pay when their in-house efforts fall short. If they’ll license aesthetics, why not journalism or scientific data? FT · TechCrunch

Usage-based licensing isn’t fringe anymorePublishers are pushing beyond flat “training rights” into per-use economics — auditability, per-query rev shares, minimum guarantees. This means contracts are starting to look like utilities billing, not one-time sales. The Information · Nieman Lab

Perplexity tries the rev-share experimentIts new $5/mo Comet Plus will pool $42.5M, with 80% going back to publishers when their content powers answers. The takeaway: adoption is the risk, but they’re at least putting money where their mouth is. Bloomberg Law · Perplexity AI

Courts keep the pressure onA U.S. court kept News Corp’s copyright case against Perplexity alive in New York. What’s next? Lawsuits may not settle economics, but they give publishers leverage at the table. Reuters

Amazon closes the gatesAmazon is quietly blocking AI crawlers from its e-commerce site. Translation: access is moving toward whitelists and contracts. Clearly robots.txt is no longer sufficient. Modern Retail

Crawlers aren’t abstract — they’re measurableFastly reports Meta, Google, and OpenAI made up the bulk of AI crawler traffic April–July. Bottom line - if you’re not logging and managing this, you’re already behind. Fastly · SDxCentral

Why this matters

The licensing wave is moving into its next phase. Flat-fee deals are giving way to metered use. Platforms are shifting from “scrape first, ask later” to hard gates that require contracts. And legal pressure is making AI firms more willing to experiment with payouts.

For publishers, that means three things:

1. Get audit rights and usage metering in place.
2. Expect contracts (not robots.txt) to determine who gets access.
3. Recognize that even Meta will pay for quality — if you price and package it right.

The ecosystem is converging toward structured access and real-time licensing. The question is no longer if AI will pay for content, but how much, and on whose terms.

Darts, Bots, and Missed Shots

The Hard Fork podcast this week featured Perplexity’s CEO, Aravind Srinivas. His remarks about AI agents, publishers, and the future of the web are interesting, but frequently miss the target - like a bunch of darts stuck in a wall next to the board. From a paywalls.net perspective, here is where his framing falls short, and what is really at stake. Get out your popcorn.

Response to Cloudflare’s Accusations of Stealth Crawling
Aravind argued that Perplexity distinguishes between a crawler bot and a user-agent acting on behalf of a human. That is technically valid. But the larger issue is that the rules of engagement are still unstructured. When every actor can claim “this is just a user,” the line between legitimate browsing and industrial-scale scraping blurs. Cloudflare’s ‘stealth crawling’ rhetoric may be overblown, but Perplexity’s dismissal misses the point entirely - like arguing over the name of the fire while the building is burning. Without machine-readable access rules, publishers and intermediaries are left to rely on guesswork and accusations. That is not sustainable. paywalls.net exists to close this gap with structured permissions, clear signaling, auditable logs, and compensation where due.

Aravind’s swipe at Matthew Prince as a “new gatekeeper” also misses the mark. What the ecosystem needs is not another gatekeeper but a neutral access layer. That is why we emphasize open protocols such as HTTP, OAuth, MCP and the nascent AI-Prefs, along with standards-based monetization rather than proprietary toll booths. As a CEO, Aravind should be charting a course to a better future and not shadowboxing with rivals while the real problems go unsolved.

Impact of AI Agents on Web Economics and Content Creation
Perplexity’s framing of creators as either “good” (truth-seeking, wise) or “bad” (spammers, clickbait) is naïve and dangerous. It reduces the diversity of the web into a binary ranking system. Historically, such filters are easily weaponized. A search company should not be in the business of moral classification. The web thrives on pluralism. What it needs is enforceable choice and transparent economics, not value judgments built into agents.

Equally striking is the misunderstanding of how content is funded. Suggesting that “quality creators may be able to charge more” ignores that most publishing is ad-supported. Creators do not sell content directly to readers, they sell reader attention to advertisers. Without a real-time payment layer, talk of users paying more for quality is just a hallucination. This is why licensing infrastructure matters. paywalls.net enables exactly that - policy enforcement, attribution, and microtransactions that make compensation real instead of hypothetical.

Publisher Incentive Model
Aravind’s hint at a model “between Apple News and licensing deals” is the most interesting thread. Apple News is human-curated, while bulk access training deals are blunt instruments. The missing middle is a real-time, per-use licensing layer, which is exactly what paywalls.net is building. For Perplexity to succeed here, they will need interoperable contracts, publisher trust, and infrastructure-grade enforcement. Otherwise, this remains rhetoric.

Future of the Internet
Aravind rejects the idea of a “parallel internet for AIs.” On this we agree: the future is one internet, shared by humans and agents. But coexistence requires rules. Without structured access, we do not get truth-seeking, we get extractive opacity. paywalls.net’s role is to ensure that coexistence is not parasitic but symbiotic, preserving human incentives while enabling AI assistance.

Overall, Perplexity’s vision of an AI-powered personal agent is intriguing and one we share. The idea that agents can take on drudgery while giving people more time for meaningful discovery is powerful. Where the conversation fell short was in connecting that vision to the real economics of the web and the practical needs of creators. That is where collaboration matters. With structured access, fair licensing, and open standards, companies like Perplexity and initiatives like paywalls.net can ensure that the next generation of the web benefits both users and publishers. The vision is sound, but the infrastructure to sustain it must be built together. The ideas are big, the stakes are real, and the execution will decide who wins. Until then, enjoy the show.

Nashville Briefing: What a Good AI Deal Actually Buys

Next week the sell side gathers in Nashville. Before the handshakes and hallway promises, we should get clear on what has actually changed with AI content licensing.

In the last 60 days, AI licensing jumped from polite exploratory calls to signed checks. Some of the deals are sharp. Some are not. Either way, the pace tells you this is not a distant conversation anymore.

This is not the endgame. It is the opening position.

If you run a newsroom or a portfolio of sites, the question is not “should we sign,” it is “what are we selling, and what are we keeping.” Training rights fill an AI model’s memory. That is a one-time transfer of value for you and evergreen value creation for them. Real-time use is where your work shows up in front of a user, and it must be tracked, attributed, and paid each time it happens.

This brief is meant to be useful. What follows is a plain read on recent deals and what a good deal looks like: the rights they actually buy, the credit your brand keeps, and the controls that return value to you. Walk into Nashville with that frame, compare notes with peers, and you’ll be ready to evaluate AI licensing deals on solid footing.

The New Map: Purpose, Attribution, Transformation

If you want to make sense of these agreements, and see the risk buried in the fine print, you need a better map than “training rights” and “distribution rights.” Those were built for the search engine era.

We use a different framework - a content usage taxonomy - built for the AI economy. It cuts through the marketing gloss and tells you exactly what’s being bought, sold, and risked. Three dimensions matter most: Purpose, Attribution, and Transformation.

Purpose: Learning vs. Real-Time Use

Why is the buyer touching your content at all?

• Learning is about building something that persists: an index, a model, a knowledge graph, insights and analytics. It’s “train once, use forever” territory. Once the model has your content in its weights, you can’t pull it back.
• Real-Time Use is about applying your content in the moment: answering a user’s question, powering a surface, generating an experience on demand. Every use is a new transaction, which means every use is a new opportunity to track, attribute, and bill.

These are not interchangeable rights. Training is a one-off premium. Real-time use is a metered utility. Smart deals price them separately.

Attribution: From Source of Truth to Footnote

Once your content is in the output, what does the user actually see?

• Citation is clear, clickable, and on the surface where the user’s eyes land.
• Mention is buried, a hover state, a footnote, a secondary tab.
• No attribution: the user sees the AI as the source.

AI platforms have every incentive to downplay source credit. Unless your deal locks in on-surface citation, you’re betting on good faith in an environment where the economics reward the opposite.

Transformation: Verbatim, Summarized, Synthesized

How does your content survive the trip through the system?

• Verbatim is quoted exactly as you wrote it.
• Summarized is paraphrased or shortened, often losing nuance.
• Synthesized is blended with other sources into a “new” answer, the most common, and the most brand-erasing.

The more transformation applied, the harder it is to prove your content’s value, which is exactly why AI companies like synthesis. Your deal needs to recognize that and price accordingly.

How the Current Deals Map to the Taxonomy

The New York Times’ multi-year deal with Amazon looks impressive at first glance: training rights, real-time use in Alexa answers, and a reported $20–25 million a year. But here’s the danger: it’s a blended scope. Amazon gets to learn from the Times’ archive, a one-way transfer that lives forever in the model weights, and also to use that content in real time without coming back to the well. That’s permanent value for them, recurring cost for the Times. Attribution? Likely there, but Alexa’s UI doesn’t exactly make citations the hero; it’s a polite mention at best. Transformation is almost always summary or synthesis, which means Alexa owns the final voice. Without ironclad language on visible credit and detailed usage telemetry, this is the kind of deal that pays well up front and quietly underperforms over time. At minimum, require per-use telemetry that captures surface, timestamp, query class, and content URI, then tie monthly revenue recognition to those logged events.

Condé Nast and Hearst’s decision to feed Amazon’s Rufus is a different play: pure real-time use in a high-intent commerce environment. When someone is asking for “the best winter boots for icy streets” and your review powers the answer, that’s influence at the moment of purchase. Done right, that’s gold. The risk? In retail flows, attribution is often reduced to a product mention, and your content is rarely left intact. Rufus will summarize or blend your work with competitors’, stripping away your brand’s authority. If the rate card scales with actual conversions, this is sharp. If it’s a flat fee for a crawl and some summaries, Amazon walks away with the upside. Add explicit controls: rate limits, category or SKU scoping, and a hard kill switch if placement degrades or usage exceeds agreed terms.

Gannett’s arrangement with Perplexity should have been a local-news power move. Two hundred markets’ worth of coverage flowing into real-time Q&A could anchor Perplexity’s authority on community-level queries. But instead of focusing on per-answer monetization and placement guarantees, the deal headlines “free Perplexity Pro access for Gannett employees” as part of the value. That’s a perk that will get cheaper every year as AI assistants commoditize, while the cost of producing local journalism doesn’t. Attribution in Perplexity answers is better than nothing, but with heavy synthesis across sources, click-through is not guaranteed. This is the danger of trading durable rights for short-term perks. Define the unit price: per displayed answer that includes a Gannett source, or per unique session where the answer cites Gannett, with a higher rate for top placement or featured callouts.

Google’s active licensing talks might be the most important test case yet. Their Gemini-powered AI overviews combine Learning and real-time use in a single scope, just like the NYT–Amazon deal. But the interface leans heavily on synthesis, and current citation patterns are mention-level at best. Without hard terms on placement, attribution, and per-use payments, this risks becoming the template for “pay-to-be-summarized,” locking in a model where AI is the front door to your work, and the front door is unmarked. Anchor economics to a minimum guarantee plus per-use fees for each overview impression that includes your content, with escalators as usage grows and as new Gemini surfaces roll out.

If I Were Negotiating Today

If I were in those rooms, I wouldn’t start with “what will you pay me?” I’d start with how will we measure value, and how do I stop you from taking more than you’re buying? Because in this market, the price tag is the last thing you negotiate, not the first.

• Separate the scope. Training rights and real-time use are different products with different economics. Price them separately.
• Lock in attribution with teeth. On-surface, in-context, unavoidable. Not a hover state.
• Price for transformation. The more they strip your voice, the more they pay.
• Demand measurement and reporting. Per-use telemetry, placement logs, audit rights. No data = no deal.
• Enforce with controls. Rate limits, kill switch, per-class toggles, machine-readable policy compliance.

Pushback is inevitable. You’ll hear, “If we can’t get terms we like, we’ll just scrape.” Maybe they will, but the market is shifting. The biggest buyers are paying because they need to. The sellers who set enforceable terms now will define the benchmarks. The ones who don’t will be negotiating against those benchmarks later, from a position of weakness.