IAB Tech Lab proposes rules around AI content scraping

Publishers have spent the past few years watching their traffic slide — in some cases by more than 50% from search alone — while AI systems ingest and summarize their content at scale. The imbalance was obvious. The infrastructure to enable AI is highly commercialized. The feed information is missing.
The IAB Tech Lab thinks it’s time to fix that.
Today, the standards body released version 1.0 of its Content Monetization Protocols (COMP) specification for public comment. The idea is straightforward: create a standardized way for publishers and AI systems to establish commercial terms before any crawling or consumption of content takes place.
Creating payment channels for information
“AI systems require chips, power, and knowledge. Knowledge is the only input to that equation that doesn’t yet have a consistent commercial infrastructure around it,” said Anthony Katsur, CEO of the IAB Tech Lab, in a statement. “If we expect high-quality content to continue to fuel AI-driven products, we need clear terms of engagement and an approach that supports compensation, accountability, and long-term sustainability. CoMP is designed to help the industry move in that direction.”
If AI products are to continue to rely on high-quality journalism and premium content, there needs to be a consistent mechanism to support compensation and accountability. CoMP is designed to create that commercial layer.
Instead of every publisher cutting one-time licensing deals or building custom integrations with each AI platform, CoMP proposes a shared protocol. Content owners can sign consents and terms of sale in a standardized way. AI systems can look for those signals before accessing or using content.
In theory, that reduces friction on both sides.
Not a blockade, but a bridge
CoMP is not intended to install paywalls, robots.txt files or other access controls. The framework assumes that publishers already have strong blocking mechanisms in place at the edge or CDN level.
Your customers are searching everywhere. Make sure it’s your product he appears.
The SEO toolkit you know, and the AI visibility data you need.
Start a Free Trial
Start with
What CoMP does is create a pathway from “non-attainment” to “attainment under agreed conditions.” It’s less about locking the doors and more about putting a payment counter next to it.
For AI companies, that can reduce legal and operational risks. For publishers, it opens the door to using AI in a more systematic way.
This is “an important step in establishing interoperable, transparent standards for fair value exchange in the AI ecosystem,” Achim Schlosser, VP Global Data Standards at Bertelsmann, said in a statement. “We believe that strong compensation structures, coupled with visibility and attribution of content usage – are critical to sustaining quality journalism and premium content in the AI era.”
Why marketers should care
At first glance, this may sound like a publisher issue only. It is not.
AI solutions and agents are increasingly shaping the way people access information and products. If the economics of accessing content change, that can ripple through the entire media ecosystem.
If premium publishers find sustainable AI monetization models, the quality and availability of trusted content may stabilize. If they do not, the provision of authoritative content using AI responses may be reduced or shifted. That’s especially important for brands that rely on earned media, partnerships or high-profile publishers.
There is also a major structural change here. We are watching the industry try to formalize the AI information market. Instead of informal scratching and patchwork licensing, CoMP aims to create common rails.
As AI becomes the intermediary between consumers and content, the business rules surrounding that content are moving front and center. For marketers working in an AI-shaped discovery environment, that’s not a side issue. It’s part of the new digital media pipeline.
The public comment period begins on April 2, 2026. Click here to go to the site.



