Skip to content
Mera News

Publisher guide, chapter 7

AI scraping: what each control really does

Last reviewed November 2026

AI companies gather news from the web to train models and to answer questions, and some of their crawlers copy whole articles. No single setting stops this. This chapter shows what each control covers, how to use them without harming your readers, and where Mera stands. It isn't legal advice.

What each control does

ControlWhat it doesWhat it doesn't do
robots.txt (RFC 9309)Asks crawlers to stay out of paths you nameIt is voluntary, and the RFC says it isn't access authorization
__usage_ai-training tokenStates a preference against AI training, in robots.txtIt isn't an IETF standard yet, and it covers training only. It doesn't limit crawling and doesn't address content already collected
Google-ExtendedManages use of your content for Gemini training and for grounding in Gemini Apps and Vertex AIIt has no effect on Google Search inclusion or ranking, and it doesn't keep your content out of AI Overviews or AI Mode
nosnippet, max-snippetLimit your text as direct input for AI Overviews and AI ModeThey also cost you normal snippets in Search
Search Console, Search generative AIExcludes a property from AI Overviews, AI Mode and generative AI features in DiscoverIt doesn't affect AI training. Excluded sites get no traffic or impressions from these features
RSL 1.0Publishes machine-readable licence terms, through a License: line and a license.xml fileIt doesn't change Allow or Disallow rules, and on its own it blocks nothing
EU reservation (Art. 4(3), Directive 2019/790)Reserves your rights against text and data mining, by machine-readable means for online contentIt is a legal statement and not a technical block
Cloudflare Bot Preference SyncAdds managed rules for AI crawlers to your robots.txt, based on your access policiesCompliance with the file is voluntary. Edge blocking is the next control in this table
Cloudflare access policiesBlocks AI crawlers at the edge, by behaviour: Search, Agent or TrainingChoose each behaviour with care. Blocking Search also stops crawlers that index your pages to answer questions later
WAF rate limitsLimit requests per client over a set periodCounters lag by a few seconds, and rules on verified bots can affect SEO

A starting robots.txt

Name the crawlers you want to keep out, add the shared token, and point to your licence.

License: https://example.com/license.xml

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

User-agent: __usage_ai-training
Disallow: /

User-agent: *
Allow: /

Notes on this file:

  • Serve the file at /robots.txt, on every subdomain.
  • Name the token directly. The draft says a User-agent: * group doesn't count as matching it.
  • Companies run several bots. OpenAI's GPTBot is for training and OAI-SearchBot is for ChatGPT search. Anthropic's ClaudeBot is for training and Claude-SearchBot is for search. Blocking a search bot can reduce your visibility there, so choose each one on purpose.
  • User-requested fetchers differ. OpenAI says robots.txt rules may not apply to ChatGPT-User.

The shared training token

OpenAI, Google, Microsoft and Anthropic announced support for __usage_ai-training on 30 October 2026. It comes from an individual Internet-Draft that has no formal standing at the IETF, and rollout differs by company.

CompanyStatusNote
AnthropicHas started implementingSupport across its services will take time. A rule that names ClaudeBot takes priority over the token
OpenAIAims for the coming monthsExisting crawler controls stay in place
MicrosoftFrom 1 February 2027It also adds page-level preferences
GoogleAims for 2027, and early 2027 in its crawler documentationIt plans to read the token as disallowing Google-Extended, unless you name Google-Extended yourself

Google-Extended covers both training and grounding. Once Google supports the shared token, using it without naming Google-Extended also gives up grounding in Gemini Apps. A User-agent: Google-Extended group with Allow: / keeps you eligible for grounding, but it allows Gemini training as well.

If you use NoArchive or NoCache to signal preferences to Microsoft, its new preferences replace them, and it stops reading them that way after 1 March 2027.

Licence terms with RSL

RSL 1.0 relies mainly on compliance by crawlers, and servers may add enforcement. This license.xml permits search with attribution and prohibits AI uses.

<rsl xmlns="https://rslstandard.org/rsl">
  <content url="/">
    <license>
      <permits type="usage">search</permits>
      <prohibits type="usage">ai-all</prohibits>
      <payment type="attribution"/>
    </license>
  </content>
</rsl>

Search and AI Overviews

To limit one page's text in AI Overviews and AI Mode, use a snippet control. max-snippet sets a length, and nosnippet removes the text.

<meta name="robots" content="max-snippet:160">

For a whole site or section, go to Settings, then Search generative AI, in Search Console. A child property can inherit or override the setting, and changes generally take a few days. Google says it isn't a ranking signal for other parts of Search.

Keeping the site accessible while you do this

Blocking crawlers must not break screen readers, reader modes, search or your RSS feed.

  • Keep articles as real text. Text drawn on a canvas, turned into images or scrambled in the HTML fails screen readers and reader modes, which read the page in the reader's own browser.
  • Keep CAPTCHAs off reading pages. A W3C draft note says CAPTCHAs often stop people with disabilities from completing the task. Prefer non-interactive checks, and offer alternatives where a challenge can't be avoided.
  • Protect search. Verify Googlebot by DNS or by Google's IP ranges, and don't block it when you mean Google-Extended.
  • Keep robots.txt reachable. Under RFC 9309, a 5xx response makes a crawler assume everything is disallowed, and a 4xx lets it fetch anything. Exempt the file from bot rules.
  • Use error codes briefly. Google says repeated 500, 503 or 429 responses over several days can drop a URL from its index.
  • Exempt your feed. A browser challenge returns a web page, which a feed reader can't use. Cloudflare's Super Bot Fight Mode can be skipped with a custom rule for your feed path. Bot Fight Mode, on the Free plan, can't be skipped that way.

Where Mera stands

Mera News is on the publisher's side here. It isn't a crawler and takes nothing from your pages. Its news comes only from the RSS feed you publish, which is usually a little text and a link back. You control what goes in it, and Mera shows the headline and sends the reader to you. Full-content crawlers are the problem this chapter addresses.

The feed fetcher identifies itself as MeraNewsBot/1.0 (mera.news/bot), so make sure bot protection doesn't block your feed. To test it, run this and look for 200:

curl -s -o /dev/null -w "%{http_code}\n" -A "MeraNewsBot/1.0 (+https://mera.news/bot)" https://example.com/feed.xml

To leave Mera, filter that User-Agent or write to publishersupport@mera.news. The fetcher requests only your feed URL and doesn't read robots.txt, so a robots.txt rule won't reach it. Crawler and feeds sets out what Mera does with feed text, and a separate tool, started by hand, that looks for subscription pages.

Checklist

  • Decide what you want for training, for AI search and for grounding. Each company separates them differently.
  • Add named crawler groups and the __usage_ai-training group to robots.txt.
  • Set Google-Extended on purpose, knowing it covers both Gemini training and grounding.
  • Decide on the Search Console control, and record your reason.
  • Publish a machine-readable rights reservation, and ask counsel which form suits you.
  • Add edge rules and rate limits for crawlers that ignore your file.
  • Confirm robots.txt and your feed return 200 to bots.
  • After every rule change, test an article with a screen reader and a reader mode, and check Googlebot with the URL Inspection tool.

Sources