Meta's New Business Terms and Your Website: What Publishers Should Set in robots.txt Before November 20
Publisher In a Box9 min read
Table of Contents
If your business runs a Facebook Page, Meta's updated Commercial Terms give Meta permission to crawl the websites associated with that business. The change drew sharp criticism from publishers in mid October, Meta rewrote the clause within days, and the effective date on Meta's own terms page now reads November 20, 2026, not the October 30 date that most early coverage reported.
That leaves publishers with a few weeks and one practical question. What exactly are you agreeing to, and which settings on your own website still control what Meta's crawlers do? This guide covers what the terms say now, the five crawlers Meta documents, and the robots.txt choices that matter for GEO, often referred to as AI search optimization, where being crawled is also how you get cited.
What Meta's new Commercial Terms say about crawling
Meta's Commercial Terms apply whenever you use Meta's products for a business purpose, and the updated version lists what counts, including running ads, selling products, creating a product catalog, using Meta Business Suite and managing a Page. For most digital publishers, managing a Facebook Page alone is enough to be covered.
The new clause, as it reads on Meta's preview page today, authorizes Meta "or any entity acting on our behalf" to use automated software, naming web scrapers, bots, crawlers and user agents, and to create test credentials to access, index, cache, analyze and crawl the domains and web pages associated with your products or services. It also lets Meta periodically pull updates from those pages. The stated purpose is to provide, develop, improve, research and maintain advertising and business products, and to maintain the integrity and security of Meta's products.
Under the updates section, Meta says that continuing to use its products for business purposes after notice of an update means you agree to it. There is no separate opt-out form for this clause.
What changed after the backlash
The first version of the clause went further. As reported by Engadget, Search Engine Journal and others in mid October, it said Meta could use content from those sites to "provide, develop, improve, research, train, and maintain" its products, which publishers read as permission to use their websites for AI training.
Meta then revised the language. The word "train" was removed, and the purpose was narrowed to advertising and business products. Meta spokesperson Andy Stone said the crawling terms only applied to ads and business products, that the changes "were not about scraping news sites", and that content owners can still use robots.txt to signal their preferences.
Two concerns remain, and both are worth knowing before you decide what to do. The News/Media Alliance told the Mercury News, as reported by The Next Web, that training was taken out of the listed uses but not expressly banned. And as Search Engine Journal noted, neither the updated Commercial Terms nor the updated Self-Serve Ad Terms mention robots.txt, so Meta's commitment to honor it lives in its statements and its developer documentation rather than in the contract itself.
November 20, 2026
The effective date now shown on Meta's preview of its updated Commercial Terms, replacing the October 30 date in early coverage
Source: Meta, Commercial Terms preview page (read October 23, 2026)
Meta's five web crawlers, and what each one does
This is where publishers keep real control, because Meta does not use a single crawler. Its developer documentation, last updated May 21, 2026, lists five, each with its own user agent name and its own job.
FacebookExternalHit builds the link preview, the title, description and thumbnail image, when a link to your site is shared on Facebook, Instagram or Messenger. Meta says it may bypass robots.txt when performing security or integrity checks.
Meta-WebIndexer crawls the web to improve Meta AI search results. Meta says allowing it in robots.txt "helps us cite and link to your content in Meta AI's responses."
Meta-ExternalAds crawls for uses such as improving advertising and other business products and services.
Meta-ExternalAgent crawls for uses such as training foundation AI models or improving products by indexing content directly.
Meta-ExternalFetcher fetches individual links at a user's request, including to help AI complete tasks for users. Meta says it may bypass robots.txt rules because the fetch was requested by a user.
Read against the new terms, the split matters. The revised clause is about advertising and business products, which is the stated job of Meta-ExternalAds. Training is the stated job of Meta-ExternalAgent. Being cited in Meta AI's answers depends on Meta-WebIndexer. Each can be allowed or blocked on its own.
Blocking every Meta crawler is the easy reaction and the expensive one. Block the wrong name and your Facebook link previews break, or you disappear from Meta AI's answers.
What to set in robots.txt
Meta's documentation says to block a crawler you add a disallow rule for that crawler's name in your robots.txt file, and that changes can take up to 24 hours to apply because crawlers cache the file. The decisions below are the ones we would want every publisher to make deliberately rather than by default.
Keep FacebookExternalHit allowed
If you share links to your website on Facebook, this is the crawler that builds the preview card. Blocking it can leave your link posts without a title or image, which hurts click through on the posts you most want people to tap. It is also the one Meta says may bypass robots.txt for security checks, so blocking it buys little.
Decide on Meta-WebIndexer as an AI search decision
Meta-WebIndexer is the crawler tied to being cited and linked in Meta AI's responses. For a publisher working on AI search visibility, blocking it trades away a citation channel. We explain how citation traffic fits into revenue in our guide to how publishers make money from AI, and why readers now split their time between AI answers and Google in is AI search replacing Google.
Decide on Meta-ExternalAgent as a training decision
If you do not want your content used for training foundation AI models, Meta-ExternalAgent is the name to disallow. It is a separate choice from Meta AI citations, so you can block training while still allowing Meta-WebIndexer.
Decide on Meta-ExternalAds as an advertising decision
Meta-ExternalAds is the crawler for advertising and business products, which is what the revised clause describes. The clause says that use includes helping you create and deliver ads, giving users information about your products and services, and connecting you with users who may be interested in them. If you advertise on Meta or sell through its product tools, those are the uses you would be limiting by blocking it. If you do neither, the case for allowing it is weaker.
Know what robots.txt cannot do
Meta-ExternalFetcher may bypass robots.txt for user-requested fetches, and FacebookExternalHit may bypass it for security checks, so robots.txt is a preference signal, not a lock. If you need a hard block, that is a firewall or server rule, and it carries the same tradeoffs.
A configuration that blocks training while keeping link previews and Meta AI citations looks like this:
Check what your robots.txt says today
If your robots.txt does not name Meta's crawlers, your general rule decides for every one of them. Ours is an example: the Publisher in a Box robots.txt names no Meta crawler, so all five fall under our general allow rule. That is a choice we made for AI search reach, and the point of this guide is that it should be a choice for you too, not an accident.
What this means for a Facebook publisher
For a Facebook publisher, the Page and the website are now linked in Meta's terms as well as in your traffic. That is not a reason to leave Facebook, because the audience and the payouts are still there. It is a reason to treat your robots.txt as a business document, review it before November 20, and decide crawler by crawler what you are giving Meta and what you are getting back in previews, citations and reach.
Where to go from here
If you want your content set up to be found and cited in AI search, including Meta AI, ChatGPT, Perplexity and Google's AI answers, the GEO Authority System is how Publisher in a Box builds that visibility for publishers, from the technical crawl settings to the content AI engines cite.
Frequently asked questions
When do Meta's new Commercial Terms take effect?
Meta's preview page for the updated Commercial Terms gives an effective date of November 20, 2026. Early coverage in October reported October 30, so check the page itself before relying on either date.
Do Meta's new terms let Meta train AI on my website?
The revised clause no longer lists training. It authorizes crawling to develop and maintain advertising and business products and to keep Meta's products secure. The News/Media Alliance has pointed out that training was removed from the list but not expressly banned.
Can I opt out of Meta's crawling terms?
There is no separate opt-out for the clause. Meta's terms say continued business use after notice means you agree. What you can control is your robots.txt, which Meta says its crawlers use to read your preferences, with the exceptions Meta documents.
Which Meta crawler should I block to stop AI training?
Meta-ExternalAgent, which Meta documents as crawling for uses such as training foundation AI models. Blocking it does not block Meta-WebIndexer, which supports citations in Meta AI.
Will blocking Meta crawlers break my Facebook link previews?
It can. FacebookExternalHit builds link previews for links shared on Facebook, Instagram and Messenger. Keep it allowed if you share links to your site.
How long does a robots.txt change take to work for Meta?
Meta's documentation says to allow up to 24 hours, because its crawlers may cache your robots.txt for that long.
Key takeaways
Meta's updated Commercial Terms cover any business use of its products, including managing a Facebook Page.
The effective date on Meta's preview page now reads November 20, 2026.
After the backlash, Meta removed "train" from the crawling clause and limited it to advertising and business products, but the terms do not mention robots.txt.
Meta documents five crawlers: FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher.
Keep FacebookExternalHit allowed for link previews, and treat Meta-WebIndexer as an AI search citation decision.
Disallow Meta-ExternalAgent if you do not want training use, and know that Meta-ExternalFetcher may bypass robots.txt.
Sources
Meta, Commercial Terms preview (effective November 20, 2026), facebook.com
The team behind 300M+ managed followers. We help publishers scale traffic, revenue, and audience across Facebook, Google Discover, and syndication networks.
