Which Meta crawlers should you block in robots.txt?

Meta’s updated Commercial Terms will not take effect on November 20. On November 5, Meta paused them without setting a new date and said its existing terms remain in effect. They were first due on October 30, and Meta pushed the date back three weeks on October 23 after publishers objected. On October 28, Meta said it would consult news industry groups first, that the consultation could run past November 20, and that its existing terms stay in place until then. The same day, an Oslo court temporarily barred Meta from enforcing the new terms, in a case brought by Norwegian media groups. The paused terms would have kept Meta’s right to crawl websites associated with businesses that use its products. Search Engine Journal reads that as covering businesses that only manage a Facebook Page, without buying ads.

The short answer: if you don’t want your content used to train Meta’s AI models, block meta-externalagent in robots.txt. Leave facebookexternalhit and meta-externalfetcher alone. Think twice before blocking meta-webindexer, because that is the crawler behind Meta AI search, and blocking it means giving up citations and links in Meta AI answers.

Which Meta crawler does what?

Meta’s crawler documentation lists five common crawlers, each with its own user agent, and describes them like this:

• facebookexternalhit: fetches a page when someone shares it on Facebook, Instagram or Messenger, to build the title, description and thumbnail. Meta says it may bypass robots.txt for security or integrity checks. Don’t block it.
• meta-externalagent: crawls for uses such as training foundation AI models, or improving products by indexing content directly. This is the one to block if training is your concern.
• meta-webindexer: crawls to improve Meta AI search results. Meta says allowing it helps it cite and link to your content in Meta AI responses.
• meta-externalads: crawls for advertising and other business-related products and services.
• meta-externalfetcher: fetches single links when a user asks, including for agentic AI tasks. Meta says it may bypass robots.txt because the fetch was requested by a person, so a Disallow rule here does little.

One catch: Meta describes meta-externalagent as covering both training and product indexing. Blocking it is a single switch for both. You can’t keep one and refuse the other with that token.

What is happening with Meta’s new terms?

Nothing changes on November 20: Meta paused the updated terms on November 5, gave no new date, and said it has made no changes to how it crawls. Before that, on October 23, Meta told Reuters it was moving the start date from October 30 to November 20, to give itself more time to explain what is changing and what is not. Meta also said it will explain how it uses web crawlers, and that content owners can still use robots.txt to signal their preferences. The announcement came a day after the News/Media Alliance published an open letter to Mark Zuckerberg about the crawl clause. The delays and the pause buy you time, not a different answer: the crawlers and the robots.txt rules below work the same way. Google, Microsoft, OpenAI and Anthropic have since backed a shared robots.txt token for AI training, but I found no commitment from Meta, so these rules still need their own group.

The crawl clause in Meta’s updated Commercial Terms no longer lists “train” as a purpose, but Meta kept the right to crawl. Search Engine Journal reported on October 19 that the new wording allows crawling to “provide, develop, improve, research, and maintain advertising and business products”, plus integrity and security, and that the new language doesn’t mention robots.txt.

Meta told Nieman Lab the updates don’t give it new rights that supersede robots.txt preferences. The same report noted the UK English preview of the ad terms still carried the old training wording on October 19. With the wording still moving, I’d base the decision on the crawlers Meta documents, not on one clause.

How do you block meta-externalagent on WordPress?

Add a robots.txt group for meta-externalagent with Disallow: /. These are the lines:

User-agent: meta-externalagent
Disallow: /

Where they go depends on how your site serves robots.txt. WordPress generates a virtual robots.txt on request, but only when no physical robots.txt file sits in the site root. If a file exists, the web server sends that file and WordPress never runs. Yoast, Rank Math and All in One SEO also have robots.txt editors; check whether yours writes a file or edits the virtual output.

For the virtual robots.txt, core passes its output through the robots_txt filter in do_robots(), with two arguments: the output string and whether the site is public. This snippet appends the Meta group. Tested on WordPress 7.1.3, the group appears at the end of the virtual robots.txt, after core’s own lines. Put it in a must-use plugin or snippets plugin, not your theme.

add_filter( 'robots_txt', function ( $output, $public ) {
	$output .= "\nUser-agent: meta-externalagent\n";
	$output .= "Disallow: /\n";
	return $output;
}, 10, 2 );

If you use a physical file instead, the filter won’t run, so the file has to hold everything, including your sitemap line:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: meta-externalagent
Disallow: /

Sitemap: https://example.com/wp-sitemap.xml

Then check it:

1. Look for a robots.txt file in your site root over SFTP or your host’s file manager. If one is there, edit that file; the snippet will do nothing.
2. Open yourdomain.com/robots.txt in a private window. It worked if you see the meta-externalagent group with Disallow: /.
3. Paste one of your URLs into Meta’s Sharing Debugger. It worked if the title, description and image still come through, which means you haven’t caught facebookexternalhit by mistake.
4. Wait at least 24 hours. Meta’s documentation says crawlers may cache robots.txt for up to that long. You know it took effect when the log check below stops showing new meta-externalagent requests.

Step 3 matters. Security plugins and firewalls that block “bot” user agents can return a 403 to facebookexternalhit, and then links to your site shared on Facebook or Instagram show up bare. Meta asks site owners to allowlist that crawler by user agent or, more securely, by IP.

Does robots.txt actually stop Meta’s crawlers?

No, robots.txt is a request that a crawler chooses to honour, not a lock. Meta’s documentation presents robots.txt as the way to control its crawlers, with the two exceptions above, but the only way to know what happens is to look at your server logs.

Search your access log for meta-externalagent after the 24 hour window. To confirm a request really came from Meta, Meta’s documentation says its crawler IPs belong to AS32934 and gives this lookup:

whois -h whois.radb.net -- '-i origin AS32934' | grep ^route

Meta notes these addresses change often, so check against a fresh list. If you want to see your pages the way these bots do, I covered that in how to check what AI crawlers can see on your WordPress pages.

Why keep AI training and AI search separate?

Because they give you different things back. Training uses your content to build a model, with no link or credit to you. AI search reads your page to answer a question and, per Meta, can cite and link it. For anyone working on AEO or GEO, the second one is the visibility you’re after.

Blocking everything with “meta” in the name throws away both. Splitting them lets you refuse training and still compete for Meta AI citations. I’d treat each AI crawler as its own decision for that reason, the same way I think about llms.txt on WordPress: decide what each system gets, then write it down where crawlers look.

meta-externalads is a judgement call. Its purpose sits close to the new clause’s “advertising and business products” wording, but Meta doesn’t say what blocking it costs you, so I’d leave it open until there’s a clear reason. For the wider picture, I wrote up what AI search taught me about SEO, AEO, GEO and CRO.