Should You Block AI Crawlers?

Blocking an AI crawler is easy to describe and harder to evaluate. It may reduce access by a company that honours your rules, but it cannot make public content impossible to copy.

The decision is mostly about the relationship you want with model providers: what they may collect, whether they send useful traffic, and whether the cost of crawling is acceptable.

Reasons to block them

Publishers may not want articles used for model training without payment or clear attribution. Shops may see little value in letting another service reproduce carefully maintained product information. A crawler can also consume noticeable bandwidth when it fetches many pages quickly.

News organisations were early to draw that line. A 2024 tracker reported that 55% of the news homepages it checked blocked GPTBot. That number describes a particular sample and point in time, not the whole web.

Reasons to allow them

Some AI products cite and link to sources. Appearing in those answers may send readers or help a brand remain visible as discovery habits change. Researchers and smaller model projects can also be useful users of public material.

Allowing a crawler does not guarantee attribution or traffic, just as blocking one does not guarantee that the content never reaches a dataset through another route.

Decide crawler by crawler

“AI bots” are not one thing. A provider may use separate user agents for training, search, and fetching a URL at a user’s request. Read the provider’s current documentation and write rules for the access you actually object to.

I would start by checking server logs. If a crawler creates cost, ignores rules, or offers no value, blocking it is reasonable. If the concern is copyright or confidential data, robots.txt is not enforcement or access control. Keep private material behind authentication and treat crawler rules as a stated preference, not a lock.