1.5 Million Websites Block at Least One AI Crawler From Reading Their Pages

1.5 million websites, 9.9% of the 15.5 million in the public HTTP Archive crawl, block at least one AI crawler. They block the bots that collect training material about four times as often as the bots that fetch pages for live AI answers.

Share of all websites blocking it by AI crawler

GPTBot

8.7%

ClaudeBot / anthropic-ai / Claude-Web

8.3%

CCBot

8.2%

Bytespider

7.9%

Amazonbot

7.8%

Google-Extended

7.6%

Applebot-Extended

7.2%

Meta-ExternalAgent

7.1%

OAI-SearchBot or ChatGPT-User

2.1%

PerplexityBot

1.7%

Source: Suff Digital analysis of 15,470,478 websites · the public HTTP Archive crawl, September 2026

Share of all websites by AI crawler