---
title: "AI Bot Blocking by Site Size: Top Sites Block 12x as Often"
url: https://www.suffdigital.com/resources/data-studies/big-sites-block-gptbot
description: "AI bot blocking by site size, across 15.5 million sites: 14.1% of the 1,000 most visited block Perplexity, 12.3 times the small-site rate. See the data."
lang: en
---
Share
Share on X: https://twitter.com/intent/tweet?url=https%3A%2F%2Fwww.suffdigital.com%2Fresources%2Fdata-studies%2Fbig-sites-block-gptbot&text=The%201%2C000%20Most%20Visited%20Websites%20Block%20Perplexity%27s%20AI%20Crawler%2012%20Times%20as%20Often%20as%20Small%20Sites
Share on Facebook: https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.suffdigital.com%2Fresources%2Fdata-studies%2Fbig-sites-block-gptbot
Share on LinkedIn: https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.suffdigital.com%2Fresources%2Fdata-studies%2Fbig-sites-block-gptbot
Share on Reddit: https://www.reddit.com/submit?url=https%3A%2F%2Fwww.suffdigital.com%2Fresources%2Fdata-studies%2Fbig-sites-block-gptbot&title=The%201%2C000%20Most%20Visited%20Websites%20Block%20Perplexity%27s%20AI%20Crawler%2012%20Times%20as%20Often%20as%20Small%20Sites
**14.1%** of the 1,000 most visited websites block Perplexity's AI crawler, 12.3 times the 1.1% of websites ranked outside the top 10 million. The same divide holds for OpenAI's GPTBot, blocked by 23.1% of the most visited websites against 8.3% of small sites.
Analysis of 15,470,478 websites · the public HTTP Archive crawl and Google's Chrome UX Report popularity ranking, September 2026
## Key Findings
1. 1. **14.1%** of the 1,000 most visited websites block Perplexity's crawler, 12.3 times the 1.1% of websites outside the top 10 million, the widest gap of any AI crawler. Copy stat
2. 2. **23.1%** of the 1,000 most visited websites block OpenAI's GPTBot, nearly three times the 8.3% of websites ranked outside the top 10 million. Copy stat
3. 3. GPTBot blocking falls at every step down the popularity ranking, from **19.2%** of websites ranked 1,000 to 10,000 to 11.2% of websites ranked 100,000 to 1 million. Copy stat
4. 4. **28.9%** of the 1,000 most visited websites block at least one AI crawler, against 9.3% of websites outside the top 10 million. Copy stat
5. 5. **19.2%** of the 1,000 most visited websites opt out of Google's AI training with Google-Extended, against 7.4% of websites outside the top 10 million. Copy stat
## Summary
A national news site, a large retailer and a local bakery all publish a small text file called robots.txt that tells visiting crawlers which pages they may read. At the news site, that file increasingly carries a line naming GPTBot and telling it to stay out. At the bakery, it almost never does.
That choice decides which content AI companies can collect to train their models, and which sites AI engines have to learn from instead. When the biggest publishers and brands close the door, the open web that remains, most of it made up of smaller sites, carries more of the weight.
The split is widest for Perplexity. **14.1%** of the 1,000 most visited websites block its crawler, 12.3 times the 1.1% of sites ranked outside the top 10 million. OpenAI's GPTBot shows the same pattern at a larger scale: 23.1% of the most visited websites block it, against 8.3% of small sites, nearly three times the rate.
## What we measured
We read the robots.txt files of 15,470,478 websites in the public HTTP Archive crawl of September 2026 and grouped them by popularity, using Google's Chrome UX Report ranking of how often real Chrome users visit each site. The six groups run from the 1,000 most visited websites, 754 of which have a homepage in the HTTP Archive collection, down to 7,041,424 websites ranked outside the top 10 million, which we call small sites. For each group we calculated what percentage of websites block GPTBot and four other AI crawlers.
A crawler, or bot, is a program that visits web pages automatically and copies their content. GPTBot is OpenAI's crawler for collecting model training data; ClaudeBot is Anthropic's; PerplexityBot is the crawler behind Perplexity's AI search engine. Google-Extended is not a separate crawler but a name site owners use to tell Google not to use their content to train its Gemini AI models. A website blocks a bot when its robots.txt file includes a rule disallowing that bot.
## Perplexity's crawler faces the widest gap
14.1%
of the 1,000 most visited websites block Perplexity's crawler.
That is 12.3 times the rate of small sites, which almost never block PerplexityBot: 1.1% of sites outside the top 10 million do, the lowest rate for any crawler in the study. Between those two ends the rate falls at every step, from 7.9% of sites ranked 1,000 to 10,000 to 3% of sites ranked 100,000 to 1 million.
Top 1,000 by AI crawler
PerplexityBot
14.1%
Any AI crawler
28.9%
GPTBot
23.1%
ClaudeBot
21.6%
Google-Extended
19.2%
Source: Suff Digital analysis of 15,470,478 websites · the public HTTP Archive crawl and Google's Chrome UX Report popularity ranking, September 2026
| AI crawler | Top 1,000 | Beyond 10 million | Gap |
| --- | --- | --- | --- |
| PerplexityBot | 14.1% | 1.1% | 12.3x |
| Any AI crawler | 28.9% | 9.3% | 3.1x |
| GPTBot | 23.1% | 8.3% | 2.8x |
| ClaudeBot | 21.6% | 7.9% | 2.7x |
| Google-Extended | 19.2% | 7.4% | 2.6x |
Perplexity's crawler fetches pages to build answers on the spot, and those answers can replace a visit to the publisher's own site. That gives large publishers, who live on those visits, a specific reason to block it. A small business sees the same crawler as a way to get named in answers, not as a rival for its readers.
## Blocking rises with every step up in popularity
23.1%
of the 1,000 most visited websites block OpenAI's GPTBot.
The pattern is a clean staircase. Each popularity band blocks GPTBot more often than the band below it: 23.1% of the top 1,000, 19.2% of sites ranked 1,000 to 10,000, 14.7% of sites ranked 10,000 to 100,000, 11.2% of sites ranked 100,000 to 1 million, 8.8% of sites ranked 1 million to 10 million and 8.3% of sites beyond that. The same order holds for every AI crawler in the study.
Websites that block GPTBot by popularity rank
Top 1,000
23.1%
1,000 to 10,000
19.2%
10,000 to 100,000
14.7%
100,000 to 1 million
11.2%
1 million to 10 million
8.8%
Beyond 10 million
8.3%
Large sites have more to protect. News publishers, reference sites and big retailers produce content that costs money to make, and blocking a crawler keeps control over how AI companies use it. A small business site has no such deal on the table, and most leave their robots.txt file untouched.
## Three in ten top sites block at least one AI crawler
Counting any AI crawler, **28.9%** of the 1,000 most visited websites block at least one, against 9.3% of small sites, a gap of about three to one. Every major AI company's crawler faces the same divide: the top 1,000 block Anthropic's ClaudeBot at 21.6% against 7.9% for small sites, and 19.2% opt out of Google's AI training against 7.4%.
Block GPTBot by popularity rank
Top 1,000
23.1%
1,000 to 10,000
19.2%
10,000 to 100,000
14.7%
100,000 to 1 million
11.2%
1 million to 10 million
8.8%
Beyond 10 million
8.3%
| Popularity rank | Websites | Block GPTBot | Block any AI crawler | Opt out with Google-Extended | Block PerplexityBot | Block ClaudeBot |
| --- | --- | --- | --- | --- | --- | --- |
| Top 1,000 | 754 | 23.1% | 28.9% | 19.2% | 14.1% | 21.6% |
| 1,000 to 10,000 | 6,746 | 19.2% | 23.5% | 15.1% | 7.9% | 18% |
| 10,000 to 100,000 | 67,308 | 14.7% | 17.6% | 12.2% | 4.8% | 13.9% |
| 100,000 to 1 million | 708,042 | 11.2% | 13.1% | 9.6% | 3% | 10.8% |
| 1 million to 10 million | 7,646,204 | 8.8% | 10% | 7.6% | 2% | 8.3% |
| Beyond 10 million | 7,041,424 | 8.3% | 9.3% | 7.4% | 1.1% | 7.9% |
For the training crawlers the ratio is steady: the top 1,000 block GPTBot, ClaudeBot and Google-Extended about 2.6 to 2.8 times as often as small sites. Big sites treat AI training as one decision and apply it to every company at once.
## What this means for small and mid-size businesses
Do not copy the big sites. A national publisher blocking AI crawlers is protecting a content business. A local service company, a software startup or a regional retailer gains more from being read and cited by AI engines than it loses from having its pages used in training, and every big site that blocks leaves more room for the sites that stay open.
Check your own robots.txt file, which sits at yourdomain.com/robots.txt, so you know which AI crawlers you currently allow. If you want to stay out of model training, block the training crawlers such as GPTBot and Google-Extended by name, and leave the search crawlers that fetch pages for live answers open, such as PerplexityBot and OpenAI's OAI-SearchBot. Then make sure the pages you want cited load fast and state their answers plainly. As an ai seo company (https://www.suffdigital.com/seo/ai-seo-services), Suff Digital sets crawler access to match those goals and tracks where your site gets cited.
## Embed this research
Paste this on your site to embed the charts. It links back to the source automatically.
Source: The 1,000 Most Visited Websites Block Perplexity's AI Crawler 12 Times as Often as Small Sites - Suff Digital
## Cite this study
APA
Suff Digital. (2026). The 1,000 Most Visited Websites Block Perplexity's AI Crawler 12 Times as Often as Small Sites. https://www.suffdigital.com/resources/data-studies/big-sites-block-gptbot
Plain link
The 1,000 Most Visited Websites Block Perplexity's AI Crawler 12 Times as Often as Small Sites - Suff Digital - https://www.suffdigital.com/resources/data-studies/big-sites-block-gptbot
## Frequently asked questions
### Do big websites block AI crawlers?
### How many small websites block GPTBot?
### Which AI crawler do big websites block most often compared with small websites?
### How many websites opt out of Google's AI training?
## Related studies
- AI SEO 1.5 Million Websites Block at Least One AI Crawler From Reading Their Pages: https://www.suffdigital.com/resources/data-studies/ai-crawler-blocking-rate
- AI SEO Blocking Doesn't Keep You Out: 67% of Websites That Block GPTBot Still Get Cited by ChatGPT: https://www.suffdigital.com/resources/data-studies/gptbot-blocking-chatgpt-citations
- AI SEO 44.9% of the Websites AI Engines Cite Are Small, Low-Authority Sites: https://www.suffdigital.com/resources/data-studies/low-authority-sites-ai-citations
- AI SEO 47% of Top News Sites Block ChatGPT's Training Crawler vs 17.6% of All News Sites: https://www.suffdigital.com/resources/data-studies/news-sites-block-gptbot
## Structured data
```json
[
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "Organization",
"@id": "https://www.suffdigital.com/#organization",
"name": "Suff Digital",
"alternateName": "Suff Digital Marketing",
"url": "https://www.suffdigital.com/",
"logo": {
"@type": "ImageObject",
"@id": "https://www.suffdigital.com/#logo",
"url": "https://www.suffdigital.com/logo.png",
"contentUrl": "https://www.suffdigital.com/logo.png",
"caption": "Suff Digital"
},
"image": {
"@id": "https://www.suffdigital.com/#logo"
},
"description": "Suff Digital is a full-stack digital marketing agency that turns websites into predictable revenue engines, SEO, paid media, content, CRO, and web design built around revenue, not vanity metrics.",
"slogan": "Digital marketing built for what's next.",
"email": "hello@suffdigital.com",
"telephone": "+1-629-236-0099",
"foundingDate": "2022",
"founder": {
"@type": "Person",
"name": "Matt Suffoletto",
"jobTitle": "Founder & CEO"
},
"contactPoint": [
{
"@type": "ContactPoint",
"telephone": "+1-629-236-0099",
"email": "hello@suffdigital.com",
"contactType": "sales",
"areaServed": "US",
"availableLanguage": [
"en"
]
}
],
"address": {
"@type": "PostalAddress",
"addressLocality": "Nashville",
"addressRegion": "TN",
"postalCode": "37219",
"addressCountry": "US"
},
"areaServed": {
"@type": "Country",
"name": "United States"
},
"sameAs": [
"https://www.linkedin.com/company/suff-digital/",
"https://www.instagram.com/suffdigital/",
"https://www.youtube.com/@suffdigitalmarketing",
"https://www.tiktok.com/@suffdigital"
],
"knowsAbout": [
"Search Engine Optimization",
"Pay-Per-Click Advertising",
"Paid Social Advertising",
"Conversion Rate Optimization",
"Web Design",
"Digital PR",
"Link Building",
"Website Migration",
"Website Maintenance and Support",
"Technical SEO",
"Local SEO",
"AI Search Optimization"
],
"hasOfferCatalog": {
"@type": "OfferCatalog",
"name": "Suff Digital Services",
"itemListElement": [
{
"@type": "Offer",
"itemOffered": {
"@type": "Service",
"name": "Search Engine Optimization (SEO)",
"url": "https://www.suffdigital.com/"
}
},
{
"@type": "Offer",
"itemOffered": {
"@type": "Service",
"name": "Pay-Per-Click (PPC) Advertising",
"url": "https://www.suffdigital.com/services/ppc"
}
},
{
"@type": "Offer",
"itemOffered": {
"@type": "Service",
"name": "Paid Social Advertising",
"url": "https://www.suffdigital.com/services/paid-social"
}
},
{
"@type": "Offer",
"itemOffered": {
"@type": "Service",
"name": "Conversion Rate Optimization (CRO)",
"url": "https://www.suffdigital.com/services/cro"
}
},
{
"@type": "Offer",
"itemOffered": {
"@type": "Service",
"name": "Web Design",
"url": "https://www.suffdigital.com/services/web-design"
}
},
{
"@type": "Offer",
"itemOffered": {
"@type": "Service",
"name": "Digital PR & Link Building",
"url": "https://www.suffdigital.com/services/digital-pr"
}
},
{
"@type": "Offer",
"itemOffered": {
"@type": "Service",
"name": "Website Migration",
"url": "https://www.suffdigital.com/services/website-migration"
}
},
{
"@type": "Offer",
"itemOffered": {
"@type": "Service",
"name": "Website Maintenance & Support",
"url": "https://www.suffdigital.com/services/website-maintenance"
}
}
]
}
},
{
"@type": "WebSite",
"@id": "https://www.suffdigital.com/#website",
"url": "https://www.suffdigital.com/",
"name": "Suff Digital",
"inLanguage": "en-US",
"publisher": {
"@id": "https://www.suffdigital.com/#organization"
}
}
]
},
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "Article",
"@id": "https://www.suffdigital.com/resources/data-studies/big-sites-block-gptbot#article",
"headline": "The 1,000 Most Visited Websites Block Perplexity's AI Crawler 12 Times as Often as Small Sites",
"description": "14.1% of the 1,000 most visited websites block Perplexity's AI crawler, 12.3 times the 1.1% of websites ranked outside the top 10 million. The same divide holds for OpenAI's GPTBot, blocked by 23.1% of the most visited websites against 8.3% of small sites.",
"image": [
"https://www.suffdigital.com/images/suff-digital-share-16x9.jpg"
],
"url": "https://www.suffdigital.com/resources/data-studies/big-sites-block-gptbot",
"datePublished": "2026-09-24",
"dateModified": "2026-09-24",
"author": {
"@type": "Person",
"name": "Matt Suffoletto",
"jobTitle": "Founder & CEO",
"url": "https://www.suffdigital.com/who-we-are"
},
"publisher": {
"@id": "https://www.suffdigital.com/#organization"
},
"mainEntityOfPage": {
"@id": "https://www.suffdigital.com/resources/data-studies/big-sites-block-gptbot",
"@type": "WebPage"
},
"isPartOf": {
"@id": "https://www.suffdigital.com/#website"
}
},
{
"@type": "BreadcrumbList",
"@id": "https://www.suffdigital.com/resources/data-studies/big-sites-block-gptbot#breadcrumb",
"itemListElement": [
{
"@type": "ListItem",
"position": 1,
"name": "Home",
"item": "https://www.suffdigital.com/"
},
{
"@type": "ListItem",
"position": 2,
"name": "Resources",
"item": "https://www.suffdigital.com/resources"
},
{
"@type": "ListItem",
"position": 3,
"name": "Data Studies",
"item": "https://www.suffdigital.com/resources/data-studies"
},
{
"@type": "ListItem",
"position": 4,
"name": "The 1,000 Most Visited Websites Block Perplexity's AI Crawler 12 Times as Often as Small Sites"
}
]
},
{
"@type": "Dataset",
"@id": "https://www.suffdigital.com/resources/data-studies/big-sites-block-gptbot#dataset",
"name": "The 1,000 Most Visited Websites Block Perplexity's AI Crawler 12 Times as Often as Small Sites",
"description": "14.1% of the 1,000 most visited websites block Perplexity's AI crawler, 12.3 times the 1.1% of websites ranked outside the top 10 million. The same divide holds for OpenAI's GPTBot, blocked by 23.1% of the most visited websites against 8.3% of small sites.",
"url": "https://www.suffdigital.com/resources/data-studies/big-sites-block-gptbot",
"creator": {
"@id": "https://www.suffdigital.com/#organization"
},
"datePublished": "2026-09-24",
"dateModified": "2026-09-24",
"license": "https://creativecommons.org/licenses/by/4.0/",
"isAccessibleForFree": true,
"measurementTechnique": "The public HTTP Archive crawl of 15.5 million websites; Google's Chrome UX Report popularity ranking"
},
{
"@type": "FAQPage",
"@id": "https://www.suffdigital.com/resources/data-studies/big-sites-block-gptbot#faq",
"mainEntity": [
{
"@type": "Question",
"name": "Do big websites block AI crawlers?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes, far more than small ones. 23.1% of the 1,000 most visited websites block OpenAI's GPTBot and 28.9% block at least one AI crawler (September 2026), against 8.3% and 9.3% of websites ranked outside the top 10 million."
}
},
{
"@type": "Question",
"name": "How many small websites block GPTBot?",
"acceptedAnswer": {
"@type": "Answer",
"text": "8.3% of websites ranked outside the top 10 million block OpenAI's GPTBot, against 23.1% of the 1,000 most visited websites."
}
},
{
"@type": "Question",
"name": "Which AI crawler do big websites block most often compared with small websites?",
"acceptedAnswer": {
"@type": "Answer",
"text": "PerplexityBot. 14.1% of the 1,000 most visited websites block Perplexity's crawler, 12.3 times the 1.1% rate among websites outside the top 10 million."
}
},
{
"@type": "Question",
"name": "How many websites opt out of Google's AI training?",
"acceptedAnswer": {
"@type": "Answer",
"text": "19.2% of the 1,000 most visited websites use Google-Extended to keep their content out of Google's AI model training, against 7.4% of websites ranked outside the top 10 million."
}
}
]
}
]
}
]
```